Caduceus

Caduceus (Schiff et al., arXiv 2024) is a family of DNA foundation models built on Mamba state-space blocks instead of attention, which lets them scale to very long sequences at sub-quadratic cost. Its two innovations are BiMamba, a parameter-efficient bidirectional extension of the Mamba block, and MambaDNA, which adds reverse-complement equivariance: the network treats a DNA sequence and its reverse complement as the same object, an inductive bias matched to how DNA actually works.

Two variants are released: Caduceus-PS, which is equivariant by construction, and Caduceus-Ph, which achieves it via post-hoc conjoining. Both are pretrained on the human reference genome with 131k-base-pair contexts and load from Hugging Face. The authors report it beats previous long-range models on downstream benchmarks and outperforms models up to 10 times larger that lack bidirectionality and equivariance on a long-range variant-effect task. Those comparisons are author-reported and await independent replication.

Author-reported (Schiff et al., arXiv 2024): outperforms previous long-range models on downstream benchmarks; beats models up to 10x larger lacking bidirectionality/equivariance on a long-range variant-effect task; no independent replication published