Beyond Transformers

The transformer's quadratic attention cost is its one structural weakness: doubling the sequence quadruples the attention compute and memory. For most of the field that hasn't mattered — context windows were short, and engineering tricks kept it manageable. But as context stretches to hundreds of thousands of tokens, a wave of architectures is trying to do what attention does at lower cost. None has dethroned the transformer; several are quietly showing up inside hybrid models.

State-Space Models (SSMs) and Mamba

State-space models process sequences like a recurrence: they maintain a fixed-size hidden state and update it token by token. The cost is linear in sequence length, and inference uses constant memory per step — no growing KV cache.

Mamba made SSMs competitive by adding selectivity: the state-update parameters become input-dependent, so the model can choose what to remember and what to forget. That input-dependence was the missing piece that let SSMs match transformers on language.

• Strength — linear scaling, constant-memory generation, strong on very long sequences (genomics, audio, long documents).
• Weakness — the fixed-size state is a bottleneck for tasks needing precise recall of arbitrary earlier tokens ("what was the exact number on line 4,000?"). Attention's KV cache stores everything; an SSM compresses, and compression loses detail.

Linear and Sub-Quadratic Attention

A family of methods approximates attention to drop below O(n²): kernelized/linear attention, low-rank approximations, and sliding-window attention that only attends locally. They trade some quality for speed. Sliding-window attention in particular is now standard in production models (Mistral, Gemma) because local attention captures most of what matters cheaply.

Hybrids Are Winning

The pragmatic answer turned out not to be "replace the transformer" but "mix." Models like Jamba and several recent releases interleave SSM/Mamba layers with a smaller number of full-attention layers. The attention layers handle precise recall and in-context lookups; the SSM layers carry the long-range, linear-cost load. You get most of attention's quality at a fraction of its long-context memory.

Why an Engineer Should Care

You probably won't choose the architecture — but it shows up in the model card:

• Long-context economics — an SSM-hybrid can offer a long context window far more cheaply than a pure-attention model, because there's no quadratic KV cache to hold. If your workload is long-document heavy, these models change the cost math.
• Recall behavior — pure or heavily-SSM models can be weaker at needle-in-haystack exact retrieval. Test recall on your data before trusting a non-transformer model for long-context lookup.
• Throughput — constant-memory generation means more concurrent sequences per GPU, which matters for high-traffic serving.

What to Watch For

Treat "linear attention" or "1M context" claims as hypotheses to verify, not guarantees. Cheap long context is only useful if the model actually uses the tokens — and architectures that compress state often look great on perplexity while quietly failing exact-recall evals. The needle-in-a-haystack test exists precisely because this gap is common.