@ahatamiz1
I agree with one thing. Making linear attention work at scale takes talent, and not every lab has it. The rest doesn’t hold up. In order: (1) “Why are the Western labs choosing to drop context instead of compressing it?” The first rigorous, controlled, large-budget test of compression was what we did at NVIDIA’s in June 2024(https://t.co/OQ3pkwmXXV). We trained 8B Mamba, Mamba-2 and Transformer models on the same data for up to 3.5T tokens. Additionally we trained an 8B hybrid of 43% Mamba-2, 7% attention and 50% MLP and we showed it beats the 8B Transformer on all 12 standard tasks (+2.65 points on average). Interestingly this hybrid variant was up to 8× faster at generation, and matched or beat the Transformer across 23 additional long-context tasks. This is seven months before MiniMax-01, fifteen before Qwen3-Next, sixteen before Kimi Linear. And since then NVIDIA has been shipping a hybrid architecture, with modifications, repeatedly. Good examples are Nemotron-H, Nemotron Nano 2, and Nemotron 3 (Nano, Super, Ultra) which is a Mixture-of-Experts hybrid Mamba-Transformer family with context up to 1M tokens, post-trained with multi-environment RL. Nano keeps exactly 6 attention layers out of 52 and the other 46 are Mamba-2 and MoE blocks. Super is 120B total / 12B active, pre-trained on 25 trillion tokens. IBM’s Granite 4.0 (October 2025) interleaves Mamba-2 and attention layers at 9:1. To this list, you can also add Jamba (AI21, March 2024, a 52B production hybrid), RecurrentGemma (Google, April 2024), Codestral Mamba (Mistral, July 2024) and Phi-4-mini-flash (Microsoft, July 2025). (2) “SWA is a lossless local transformation, therefore superior.?” SWA is lossless inside the window and zero outside of it and it is exactly why it fails so badly. (https://t.co/sY3p9p9e6u) for example shows that larger sliding windows hurts the long-context performance. This is because a precise local softmax lets the model avoid training the long-term memory it would otherwise rely on. They show that a hybrid with a 128-token window beat the same hybrid with a 2048 window by 16 NIAH points (88.9% higher recall), and at 131k the short-window models still retrieved about 30% of needles while the 2048-window model was near 0%. And the same degradation from larger training windows shows up in local-global stacks that alternate SWA with full attention ( e.g. the GPT-OSS/Gemma recipe). (3) “Compression is strictly harder to serve.” This was true in 2024. But now vLLM, FlashInfer, SGLang and TensorRT-LLM serve hybrid linear models extensively and without any issues. The most exciting part is the recent FlashInfer additions that let you use chunked gated-delta-rule prefill and gated-delta-rule decode kernels, with an SM100 (Blackwell) path. It even supports much newer models like Gated DeltaNet-2 and also lets you use efficient kernels for training. Also, let's not forget ReplaySSM (https://t.co/zSL7l5o8kO) which enables a very optimized speculative decoding path for linear attention. (4) “If you write all the kernels anyway, why not roll your own attention in the meantime?” On NVIDIA hardware, linear attention is just as free as attention. The Mamba CUDA kernels exist. flash-linear-attention, the Triton library that the GDN and KDA layers in Qwen3-Next, Qwen3.5 and Kimi Linear use, was written for NVIDIA GPUs, and vLLM and SGLang use it directly instead of reimplementing it. Megatron has supported Mamba-2 training since 2024. A Western lab on H100s that wants a hybrid doesn’t need to write a kernel. It can just import it. So kernel cost can’t be what keeps them on using the SWA. Also consider that decode is memory-bandwidth-bound, so the cost of every generated token is dominated by reading the KV cache, which grows with context length. A constant-size state removes that growth. That’s why Nemotron is mostly Mamba-2, and why Nemotron 3 Super gets 2.2× the throughput of GPT-OSS-120B at 8K in / 16K out. (5) “For your first release, it’s a much safer option to ship SWA. Everyone knows it’s working just fine.”? Safe compared to what? “Safe” should mean proven in production at scale. By that standard the hybrids are the safe choice and SWA is the bet. Look at what has shipped on Gated DeltaNet alone. Qwen3-Next, then Qwen3.5 from 27B up to 397B-A17B, then Qwen3.6. Kimi Linear, then Kimi K3 which is a 2.8 trillion parameters, open weights, 1M-token context, a 3:1 hybrid of Kimi Delta Attention and gated MLA. GLM-5.3-Flash on the same KDA layout. KDA is Gated DeltaNet with channel-wise instead of scalar gating. Right now, three frontier labs run their model in production with 1M context, RL post-training, etc. all on the same layer. Meanwhile SWA is the architecture with the documented issue. It fails past the window and drags any hybrid it sits in down to pure-SWA recall. And when MiniMax reached for an SWA hybrid as the fallback in M2, it failed and they went to full attention instead. In my view, shipping SWA for a first release in late 2026 isn’t conservative. It’s picking the layer with a known failure mode because it needs no new code.