@burkov
NeurIPS 2025 Best Paper Award: Attention lets language models decide which tokens matter at each position, but it has limitations—for example, a tendency to over-focus on early tokens regardless of their relevance. Gating mechanisms, which selectively suppress or amplify information flow in neural networks, have improved other architectures, so researchers have tried adding them to attention as well. However, prior attempts usually package gating together with other architectural changes, making its specific contribution hard to isolate. This paper separates those effects by systematically testing over 30 gating variants on dense models and mixture-of-experts models with up to 15 billion parameters. In a standard transformer layer, each attention head computes a weighted combination of values; the head outputs are concatenated and passed through a final linear projection. The winning approach identified in the paper inserts one extra operation before concatenation: each head's output is multiplied (element-wise or head-wise, with element-wise performing best) by a learned gate computed from the current token's representation. This allows each head to dampen or preserve its contribution depending on context. These architectural changes deliver practical benefits beyond small benchmark gains: 1. Training becomes more stable, supporting learning rates that cause baseline models to diverge. 2. The gating also greatly reduces "attention sinks"—the situation where early tokens absorb excessive attention—which in turn is associated with strong improvements on long-context benchmarks once the context window is extended using standard techniques. Talk to the paper on ChapterPal: https://t.co/sbBtE8y3RH Read the PDF: https://t.co/PS92Jg6GZq
