@iScienceLuvr
How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference New paper from Meta. "Our experiments span models with up to 1B parameters, trained on up to 155B tokens using up to 1.07 × 10^21 training FLOPs, with up to 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained." "We compare exact weight sharing across 2 loops with small loop-specific low-rank adapters that allow each loop to learn a slightly different transformation. These adapters provide only modest gains when few loops are used, but become consistently more useful as the same weights are reused across more loops" "Naively applying Parallel Looped Transformers (PLT) in our setting does not preserve the gains from looping: quality can degrade as more loops are added. We find that preserving the current-token representation, separately normalizing it and the recurrent state, and controlling how they are combined substantially improves PLT."
