{
  "type": "tweet",
  "id": "2068724224156397649",
  "url": "https://x.com/gerardsans/status/2068724224156397649",
  "twitterUrl": "https://twitter.com/gerardsans/status/2068724224156397649",
  "text": "🚨 New DeepMind’s Category Error on Transformer “State” (Mozer et al. 2026)\n\nPaper correctly flags transformers’ sequential consistency limits and offers a useful recurrent taxonomy. \n\nCore claim: “depth exhaustion” from pushing state deeper, misreads the transformer architecture.\n\nThe residual stream isn’t a persistent state structure or evolving “mental model”. It’s an intermediate calculation within the single pass forward based on token isomorphism and training corpus regularities devoid of any system-wide coherent meaning but context dependent and fragile per inference:\n\nh0 = prefill(prompt)\n- - - - - - - - - \nh1 = h0 + layer0\nh2 = h1 + layer1\n. . .\n/ layer_i = Attn(i) + FFN(i) /\n. . .\n- - - - - - - - - \n\nA running sum across layers (not time). \n\nThe full stack computes hL once, then last_FFN + unembed → logits → one token. \n\n- - - - - - - - - \nt = unembed(last_FFN)\n- - - - - - - - - \n\nThat token alone seeds the next auto-regressive step. \n\n- - - - - - - - - \ninput = input + t\n- - - - - - - - - \n\nNo internal carry-over. No “beliefs.” Just partial aggregates overwritten in the next pass.\n\n“Depth exhaustion” is an artifact: treating mid-layer partial sums as final outputs. Reading tea leaves without tasting.\n\nIntermediates at layer 6 aren’t disambiguated interpretations, they’re provisional values further refined by remaining layers. \n\nMany paths can converge to the same final token. Only the emitted token is causally decisive.\n\nChain-of-thought isn’t a hack. It’s the only state mechanism: tokens are the architecture’s persistent memory.\n\nProposing hidden recurrent dynamics just buries state in unauditable vectors while ignoring the real bottleneck, training data for trajectory coherence.\n\nThe math hasn’t changed since Vaswani 2017. \n\nThe field has: rewarding narratives of “interpretable belief states” over the simpler truth of ephemeral partial sums in a fixed approximator. \n\nIncentives > clarity\n\nNon-anthropomorphic Terminology refresh for precision:\n\n• “State tracking” → trajectory coherence across autoregressive steps not single pass layers\n \n• “Belief at layer N” → intermediate residual stream calculation sum reachable by many paths\n\n•“Depth exhaustion” → attention softmax competition under pretraining density\n\n•“Implicit reasoning” → unobservable vector transforms\n\nMechanistic Interpretation unfalsified anthropomorphic assumptions are leading the whole field astray.",
  "source": "Twitter for iPhone",
  "retweetCount": 0,
  "replyCount": 1,
  "likeCount": 0,
  "quoteCount": 5,
  "viewCount": 34,
  "createdAt": "Sun Jun 21 15:54:24 +0000 2026",
  "lang": "en",
  "bookmarkCount": 0,
  "isReply": false,
  "inReplyToId": null,
  "conversationId": "2068724224156397649",
  "displayTextRange": [
    0,
    266
  ],
  "inReplyToUserId": null,
  "inReplyToUsername": null,
  "author": {
    "type": "user",
    "userName": "gerardsans",
    "url": "https://x.com/gerardsans",
    "twitterUrl": "https://twitter.com/gerardsans",
    "id": "9284062",
    "name": "Gerard Sans | Axiom 🇬🇧",
    "isVerified": false,
    "isBlueVerified": true,
    "verifiedType": null,
    "profilePicture": "https://pbs.twimg.com/profile_images/1938955632763105280/aBJaOCJJ_normal.jpg",
    "coverPicture": "https://pbs.twimg.com/profile_banners/9284062/1780071348",
    "description": "",
    "location": "London ☔",
    "followers": 37417,
    "following": 8695,
    "status": "",
    "canDm": true,
    "canMediaTag": false,
    "createdAt": "Sat Oct 06 20:04:48 +0000 2007",
    "entities": {
      "description": {
        "urls": []
      },
      "url": {}
    },
    "fastFollowersCount": 0,
    "favouritesCount": 26680,
    "hasCustomTimelines": true,
    "isTranslator": false,
    "mediaCount": 5212,
    "statusesCount": 54496,
    "withheldInCountries": [],
    "affiliatesHighlightedLabel": {},
    "possiblySensitive": false,
    "pinnedTweetIds": [
      "1741153588959654329"
    ],
    "profile_bio": {
      "description": "Founder Axiom // Forging skills for the new era of AI. GDE in AI, Cloud & Angular. Building London's tech & art nexus @nextai_london. Speaker | MC | Trainer.",
      "entities": {
        "description": {
          "user_mentions": [
            {
              "id_str": "",
              "indices": [
                118,
                132
              ],
              "name": "",
              "screen_name": "nextai_london"
            }
          ]
        },
        "url": {
          "urls": [
            {
              "display_url": "aws.amazon.com/amplify",
              "expanded_url": "http://aws.amazon.com/amplify",
              "indices": [
                0,
                23
              ],
              "url": "https://t.co/ufepRUvlgW"
            }
          ]
        }
      }
    },
    "isAutomated": false,
    "automatedBy": null
  },
  "extendedEntities": {
    "media": [
      {
        "allow_download_status": {
          "allow_download": true
        },
        "display_url": "pic.twitter.com/YtJBPgFXz7",
        "expanded_url": "https://twitter.com/gerardsans/status/2068724224156397649/photo/1",
        "ext_master_playlist_only": [],
        "ext_media_availability": {
          "status": "Available"
        },
        "ext_playlists": [],
        "features": {
          "large": {
            "faces": []
          },
          "orig": {
            "faces": []
          }
        },
        "id_str": "2068724212903141376",
        "indices": [
          267,
          290
        ],
        "media_key": "3_2068724212903141376",
        "media_results": {
          "id": "QXBpTWVkaWFSZXN1bHRzOgwAAQoAARy1lbYIV2AACgACHLWVuKcWcFEAAA==",
          "result": {
            "__typename": "ApiMedia",
            "id": "QXBpTWVkaWE6DAABCgABHLWVtghXYAAKAAIctZW4pxZwUQAA",
            "media_key": "3_2068724212903141376"
          }
        },
        "media_url_https": "https://pbs.twimg.com/media/HLWVtghXYAARx6E.jpg",
        "original_info": {
          "focus_rects": [
            {
              "h": 946,
              "w": 1689,
              "x": 0,
              "y": 0
            },
            {
              "h": 946,
              "w": 946,
              "x": 191,
              "y": 0
            },
            {
              "h": 946,
              "w": 830,
              "x": 249,
              "y": 0
            },
            {
              "h": 946,
              "w": 473,
              "x": 428,
              "y": 0
            },
            {
              "h": 946,
              "w": 2048,
              "x": 0,
              "y": 0
            }
          ],
          "height": 946,
          "width": 2048
        },
        "sizes": {
          "large": {
            "h": 946,
            "w": 2048
          }
        },
        "type": "photo",
        "url": "https://t.co/YtJBPgFXz7"
      }
    ]
  },
  "card": null,
  "place": {},
  "entities": {
    "hashtags": [],
    "symbols": [],
    "timestamps": [],
    "urls": [],
    "user_mentions": []
  },
  "quoted_tweet": null,
  "retweeted_tweet": null,
  "isLimitedReply": false,
  "communityInfo": null,
  "article": null
}