🐦 Twitter Post Details

Viewing enriched Twitter post

@omarsar0

Transformers Struggle to Learn to Search Finds that transformer-based LLMs struggle to perform search robustly. Suggests that given the right training distribution, the transformer can learn to search. Also reports that performing search in-context exploration (i.e., chain-of-thought) doesn't resolve the transformer's inability to learn to search on larger graphs. The authors mentioned that it might be possible to improve search in transformers with techniques like curriculum learning and looped transformers. I think this is a nice research paper with huge implications for understanding better how these transformer models perform "reasoning" and other important capabilities.

Media 1
View on Twitter

📊 Media Metadata

{
  "data": [
    {
      "id": "",
      "type": "photo",
      "url": null,
      "media_url": "https://pbs.twimg.com/media/GeXWxAaXwAAjPe-.png",
      "media_url_https": null,
      "display_url": null,
      "expanded_url": null
    }
  ],
  "score": 1.0,
  "scored_at": "2025-08-09T13:46:07.556810",
  "import_source": "network_archive_import",
  "media": [
    {
      "type": "photo",
      "url": "https://crmoxkoizveukayfjuyo.supabase.co/storage/v1/object/public/media/posts/1866136135363092601/media_0.png?",
      "filename": "media_0.png"
    }
  ],
  "reprocessed_at": "2025-08-12T15:27:23.264424",
  "reprocessed_reason": "missing_media_array"
}

🔧 Raw API Response

{
  "user": {
    "created_at": "2015-09-04T12:59:26.000Z",
    "default_profile_image": false,
    "description": "Building with AI Agents @dair_ai • Prev: Meta AI, Elastic, Galactica LLM, PhD • I also teach how to build with LLMs, RAG & AI Agents ⬇️",
    "fast_followers_count": 0,
    "favourites_count": 27931,
    "followers_count": 216714,
    "friends_count": 532,
    "has_custom_timelines": true,
    "is_translator": false,
    "listed_count": 3688,
    "location": "",
    "media_count": 2656,
    "name": "elvis",
    "normal_followers_count": 216714,
    "possibly_sensitive": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/3448284313/1565974901",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/939313677647282181/vZjFWtAn_normal.jpg",
    "screen_name": "omarsar0",
    "statuses_count": 12439,
    "translator_type": "regular",
    "url": "https://t.co/JBU5beHQNs",
    "verified": true,
    "withheld_in_countries": [],
    "id_str": "3448284313"
  },
  "id": "1866136135363092601",
  "conversation_id": "1866136135363092601",
  "full_text": "Transformers Struggle to Learn to Search\n\nFinds that transformer-based LLMs struggle to perform search robustly. \n\nSuggests that given the right training distribution, the transformer can learn to search. \n\nAlso reports that performing search in-context exploration (i.e., chain-of-thought) doesn't resolve the transformer's inability to learn to search on larger graphs.\n\nThe authors mentioned that it might be possible to improve search in transformers with techniques like curriculum learning and looped transformers. \n\nI think this is a nice research paper with huge implications for understanding better how these transformer models perform \"reasoning\" and other important capabilities.",
  "reply_count": 4,
  "retweet_count": 53,
  "favorite_count": 273,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [],
  "urls": [],
  "media": [
    {
      "media_url": "https://pbs.twimg.com/media/GeXWxAaXwAAjPe-.png",
      "type": "photo"
    }
  ],
  "url": "https://twitter.com/omarsar0/status/1866136135363092601",
  "created_at": "2024-12-09T15:01:40.000Z",
  "#sort_index": "1866136135363092601",
  "view_count": 18561,
  "quote_count": 2,
  "is_quote_tweet": false,
  "is_retweet": false,
  "is_pinned": false,
  "is_truncated": true,
  "startUrl": "https://x.com/omarsar0/status/1866136135363092601"
}