🐦 Twitter Post Details

Viewing enriched Twitter post

@omarsar0

Instruction Tuning the Largest Pretrained Retrieval-Augmented LLM This exciting new paper from NVIDIA introduces Retro 48B, the largest LLM pretrained with retrieval. Continues pretraining a 43B parameter GPT model on additional 100B tokens by retrieving from 1.2T tokens (using the Retro augmentation method). The Retro 48B model shows significant perplexity improvement over its GPT 43B counterpart. Scaling the Retro model to 48B means it can be instruction-tuned more effectively. This work applies instruction tuning to Retro 48B and demonstrates significant improvement (+7%) over the instruction-tuned GPT on zero-shot question-answering tasks. The important insight from this work is the potential benefit attained from pretraining with retrieval. Results highlight the promising direction to obtain a better GPT decoder for QA through continued pretraining with retrieval before instruction tuning. https://t.co/EORkgCXsz2

View on Twitter

📊 Media Metadata

{
  "media": [
    {
      "id": "",
      "type": "photo",
      "url": null,
      "media_url": "https://pbs.twimg.com/media/F8PhlRNXUAEtgN2.jpg",
      "media_url_https": null,
      "display_url": null,
      "expanded_url": null
    }
  ],
  "nlp": {
    "sentiment": "positive",
    "processed_at": "2025-08-06T12:45:26.577711"
  },
  "original_structure": "had_media_only"
}

🔧 Raw API Response

{
  "user": {
    "created_at": "2015-09-04T12:59:26.000Z",
    "default_profile_image": false,
    "description": "I share insights & advances in LLMs • Building @dair_ai • Prev: Meta AI, Galactica LLM, PapersWithCode, Elastic, PhD • Author of Prompting Guide (1.8M users)",
    "fast_followers_count": 0,
    "favourites_count": 23315,
    "followers_count": 162181,
    "friends_count": 427,
    "has_custom_timelines": true,
    "is_translator": false,
    "listed_count": 3000,
    "location": "",
    "media_count": 1703,
    "name": "elvis",
    "normal_followers_count": 162181,
    "possibly_sensitive": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/3448284313/1565974901",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/939313677647282181/vZjFWtAn_normal.jpg",
    "screen_name": "omarsar0",
    "statuses_count": 9511,
    "translator_type": "regular",
    "url": "https://t.co/o4KzoHf52W",
    "verified": false,
    "withheld_in_countries": [],
    "id_str": "3448284313"
  },
  "id": "1712466049428521433",
  "conversation_id": "1712466049428521433",
  "full_text": "Instruction Tuning the Largest Pretrained Retrieval-Augmented LLM\n\nThis exciting new paper from NVIDIA introduces Retro 48B, the largest LLM pretrained with retrieval.\n\nContinues pretraining a 43B parameter GPT model on additional 100B tokens by retrieving from 1.2T tokens (using the Retro augmentation method). \n\nThe Retro 48B model shows significant perplexity improvement over its GPT 43B counterpart. \n\nScaling the Retro model to 48B means it can be instruction-tuned more effectively. This work applies instruction tuning to Retro 48B and demonstrates significant improvement (+7%) over the instruction-tuned GPT on zero-shot question-answering tasks. \n\nThe important insight from this work is the potential benefit attained from pretraining with retrieval. Results highlight the promising direction to obtain a better GPT decoder for QA through continued pretraining with retrieval before instruction tuning.\n\nhttps://t.co/EORkgCXsz2",
  "reply_count": 3,
  "retweet_count": 89,
  "favorite_count": 378,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [],
  "urls": [],
  "media": [
    {
      "media_url": "https://pbs.twimg.com/media/F8PhlRNXUAEtgN2.jpg",
      "type": "photo"
    }
  ],
  "url": "https://twitter.com/omarsar0/status/1712466049428521433",
  "created_at": "2023-10-12T13:51:36.000Z",
  "#sort_index": "1712466049428521433",
  "view_count": 86234,
  "quote_count": 4,
  "is_quote_tweet": false,
  "is_retweet": false,
  "is_pinned": false,
  "is_truncated": true,
  "startUrl": "https://twitter.com/omarsar0/status/1712466049428521433"
}