šŸ¦ Twitter Post Details

Viewing enriched Twitter post

@_lewtun

Several @huggingface users have reported loss divergences when fine-tuning Mistral 7B w/out LoRA šŸ˜± Here's a simple script that works well with TRL's SFTTrainer & DeepSpeed ZeRO-3: https://t.co/MbjtkRQU1W (Trained on a subset of UltraChat) https://t.co/Hf3Zd05DbV

Media 1
View on Twitter

šŸ“Š Media Metadata

{
  "media": [
    {
      "url": "https://crmoxkoizveukayfjuyo.supabase.co/storage/v1/object/public/media/posts/1709897775222079881/media_0.png",
      "type": "photo",
      "original_url": "https://pbs.twimg.com/media/F7rC1EeXUAAzWv3.png",
      "recovered_from_supabase": true
    }
  ],
  "conversion_date": "2025-08-13T00:28:14.170221",
  "format_converted": true,
  "original_structure": "had_media_only"
}

šŸ”§ Raw API Response

{
  "user": {
    "created_at": "2018-08-14T22:21:16.000Z",
    "default_profile_image": false,
    "description": "šŸ¤— Putting hugs in RLHF @huggingface\nšŸ“– Co-author of \"NLP with Transformers\" book\nšŸ’„ Ex-particle physicist\nšŸ¤˜ Occasional guitarist\nšŸ‡¦šŸ‡ŗ in šŸ‡ØšŸ‡­",
    "fast_followers_count": 0,
    "favourites_count": 7369,
    "followers_count": 6101,
    "friends_count": 488,
    "has_custom_timelines": true,
    "is_translator": false,
    "listed_count": 163,
    "location": "Berne, Switzerland",
    "media_count": 537,
    "name": "Lewis Tunstall",
    "normal_followers_count": 6101,
    "possibly_sensitive": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/1029493180704714753/1655469477",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/1097405296543236096/gS2C7RIq_normal.jpg",
    "screen_name": "_lewtun",
    "statuses_count": 2489,
    "translator_type": "none",
    "url": "https://t.co/F3W4xU7x2Z",
    "verified": false,
    "withheld_in_countries": [],
    "id_str": "1029493180704714753"
  },
  "id": "1709897775222079881",
  "conversation_id": "1709897775222079881",
  "full_text": "Several @huggingface users have reported loss divergences when fine-tuning Mistral 7B w/out LoRA šŸ˜±\n\nHere's a simple script that works well with TRL's SFTTrainer & DeepSpeed ZeRO-3: https://t.co/MbjtkRQU1W\n \n(Trained on a subset of UltraChat) https://t.co/Hf3Zd05DbV",
  "reply_count": 3,
  "retweet_count": 16,
  "favorite_count": 106,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [
    {
      "id_str": "778764142412984320",
      "name": "Hugging Face",
      "screen_name": "huggingface",
      "profile": "https://twitter.com/huggingface"
    }
  ],
  "urls": [
    {
      "url": "https://t.co/MbjtkRQU1W",
      "expanded_url": "https://gist.github.com/lewtun/b9d46e00292d9ecdd6fd9628d53c2814#file-sft_trainer-py",
      "display_url": "gist.github.com/lewtun/b9d46e0ā€¦"
    }
  ],
  "media": [
    {
      "media_url": "https://pbs.twimg.com/media/F7rC1EeXUAAzWv3.png",
      "type": "photo"
    }
  ],
  "url": "https://twitter.com/_lewtun/status/1709897775222079881",
  "created_at": "2023-10-05T11:46:12.000Z",
  "#sort_index": "1709897775222079881",
  "view_count": 22486,
  "quote_count": 0,
  "is_quote_tweet": false,
  "is_retweet": false,
  "is_pinned": false,
  "is_truncated": false,
  "startUrl": "https://twitter.com/_lewtun/status/1709897775222079881"
}