🐦 Twitter Post Details

Viewing enriched Twitter post

@winglian

In Axolotl, we’ve addressed this with merge-aware LoRA training: we merge the LoRA into the base weights and cast to BF16 inside the forward pass, so the model trains on exactly the weights vLLM serves. Gradients pass straight through the rounding to the LoRA weights, so they keep learning even when a single step is too small to change a BF16 weight. While there are still gaps in this method (support for various distributed parallelisms, ZeRO-3, etc), we hope we can help ensure that you aren't left wondering why what you deployed is underperforming.

View on Twitter

📊 Media Metadata

{
  "score": 0.42,
  "score_components": {
    "author": 0.09,
    "engagement": 0.0,
    "quality": 0.12,
    "source": 0.135,
    "nlp": 0.05,
    "recency": 0.025
  },
  "scored_at": "2026-10-27T23:02:37.317381",
  "import_source": "api_import",
  "source_tagged_at": "2026-10-27T23:02:37.317393",
  "enriched": true,
  "enriched_at": "2026-10-27T23:02:37.317395"
}

🔧 Raw API Response

{
  "type": "tweet",
  "id": "2115171728838029603",
  "url": "https://x.com/winglian/status/2115171728838029603",
  "twitterUrl": "https://twitter.com/winglian/status/2115171728838029603",
  "text": "In Axolotl, we’ve addressed this with merge-aware LoRA training: we merge the LoRA into the base weights and cast to BF16 inside the forward pass, so the model trains on exactly the weights vLLM serves. Gradients pass straight through the rounding to the LoRA weights, so they keep learning even when a single step is too small to change a BF16 weight. While there are still gaps in this method (support for various distributed parallelisms, ZeRO-3, etc), we hope we can help ensure that you aren't left wondering why what you deployed is underperforming.",
  "source": "",
  "retweetCount": 0,
  "replyCount": 1,
  "likeCount": 1,
  "quoteCount": 0,
  "viewCount": 202,
  "createdAt": "Tue Oct 27 20:00:12 +0000 2026",
  "lang": "en",
  "bookmarkCount": 0,
  "isReply": true,
  "inReplyToId": "2115171726279733748",
  "conversationId": "2115171721913196832",
  "displayTextRange": [
    0,
    276
  ],
  "inReplyToUserId": "24802509",
  "inReplyToUsername": "winglian",
  "author": {
    "type": "user",
    "userName": "winglian",
    "url": "https://x.com/winglian",
    "twitterUrl": "https://twitter.com/winglian",
    "id": "24802509",
    "name": "Wing Lian (caseus)",
    "isVerified": false,
    "isBlueVerified": true,
    "verifiedType": null,
    "profilePicture": "https://pbs.twimg.com/profile_images/1709990043664715778/FZmSZJaK_normal.jpg",
    "coverPicture": "https://pbs.twimg.com/profile_banners/24802509/1403302292",
    "description": "",
    "location": "Annapolis, MD",
    "followers": 11034,
    "following": 1896,
    "status": "",
    "canDm": true,
    "canMediaTag": true,
    "createdAt": "Tue Mar 17 00:12:21 +0000 2009",
    "entities": {
      "description": {
        "urls": []
      },
      "url": {}
    },
    "fastFollowersCount": 0,
    "favouritesCount": 3442,
    "hasCustomTimelines": true,
    "isTranslator": false,
    "mediaCount": 171,
    "statusesCount": 2978,
    "withheldInCountries": [],
    "affiliatesHighlightedLabel": {},
    "possiblySensitive": false,
    "pinnedTweetIds": [],
    "profile_bio": {
      "description": "@axolotl_ai OSS maintainer. Axolotl AI founder. AI/ML tinkerer. Building tools for everyone.",
      "entities": {
        "description": {
          "user_mentions": [
            {
              "id_str": "",
              "indices": [
                0,
                11
              ],
              "name": "",
              "screen_name": "axolotl_ai"
            }
          ]
        },
        "url": {
          "urls": [
            {
              "display_url": "github.com/winglian/",
              "expanded_url": "https://github.com/winglian/",
              "indices": [
                0,
                23
              ],
              "url": "https://t.co/Ym8RB0dslS"
            }
          ]
        }
      }
    },
    "isAutomated": false,
    "automatedBy": null,
    "protected": false
  },
  "extendedEntities": {},
  "card": null,
  "place": {},
  "entities": {
    "hashtags": [],
    "symbols": [],
    "urls": [],
    "user_mentions": []
  },
  "quoted_tweet": null,
  "retweeted_tweet": null,
  "isLimitedReply": false,
  "communityInfo": null,
  "article": null
}