Viewing enriched Twitter post
They are hinting at that, sure. But they're testing on OPT, as in most of those Hype-Aware Quantization papers Why? OPT's FF layers use ReLU. It sacrifices perplexity but makes activations sparse. I'm skeptical it'll work for SwiGLU in LLaMA… without retrain. (paper:MoEfication) https://t.co/LbkvP5LW5i
{
"media": [
{
"url": "https://pbs.twimg.com/media/F3tkVcDXoAAcB8B.jpg",
"type": "photo",
"original_url": "https://pbs.twimg.com/media/F3tkVcDXoAAcB8B.jpg",
"format_converted_from_list": true
},
{
"url": "https://pbs.twimg.com/media/F3tkim1WcAExARf.jpg",
"type": "photo",
"original_url": "https://pbs.twimg.com/media/F3tkim1WcAExARf.jpg",
"format_converted_from_list": true
}
],
"conversion_date": "2025-08-13T00:32:48.703667",
"format_converted": true,
"original_structure": "had_media_only"
}{
"user": {
"created_at": "2010-09-18T13:32:22.000Z",
"default_profile_image": false,
"description": "Ours is the age of unaligned utilitarians. Other problems are relatively unimportant, but sometimes I tweet about them anyway.\n(кто/кого)",
"fast_followers_count": 0,
"favourites_count": 14778,
"followers_count": 1652,
"friends_count": 686,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 24,
"location": "",
"media_count": 1112,
"name": "Teortaxes",
"normal_followers_count": 1652,
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/192201556/1682742940",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1652169745037242368/KRPTShbG_normal.jpg",
"screen_name": "teortaxesTex",
"statuses_count": 12359,
"translator_type": "none",
"verified": false,
"withheld_in_countries": [],
"id_str": "192201556"
},
"id": "1692057168659271988",
"conversation_id": "1692057168659271988",
"full_text": "They are hinting at that, sure.\nBut they're testing on OPT, as in most of those Hype-Aware Quantization papers\nWhy?\nOPT's FF layers use ReLU. It sacrifices perplexity but makes activations sparse.\nI'm skeptical it'll work for SwiGLU in LLaMA… without retrain.\n(paper:MoEfication) https://t.co/LbkvP5LW5i",
"reply_count": 2,
"retweet_count": 1,
"favorite_count": 24,
"hashtags": [],
"symbols": [],
"user_mentions": [],
"urls": [],
"media": [
{
"media_url": "https://pbs.twimg.com/media/F3tkVcDXoAAcB8B.jpg",
"type": "photo"
},
{
"media_url": "https://pbs.twimg.com/media/F3tkim1WcAExARf.jpg",
"type": "photo"
}
],
"url": "https://twitter.com/teortaxesTex/status/1692057168659271988",
"created_at": "2023-08-17T06:14:00.000Z",
"#sort_index": "1695781238999568600",
"view_count": 32546,
"quote_count": 1,
"is_quote_tweet": true,
"is_retweet": false,
"is_pinned": false,
"is_truncated": false,
"startUrl": "https://twitter.com/teortaxestex/status/1695781238999568624"
}