@jeremyphoward
Anyone know how to derive this '1.5x' communication overhead between FSDP vs DDP (from the FSDP paper)? https://t.co/73qe4XtRa7
Viewing enriched Twitter post
Anyone know how to derive this '1.5x' communication overhead between FSDP vs DDP (from the FSDP paper)? https://t.co/73qe4XtRa7
{
"user": {
"created_at": "2010-08-06T04:58:18.000Z",
"default_profile_image": false,
"description": "🇦🇺 Co-founder: @FastDotAI ;\nHon Professor: @UQSchoolITEE ;\nProudly GPU poor",
"fast_followers_count": 0,
"favourites_count": 9608,
"followers_count": 197617,
"friends_count": 4242,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 4143,
"location": "Brisbane/Queensland, Australia",
"media_count": 2120,
"name": "Jeremy Howard",
"normal_followers_count": 197617,
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/175282603/1594860705",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1279600070145437696/eocLhSLu_normal.jpg",
"screen_name": "jeremyphoward",
"statuses_count": 51168,
"translator_type": "none",
"url": "https://t.co/4goRXTRT37",
"verified": false,
"withheld_in_countries": [],
"id_str": "175282603"
},
"id": "1712972009142132937",
"conversation_id": "1712972009142132937",
"full_text": "Anyone know how to derive this '1.5x' communication overhead between FSDP vs DDP (from the FSDP paper)? https://t.co/73qe4XtRa7",
"reply_count": 2,
"retweet_count": 2,
"favorite_count": 53,
"hashtags": [],
"symbols": [],
"user_mentions": [],
"urls": [],
"media": [
{
"media_url": "https://pbs.twimg.com/media/F8Wy6F2asAAZyKS.png",
"type": "photo"
}
],
"url": "https://twitter.com/jeremyphoward/status/1712972009142132937",
"created_at": "2023-10-13T23:22:07.000Z",
"#sort_index": "1712972009142132937",
"view_count": 24035,
"quote_count": 0,
"is_quote_tweet": false,
"is_retweet": false,
"is_pinned": false,
"is_truncated": false,
"startUrl": "https://twitter.com/jeremyphoward/status/1712972009142132937"
}