🐦 Twitter Post Details

Viewing enriched Twitter post

@DrJimFan

2024 will be the year of videos. While robotics & embodied agents are just getting started, I think video AI will meet its breakthrough moments in the next 12 months. There are two parts: I/O "I": video input. GPT-4V's video understanding is still quite primitive, as it treats video as a sequence of discrete images. Sure, it kind of works, but very inefficiently. Video is a spatiotemporal volume of pixels. It is extremely high-dimensional yet redundant. In ECCV 2020, I proposed a method called RubiksNet that simply shifts around the video pixels like a Rubik's Cube along 3 axes, and then apply MLPs in between. No 3D convolution, no transformers, a bit similar to MLP-Mixer in spirit. It works surprisingly well and runs fast with my custom CUDA kernels. https://t.co/CQU8D7TZgx, w/ Shyamal Buch, @guanzhi_wang, @yukez, @drfeifei, etc. Are Transformers all you need? If yes, what's the smartest way to reduce the information redundancy? What should be the learning objective? Next frame prediction is an obvious analogy to next word prediction, but is it optimal? How to interleave with language? How to steer video learning for robotics and embodied AI? No consensus at all in the community. Part 2, "O": video output. In 2023, we have seen a wave of text-to-video synthesis: WALT (Google, cover video below), EmuVideo (Meta), Align Your Latents (NVIDIA), @pika_labs, and many more. Too many to count. Yet most of the generated snippets are still very short. I see them as video AI's "System 1" - "unconscious", local pixel movements. In 2024, I'm confident that we will see video generation with high resolution and long-term coherence. That would require much more "thinking", i.e. System 2 reasoning and long-horizon planning.

View on Twitter

📊 Media Metadata

{
  "media": [
    {
      "url": "https://crmoxkoizveukayfjuyo.supabase.co/storage/v1/object/public/media/posts/1740788207967736047/media_0.mp4",
      "type": "video",
      "original_url": "https://pbs.twimg.com/ext_tw_video_thumb/1740775191805116416/pu/img/WcqIseM65Rh-Bqlk.jpg",
      "recovered_from_supabase": true
    }
  ],
  "conversion_date": "2025-08-13T00:27:56.017872",
  "format_converted": true,
  "original_structure": "had_media_only"
}

🔧 Raw API Response

{
  "user": {
    "created_at": "2012-12-12T22:11:27.000Z",
    "default_profile_image": false,
    "description": "@NVIDIA Senior Research Scientist & Lead of AI Agents. Creating foundation models for Agents, Robotics, Gaming. @Stanford Ph.D. @OpenAI's first intern.",
    "fast_followers_count": 0,
    "favourites_count": 6923,
    "followers_count": 204531,
    "friends_count": 2910,
    "has_custom_timelines": true,
    "is_translator": false,
    "listed_count": 3645,
    "location": "Contact →",
    "media_count": 730,
    "name": "Jim Fan",
    "normal_followers_count": 204531,
    "possibly_sensitive": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/1007413134/1672408318",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/1554922493101559808/SYSZhbcd_normal.jpg",
    "screen_name": "DrJimFan",
    "statuses_count": 3418,
    "translator_type": "none",
    "url": "https://t.co/H4rXo4Ei8X",
    "verified": true,
    "withheld_in_countries": [],
    "id_str": "1007413134"
  },
  "id": "1740788207967736047",
  "conversation_id": "1740788207967736047",
  "full_text": "2024 will be the year of videos. While robotics & embodied agents are just getting started, I think video AI will meet its breakthrough moments in the next 12 months. There are two parts: I/O\n\n\"I\": video input. GPT-4V's video understanding is still quite primitive, as it treats video as a sequence of discrete images. Sure, it kind of works, but very inefficiently. Video is a spatiotemporal volume of pixels. It is extremely high-dimensional yet redundant.   \n\nIn ECCV 2020, I proposed a method called RubiksNet that simply shifts around the video pixels like a Rubik's Cube along 3 axes, and then apply MLPs in between. No 3D convolution, no transformers, a bit similar to MLP-Mixer in spirit. It works surprisingly well and runs fast with my custom CUDA kernels.\n\nhttps://t.co/CQU8D7TZgx, w/ Shyamal Buch, @guanzhi_wang, @yukez, @drfeifei, etc.\n\nAre Transformers all you need?\nIf yes, what's the smartest way to reduce the information redundancy? \nWhat should be the learning objective? \nNext frame prediction is an obvious analogy to next word prediction, but is it optimal? \nHow to interleave with language? \nHow to steer video learning for robotics and embodied AI? \n\nNo consensus at all in the community. \n\nPart 2, \"O\": video output. In 2023, we have seen a wave of text-to-video synthesis: WALT (Google, cover video below), EmuVideo (Meta), Align Your Latents (NVIDIA), @pika_labs, and many more. Too many to count. Yet most of the generated snippets are still very short. I see them as video AI's \"System 1\" - \"unconscious\", local pixel movements.\n\nIn 2024, I'm confident that we will see video generation with high resolution and long-term coherence. That would require much more \"thinking\", i.e. System 2 reasoning and long-horizon planning.",
  "reply_count": 30,
  "retweet_count": 174,
  "favorite_count": 698,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [],
  "urls": [],
  "media": [
    {
      "media_url": "https://pbs.twimg.com/ext_tw_video_thumb/1740775191805116416/pu/img/WcqIseM65Rh-Bqlk.jpg",
      "type": "video",
      "video_url": "https://video.twimg.com/ext_tw_video/1740775191805116416/pu/vid/avc1/896x512/KH9EmfC9IPRosDIb.mp4?tag=12"
    }
  ],
  "url": "https://twitter.com/DrJimFan/status/1740788207967736047",
  "created_at": "2023-12-29T17:33:45.000Z",
  "#sort_index": "1758537427139056000",
  "view_count": 122699,
  "quote_count": 14,
  "is_quote_tweet": false,
  "is_retweet": false,
  "is_pinned": false,
  "is_truncated": true,
  "startUrl": "https://twitter.com/drjimfan/status/1758537427139056055"
}