🐦 Twitter Post Details

Viewing enriched Twitter post

@_akhaliq

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention paper page: https://t.co/nXtYagDaYv Most of the existing multi-modal models, hindered by their incapacity to adeptly manage interleaved image-and-text inputs in multi-image, multi-round dialogues, face substantial constraints in resource allocation for training and data accessibility, impacting their adaptability and scalability across varied interaction realms. To address this, we present the DeepSpeed-VisualChat framework, designed to optimize Large Language Models (LLMs) by incorporating multi-modal capabilities, with a focus on enhancing the proficiency of Large Vision and Language Models in handling interleaved inputs. Our framework is notable for (1) its open-source support for multi-round and multi-image dialogues, (2) introducing an innovative multi-modal causal attention mechanism, and (3) utilizing data blending techniques on existing datasets to assure seamless interactions in multi-round, multi-image conversations. Compared to existing frameworks, DeepSpeed-VisualChat shows superior scalability up to 70B parameter language model size, representing a significant advancement in multi-modal language models and setting a solid foundation for future explorations.

🔧 Raw API Response

{
  "user": {
    "created_at": "2014-04-27T00:20:12.000Z",
    "default_profile_image": false,
    "description": "AI research paper tweets, ML @Gradio (acq. by @HuggingFace 🤗)\n\ndm for promo",
    "fast_followers_count": 0,
    "favourites_count": 26631,
    "followers_count": 237606,
    "friends_count": 1888,
    "has_custom_timelines": true,
    "is_translator": false,
    "listed_count": 3170,
    "location": "subscribe → ",
    "media_count": 13881,
    "name": "AK",
    "normal_followers_count": 237606,
    "possibly_sensitive": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/2465283662/1610997549",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/1451191636810092553/kpM5Fe12_normal.jpg",
    "screen_name": "_akhaliq",
    "statuses_count": 21880,
    "translator_type": "none",
    "url": "https://t.co/TbGnXZJwEc",
    "verified": false,
    "withheld_in_countries": [],
    "id_str": "2465283662"
  },
  "id": "1706562634902884543",
  "conversation_id": "1706562634902884543",
  "full_text": "DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention\n\npaper page: https://t.co/nXtYagDaYv\n\nMost of the existing multi-modal models, hindered by their incapacity to adeptly manage interleaved image-and-text inputs in multi-image, multi-round dialogues, face substantial constraints in resource allocation for training and data accessibility, impacting their adaptability and scalability across varied interaction realms. To address this, we present the DeepSpeed-VisualChat framework, designed to optimize Large Language Models (LLMs) by incorporating multi-modal capabilities, with a focus on enhancing the proficiency of Large Vision and Language Models in handling interleaved inputs. Our framework is notable for (1) its open-source support for multi-round and multi-image dialogues, (2) introducing an innovative multi-modal causal attention mechanism, and (3) utilizing data blending techniques on existing datasets to assure seamless interactions in multi-round, multi-image conversations. Compared to existing frameworks, DeepSpeed-VisualChat shows superior scalability up to 70B parameter language model size, representing a significant advancement in multi-modal language models and setting a solid foundation for future explorations.",
  "reply_count": 0,
  "retweet_count": 45,
  "favorite_count": 142,
  "hashtags": [],
  "symbols": [],
  "user_mentions": [],
  "urls": [
    {
      "url": "https://t.co/3J8Z05XrSu",
      "expanded_url": "https://huggingface.co/papers/2309.14327",
      "display_url": "huggingface.co/papers/2309.14…"
    }
  ],
  "media": [
    {
      "media_url": "https://pbs.twimg.com/media/F67tqRyX0AAeR3r.jpg",
      "type": "photo"
    }
  ],
  "url": "https://twitter.com/_akhaliq/status/1706562634902884543",
  "created_at": "2023-09-26T06:53:33.000Z",
  "#sort_index": "1706562634902884543",
  "view_count": 30588,
  "quote_count": 5,
  "is_quote_tweet": false,
  "is_retweet": false,
  "is_pinned": false,
  "is_truncated": true,
  "startUrl": "https://twitter.com/_akhaliq/status/1706562634902884543"
}