Viewing enriched Twitter post
New (2h13m 😅) lecture: "Let's build the GPT Tokenizer" Tokenizers are a completely separate stage of the LLM pipeline: they have their own training set, training algorithm (Byte Pair Encoding), and after training implement two functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI.
{
"user": {
"created_at": "2009-04-21T06:49:15.000Z",
"default_profile_image": false,
"description": "🧑🍳. Previously Director of AI @ Tesla, founding team @ OpenAI, CS231n/PhD @ Stanford. I like to train large deep neural nets 🧠🤖💥",
"fast_followers_count": 0,
"favourites_count": 10640,
"followers_count": 941229,
"friends_count": 893,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 12301,
"location": "Stanford",
"media_count": 666,
"name": "Andrej Karpathy",
"normal_followers_count": 941229,
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/33836629/1407117611",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1296667294148382721/9Pr6XrPB_normal.jpg",
"screen_name": "karpathy",
"statuses_count": 8590,
"translator_type": "none",
"url": "https://t.co/0EcFthjJXM",
"verified": true,
"withheld_in_countries": [],
"id_str": "33836629"
},
"id": "1759996549109776702",
"conversation_id": "1759996549109776702",
"full_text": "New (2h13m 😅) lecture: \"Let's build the GPT Tokenizer\"\n\nTokenizers are a completely separate stage of the LLM pipeline: they have their own training set, training algorithm (Byte Pair Encoding), and after training implement two functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI.",
"reply_count": 382,
"retweet_count": 1950,
"favorite_count": 14179,
"hashtags": [],
"symbols": [],
"user_mentions": [],
"urls": [],
"media": [
{
"media_url": "https://pbs.twimg.com/media/GGzDVPMasAAtLs0.jpg",
"type": "photo"
}
],
"url": "https://twitter.com/karpathy/status/1759996549109776702",
"created_at": "2024-02-20T17:40:51.000Z",
"#sort_index": "1759996549109776702",
"view_count": 1473528,
"quote_count": 337,
"is_quote_tweet": false,
"is_retweet": false,
"is_pinned": false,
"is_truncated": true,
"startUrl": "https://twitter.com/karpathy/status/1759996549109776702"
}