The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We’ll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough
00:27:02 starting the implementation
00:28:35 counting consecutive pairs, finding most common pair
00:30:36 merging the most common pair
00:34:58 training the tokenizer: adding the while loop, compression ratio
00:39:20 tokenizer/LLM diagram: it is a completely separate stage
00:42:47 decoding tokens to strings
00:48:21 encoding strings to tokens
00:57:36 regex patterns to force splits across categories
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex
01:14:59 GPT-2 released by OpenAI walkthrough
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary
01:43:27 how to set vocabulary set? revisiting transformer
01:48:11 training new tokens, example of prompt compression
01:49:58 multimodal [image, video, audio] tokenization with vector quantization
01:51:41 revisiting and explaining the quirks of LLM tokenization
02:10:20 final recommendations
02:12:50 ??? :)
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you’re getting stuck are in the minbpe code
Links:
- Google colab for the video:
- GitHub repo for the video: minBPE
- Playlist of the whole Zero to Hero series so far:
- our Discord channel:
- my Twitter:
Supplementary links:
- tiktokenizer
- tiktoken from OpenAI:
- sentencepiece from Google
1 view
112
28
1 month ago 00:04:10 1
Tina Turner - The Best (Official Music Video)
1 month ago 00:08:09 2
Free Items on Roblox 2024! HURRY!
1 month ago 00:04:01 1
Dr Funk - Roadrunner (Drop D) Marcus Miller Inspired
1 month ago 00:02:25 1
Night Raider Game Trailer - gameplay
1 month ago 00:12:42 1
Best Tools for SMMA Outreach 2024 | Get Clients Now
1 month ago 00:01:00 1
😱Homeless Talent - Amazing Street Singing🔥Boyz II Men - End Of The Road
1 month ago 00:00:59 1
From Pipes to Boat: Plastic Pipes and Foam Turned Nautical! 🌊🚣 #Shorts
1 month ago 00:00:00 1
🔴 LIVE Прославление | Церковь Божия Music
1 month ago 00:00:27 1
He’s the reason that the NBA has rules longer than the constitution 😩
1 month ago 00:11:43 1
Secret Chemtrail Pilot Speaks, on Dec 8, 2014
1 month ago 00:08:02 16
A new Era in Photoshop! AI is changing everything
1 month ago 00:30:19 1
Here’s how AWFUL Oakland, California is Today. Is it Unfixable?
1 month ago 00:10:55 1
Easy Planter made from 1x6 - Best Woodworking Projects for Spring Part 2!
1 month ago 00:08:34 1
How to Get a Rounder Butt FAST (ft. Bret Contreras)
1 month ago 01:08:20 1
5D Подкаст: «10 лет BIM: прогресс и достижения. Цели и задачи внедрения BIM в России»
1 month ago 00:04:22 1
Darren Hayes - Insatiable (Official Music Video)
1 month ago 00:16:52 1
SOLO GIRL’s bushcraft in a dugout & CAMP sauna and fishing ASMR
1 month ago 00:02:49 1
Missouri Star Show & Tell: Michaels quilts and their names!
1 month ago 00:05:24 1
twenty one pilots - Chlorine (Official Video)
1 month ago 01:43:28 1
Night Slashers-X Wide Screen Ash + cheats | OpenBoR Gamer
1 month ago 00:26:57 1
Egg Drop From Space
1 month ago 00:05:38 1
Burning Man 2020: Dreaming of Home
1 month ago 00:03:26 1
Blanke & Grant - Heavy Heart (Lyrics) feat. your friend polly
1 month ago 00:20:53 1
I Can’t Believe These are Real - Reacting to Ridiculous PC Classifieds