2. The spelled-out intro to language modeling: building makemore
We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).
Links:
- makemore on github:
- jupyter notebook I built in this video:
Useful links for practice:
- Python Numpy tutorial from CS231n . We use instead of in this video. Their design (e.g. broadcasting, data types, etc.) is so similar that practicing one is basically practicing the other, just be careful with some of the APIs - how various functions are named, what arguments they take, etc. - these details can vary.
- PyTorch tutorial on Tensor
- Another PyTorch intro to Tensor
Exercises:
E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?
E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?
E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?
E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of in favor of simply indexing into rows of W?
E05: look up and use instead. You should achieve the same result. Can you think of why we’d prefer to use instead?
E06: meta-exercise! Think of a fun/interesting exercise and complete it.
Chapters:
00:00:00 intro
00:03:03 reading and exploring the dataset
00:06:24 exploring the bigrams in the dataset
00:09:24 counting bigrams in a python dictionary
00:12:45 counting bigrams in a 2D torch tensor (“training the model“)
00:18:19 visualizing the bigram tensor
00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token
00:24:02 sampling from the model
00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting
00:50:14 loss function (the negative log likelihood of the data under our model)
01:00:50 model smoothing with fake counts
01:02:57 PART 2: the neural network approach: intro
01:05:26 creating the bigram dataset for the neural net
01:10:01 feeding integers into neural nets? one-hot encodings
01:13:53 the “neural net“: one linear layer of neurons implemented with matrix multiplication
01:18:46 transforming neural net outputs into probabilities: the softmax
01:26:17 summary, preview to next steps, reference to micrograd
01:35:49 vectorized loss
01:38:36 backward and update, in PyTorch
01:42:55 putting everything together
01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer’s weight matrix
01:50:18 note 2: model smoothing as regularization loss
01:54:31 sampling from the neural net
01:56:16 conclusion
6 views
7383
3379
1 day ago 00:04:35 1
Last of Us Season 2 Spoof Trailer - Virtual Production
2 days ago 00:28:58 1
Успейте сделать!❄️3 чудесные ИДЕИ декора на Новый Год🎄Просто, быстро и оригинально!
2 days ago 00:00:00 1
. 2 HEART OF CHERNOBYL➤ИДУ СВОЕЙ ДОРОГОЙ➤СТАЛКЕР 2➤ПРОХОЖДЕНИЕ➤Часть 4🎮
2 days ago 00:36:25 1
Decree x40 “Beloved Surya“ by Elizabeth Clare Prophet
2 days ago 00:20:33 1
Movie “Wicked” Sign of the End Time
2 days ago 00:24:07 1
UKRAINE : C’EST ENCORE PIRE QUE TOUT CE QU’ON IMAGINAIT ! | XAVIER MOREAU | GPTV ACTU
2 days ago 00:03:59 2
HAUSER - Waltz No. 2 (Shostakovich)
3 days ago 02:01:18 1
Half-Life 2: 20th Anniversary Documentary
3 days ago 00:01:14 1
Гиперзвуковая ракета Орешник России, первые данные
3 days ago 00:29:51 3
Попробуй World of Warcraft
3 days ago 00:14:16 1
ЧТО??? ЭТО PRODIGY??? (как делается брейкбит хардкор)
3 days ago 00:03:37 1
Админы Актуального Курска и другие куряне, покажите общественные туалеты Курска .?!) #курскиймайор
3 days ago 00:00:00 1
🔴 LIVE Прославление | Церковь Божия Music | #cogmosmusic
3 days ago 00:31:06 1
Whistleblow Uncovers Covid Scam
4 days ago 00:00:32 1
…but the people are retarded
4 days ago 00:16:46 1
Поиграл в Ready or Not и получил депрессию
5 days ago 00:13:15 1
대왕 꽈배기 4개에 천원? 사장님이 봉사 정신으로 판매하는! 30년 경력의 꽈배기, 도너츠 / Super Speed Donuts Master / Korean street food
1 week ago 00:20:35 1
3-1-2. Страшные Гик Сказки - “The Parade“
1 week ago 00:13:15 1
BREAKING: MASS EXODUS Of Soldiers Rock IDF After BLOODIEST DAY EVER in Lebanon
1 week ago 00:15:33 6
Spotlight 2 класс (Спотлайт 2) Английский в фокусе 2кл./ Урок 15 “In the Bath!“ 3b стр. 36 - 38