512 (d_model) in the base model. The larger Transformer (big) in the same paper used 1024.
Proposed by @GokuMohandas
Card 1 of 276. Answer: 512 (d_model) in the base model. The larger Transformer (big) in the same paper used 1024.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
512. In the base model of "Attention Is All You Need" by Ashish Vaswani and colleagues at Google, presented at NIPS 2017, every embedding layer and sub-layer outputs vectors of size d_model = 512. d_model is the width of the vector that represents each token as it moves through the network.
The other base hyperparameters fit around it: 6 encoder and 6 decoder layers, 8 attention heads of 64 dimensions each (8 × 64 = 512), and a feed-forward inner layer of 2048, four times d_model. The model had about 65 million parameters and trained in 12 hours on 8 P100 GPUs. Transformer (big) doubled d_model to 1024, with 16 heads.
"All" is loose: each head works in 64 dimensions and the feed-forward layer widens to 2048, but every sub-layer's output returns to 512 so the residual additions line up. The embeddings are also multiplied by √512 ≈ 22.6. Later models are wider: BERT-base uses 768 and GPT-3's largest model 12,288.
