Because of the residual connections.
Proposed by @GokuMohandas
Card 1 of 259. Answer: Because of the residual connections.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Because of the residual connections. Each sub-layer, self-attention or feed-forward, is wrapped as LayerNorm(x + Sublayer(x)), and two vectors can only be added if they have the same length. The paper says so: to facilitate these residual connections, all sub-layers and embedding layers output dimension d_model.
A 10-token sentence enters as a 10 × 512 matrix. Eight attention heads each output 10 × 64; concatenated they give 10 × 512, and the output projection W^O keeps that width. The feed-forward block widens each token to 2048, then projects back to 512 before the addition. Positional encodings are added to the embeddings, not concatenated, so they need width 512 too.
Is equal width strictly required? No: ResNet uses a projection shortcut, a 1 × 1 convolution, where dimensions change. Transformers keep an identity shortcut, which costs no parameters and gives the error a direct route back, since the gradient of x + F(x) is I + ∂F/∂x. Later models move LayerNorm inside the branch, x + F(LayerNorm(x)), to keep that route clean.
