BERT is an encoder: bidirectional, trained by masking tokens, so every position sees both sides, which suits classification and embeddings but cannot generate. GPT is a causal decoder trained to predict the next token, which is what enables generation.
Card 1 of 292. Answer: BERT is an encoder: bidirectional, trained by masking tokens, so every position sees both sides, which suits classification and embeddings but cannot generate. GPT is a causal decoder trained to predict the next token, which is what enables generation.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
BERT is a bidirectional Transformer encoder trained to fill in masked tokens, while a GPT-style model is a decoder trained to predict each next token from left context. The blocks are nearly identical; the mask and objective differ. Devlin and colleagues introduced BERT in 2018, months after OpenAI's first GPT.
In "the cat sat on the mat", GPT's representation of "sat" sees only "the cat sat"; BERT's sees all six words. A causal mask over n tokens allows n(n + 1)/2 attention pairs, 21 of 36 here. BERT's loss is also sparser: it predicts only the 15% of tokens it selects, about 77 of 512. Of those, 80% become [MASK], 10% a random token and 10% stay unchanged, since [MASK] never appears in fine-tuning. GPT gets a loss at every position.
Why can't BERT generate? It can fill a blank, but has no left-to-right factorization, so writing text needs awkward repeated masking. Why still use encoders? A 110-million-parameter BERT-base fine-tuned for classification or embeddings is fast and cheap. RoBERTa (2019) dropped BERT's second objective, next-sentence prediction, at no loss.
