False. Build the vocabulary from the training split only (or from an external corpus, such as a pretrained model's vocabulary); tokens seen only in validation or test data map to UNK, or are split into known subword pieces by a subword tokenizer.
Proposed by @GokuMohandas
Card 1 of 259. Answer: False. Build the vocabulary from the training split only (or from an external corpus, such as a pretrained model's vocabulary); tokens seen only in validation or test data map to UNK, or are split into known subword pieces by a subword tokenizer.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. Build the vocabulary from the training split only. A token that appears only in validation or test data maps to the unknown token, UNK, exactly as a new word would in production. Building it from the full dataset is data leakage: preprocessing is fit on the data the model will be judged on.
Shrinking the out-of-vocabulary list this way gains nothing real. Suppose "chargeback" appears only in the test set. Given its own ID, its embedding row never receives a gradient and keeps its random initial values, so the model reads noise instead of UNK. The evaluation also hides a real problem: production text will contain unseen words, and the test set no longer measures how the model copes.
In practice, also map rare training tokens, such as those seen only once, to UNK so the UNK embedding is actually learned. Fit frequency cutoffs and TF-IDF weights on training data too. Subword tokenizers such as BPE and WordPiece shrink the problem by splitting unseen words into known pieces.
