When building your vocabulary set, it’s ok to use the tokens from the entire dataset to reduce the size of your out-of-vocab list?

False, any tokens not seen in the training set need to be UNK tokens.

Proposed by @GokuMohandas