TF-IDF weights a word in a document by its term frequency (how often it appears there) times its inverse document frequency, log(N / df), which is high for words few documents contain. Words frequent in one document but rare across the corpus score highest.
Proposed by @Al_Grigor
Card 1 of 259. Answer: TF-IDF weights a word in a document by its term frequency (how often it appears there) times its inverse document frequency, log(N / df), which is high for words few documents contain. Words frequent in one document but rare across the corpus score highest.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
TF-IDF, term frequency times inverse document frequency, weights each word in a document by how characteristic it is: tfidf(t, d) = tf(t, d) × idf(t). Term frequency is how often t appears in document d; idf(t) = log(N / df(t)), where t occurs in df(t) of the N documents, rewards rare words. Words frequent in one document but rare overall score highest.
Example: a 100-word article among 1,000 documents. 'Learning' appears 5 times and occurs in 100 documents: tf = 0.05, idf = log₁₀(1000/100) = 1, weight 0.05. 'The' appears 6 times but occurs in all 1,000: idf = log₁₀(1000/1000) = 0, weight 0. Karen Spärck Jones introduced IDF in 1972.
Why the log? Without it, a word in 1 document would outweigh one in 10 by a factor of 10; here log₁₀ makes it 3 against 2. Libraries vary: scikit-learn uses raw counts, a smoothed idf plus 1, and unit-length rows, so its numbers differ. Fit IDF on training documents only. TF-IDF ignores word order and meaning.
