The Kullback-Leibler divergence, KL(P ‖ Q) = Σ P(x) log(P(x) / Q(x)), measures how much information is lost when Q is used to approximate P. It is never negative, is zero only when P = Q, and is not symmetric, so it is not a distance.
Card 1 of 228. Answer: The Kullback-Leibler divergence, KL(P ‖ Q) = Σ P(x) log(P(x) / Q(x)), measures how much information is lost when Q is used to approximate P. It is never negative, is zero only when P = Q, and is not symmetric, so it is not a distance.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
KL divergence is the expected extra cost, in bits with log base 2 or nats with the natural log, of encoding samples from P with a code built for Q. It equals the cross-entropy of P and Q minus the entropy of P, which is why minimizing cross-entropy loss against fixed labels also minimizes KL(labels ‖ model): the labels' entropy is a constant.
Example: P is a fair coin, (0.5, 0.5), and Q is a biased one, (0.9, 0.1). KL(P ‖ Q) = 0.5 ln(0.5/0.9) + 0.5 ln(0.5/0.1) = 0.51 nats, but KL(Q ‖ P) = 0.9 ln(0.9/0.5) + 0.1 ln(0.1/0.5) = 0.37 nats. The order matters. If Q gives zero probability to something P can produce, KL(P ‖ Q) is infinite.
It shows up across machine learning: the regularizer in a variational autoencoder's loss, the penalty that keeps an RLHF-tuned model close to its reference, the objective t-SNE minimizes, and, summed in both directions, the population stability index used to monitor drift. Fitting Q by minimizing KL(P ‖ Q) spreads it to cover all of P's modes; minimizing KL(Q ‖ P) locks onto one. Solomon Kullback and Richard Leibler introduced it in 1951.
