A symmetric, bounded measure of the difference between two distributions: the average KL divergence of each from their mixture M = (P + Q) / 2. It is 0 for identical distributions, at most ln 2 (1 bit), and stays finite even when the distributions don't overlap.
Card 1 of 228. Answer: A symmetric, bounded measure of the difference between two distributions: the average KL divergence of each from their mixture M = (P + Q) / 2. It is 0 for identical distributions, at most ln 2 (1 bit), and stays finite even when the distributions don't overlap.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The Jensen-Shannon divergence is JSD(P ‖ Q) = ½ KL(P ‖ M) + ½ KL(Q ‖ M), with M the average of P and Q. Comparing each distribution with the mixture fixes KL divergence's two practical problems: the result is the same in both directions, and it never blows up, because M is positive wherever P or Q is. Its square root, the Jensen-Shannon distance, is a true metric.
Example: P = (0.5, 0.5, 0) and Q = (0, 0.5, 0.5) over three categories. KL(P ‖ Q) is infinite, since P puts mass on the first category where Q has none. The mixture is M = (0.25, 0.5, 0.25), and KL(P ‖ M) = KL(Q ‖ M) = 0.5 ln 2, so JSD = 0.35 nats, or exactly 0.5 bits: halfway between identical (0) and fully disjoint (1 bit).
Because it is bounded, the same threshold means the same thing for every feature, which makes it a common drift metric in model monitoring. It is also what the original GAN objective minimizes when the discriminator is optimal (Goodfellow et al., 2014). When the distributions don't overlap at all it stays stuck at its maximum and gives no gradient, one motivation for the Wasserstein GAN.
