In
probability theory and
information theory, the
Kullback–Leibler divergence (also
information divergence,
information gain,
relative entropy,
KLIC, or
KL divergence) is a measure of the difference between two probability distributions
P and
Q. It is not symmetric in
P and
Q. In applications,
P typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while
Q typically represents a theory, model, description, or approximation of
P.