Contents
How to calculate the Kullback-Leibler divergence for two samples?
Lets say p and q are uniform densities on [ 0, 1] and [ 0, 10] respectively. Then KL ∞. This results are reasonable from the interpretation I give in Intuition on the Kullback-Leibler (KL) Divergence Returning to the main question. It is asked in a very nonparametric way, and no assumptions are stated on the densities.
Is the KL divergence measure of the distance between two distributions?
KL Divergence – Actually provides a measure of how big of a difference are two distributions from each other. As mentioned by the previous answer, this measure isnt an appropriate distance metric since its not symmetrical. I.e. distance between distribution A and B is a different value from the distance between distribution B and A.
Is the K-L divergence between discrete probability symmetric?
A simple example shows that the K-L divergence is not symmetric. This is explained by understanding that the K-L divergence involves a probability-weighted sum where the weights come from the first argument (the reference distribution).
Which is the measure of dissimilarity between two probability distributions?
The Kullback–Leibler divergence is a measure of dissimilarity between two probability distributions. It measures how much one distribution differs from a reference distribution. This article explains the Kullback–Leibler divergence and shows how to compute it for discrete probability distributions.
Is the KL divergence of two distributions always the same?
But assuming the two densities are proposed as competing models for the same phenomenon, we can probably assume they have the same dominating measure: KL divergence between a continuous and a discrete probability distribution would always be infinity, for example.
How to estimate KL divergence with IID samples?
Some ideas from that paper, which is about estimation of KL divergence with iid samples from absolutely continuous distributions. I show their proposal for one-dimensional distributions, but they give a solution for vectors also (using nearest neighbor density estimation).
How to calculate the divergence of two samples?
To debug the implementation draw the samples from two normal distributions N ( 0, 1) and N ( 1, 2). For a simple estimate I generated two histograms and tried to numerically approximate the integral.