Contents
What is the relationship between correlation and mutual information?
More specifically, we will compare- and discuss the relationship between- correlation and mutual information. The Mutual Information between 2 random variables is the amount of information that one gains about a random variable by observing the value of the other.
Ranking the features by p-value from a chi-squared test with the response is equivalent to ranking the features by absolute correlation with the response. For fixed sample size N N, the p-value itself is a measure of association strength. When both X X and Y Y are binary, we can view X X as defining group membership and Y Y as defining an outcome.
What is mutual information between two random variables?
The Mutual Information between 2 random variables is the amount of information that one gains about a random variable by observing the value of the other. It is linked to the entropy – a key concept in Information Theory (which can be thought of as how surprised we are on average when observing a random variable – as discussed in a previous post).
Why does the correlation method work so well?
In this post, I want to discuss why any of the above approaches should work well. The basic insight is that: The correlation is a reparametrization of p-values obtained via t-tests, F-tests, proportion tests, and chi-squared tests, meaning that ranking features by p-value is equivalent to ranking them by correlation (for fixed sample size N N)
Mutual information is a fundamental and interesting way of thinking about how two random variables are related. In the case of the bivariate Gaussian distribution it is a highly non-linear function of the usual measure of how random variables are related, i.e. correlation.
How to calculate mi with a correlation coefficient?
If follows a bivariate Gaussian distribution, with mean vector , unit variance for both variables and a correlation coefficient , we write the pdf as: To calculate the MI we need the following results, shown without proof: