Interaction Information Images

The interaction information (McGill 1954) or co-information (Bell 2003) is one of several generalizations of the mutual information, and expresses the amount information (redundancy or synergy) bound up in a set of variables, beyond that which is present in any subset of those variables. Unlike the mutual information, the interaction information can be either positive or negative. This confusing property has likely retarded its wider adoption as an information measure in machine learning and cognitive science.

where, for example, is the mutual information between variables and , and is the conditional mutual information between variables and given . Formally,

For the three-variable case, the interaction information is the difference between the information shared by when has been fixed and when has not been fixed. (See also Fano's 1961 textbook.) Interaction information measures the influence of a variable on the amount of information shared between . Because the term can be zero — for example, when the dependency between is due entirely to the influence of a common cause , the interaction information can be negative as well as positive. Negative interaction information indicates that variable inhibits (i.e., accounts for or explains some of) the correlation between , whereas positive interaction information indicates that variable facilitates or enhances the correlation between .