Information-theoretical analysis of the statistical dependencies among three variables: Applications to written language

We develop the information-theoretical concepts required to study the statistical dependencies among three variables. Some of such dependencies are pure triple interactions, in the sense that they cannot be explained in terms of a combination of pairwise correlations. We derive bounds for triple dependencies, and characterize the shape of the joint probability distribution of three binary variables with high triple interaction. The analysis also allows us to quantify the amount of redundancy in the mutual information between pairs of variables, and to assess whether the information between two variables is or is not mediated by a third variable. These concepts are applied to the analysis of written texts. We find that the probability that a given word is found in a particular location within the text is not only modulated by the presence or absence of other nearby words, but also, on the presence or absence of nearby pairs of words. We identify the words enclosing the key semantic concepts of the text, the triplets of words with high pairwise and triple interactions, and the words that mediate the pairwise interactions between other words.

Figure 1

Different ways in which three variables may interact. (a) The three variables are independent. (b) Only pairwise interactions exist. These may involve 1, 2, or 3 links (from left to right). (c) The three variables are connected by a single triple interaction. (d) Double and triple interactions may coexist. The most general case is illustrated in the bottom-right panel.

Figure 2

(a) Density plot of the two bivariate probability distributions that have $I=1$ bit. Dark states have zero probability, and white states have $p(x_1,x_2)=1/2$. (b) Density plot of the two trivariate probability distributions with $D_{ijk}^{\left(3\right)}=1$ bit. Dark states have zero probability, and white states have $p(x_1,x_2,x_3)=1/4$. (c) Gradual change between a uniform distribution and a XOR distribution, for different values of $\theta$ [Eq. (13)]. (d) Amount of triple interactions as a function of the parameter $\theta$.

Figure 3

Examples illustrating the effects of marginalization in a pair of variables (a) or a triplet (b). In each case, the variable represented in black drives the other slave variables, which do not interact directly with each other (top). However, after marginalizing over the driving variable, a statistical dependence between the remaining variables appears. The new interaction can be pairwise (a), or pairwise and tripletwise (b).

Figure 4

Entropy of the 400 selected words in each book (one data point per word), compared to the expected entropy for a binomial variable with the same total count $n_i$ (continuous line), as a function of the total count. Entropies are calculated with the maximum likelihood estimator. The analytical expression of Eq. (24) is represented with the black line, and the gray area corresponds to the percentiles 1%–99% of the dispersion expected in the binomial model, when using a sample of $n_i$ words. Data points outside the gray area, hence, are highly unlikely under the binomial hypothesis, even when allowing for inaccuracies due to limited sampling. (a) OS. (b) AM.

Figure 5

Central graph: network of pairwise interactions in OS. Width of links proportional to the mutual information between the two connected words. Insets: detail of selected subnetworks. Top graph: links proportional to mutual information. Bottom graph: links proportional to irreducible interaction.

Figure 6

Fraction of the total interaction within a triplet ${\mathrm{\Delta }}_{ijk}$ that corresponds to tripletwise dependencies $D_{ijk}^{\left(3\right)}/{\mathrm{\Delta }}_{ijk}$ as a function of the total interaction. The gray level of each data point is proportional to the (logarithm of the) number of triplets at that location (scale bars on the right). ${\mathrm{\Delta }}_{ijk}$ values above 0.01 bits are significant (see Appendix). (a) OS. (B) AM. Dashed line: averages over all triplets with the same ${\mathrm{\Delta }}_{ijk}$.

Figure 7

Triple information $D_{ijk}^3$ as a function of the co-information $I_{ijk}$ for all triplets. The gray level of each data point is proportional to the (logarithm of the) number of triplets at that location (scale bars on the right). ${\mathrm{\Delta }}_{ijk}$ values above 0.01 bits are significant (see Appendix). (a) OS. (b) AM.

Figure 8

Interactions ${\mathrm{\Delta }}_{123}, D_{123}^{\left(2\right)}$, and $D_{123}^{\left(3\right)}$ as a function of the field $\beta$ acting on $X_4$.

Figure 9

Standard deviation of the triple information $D_{ijk}^3$ as a function of the $D_{ijk}^3$, for the triplets that satisfy $D_{ijk}^3>0.01$. (a) OS. (b) AM. The dashed line indicates the identity.