Information trimming: Sufficient statistics, mutual information, and predictability from effective channel states

One of the most basic characterizations of the relationship between two random variables, $X$ and $Y$, is the value of their mutual information. Unfortunately, calculating it analytically and estimating it empirically are often stymied by the extremely large dimension of the variables. One might hope to replace such a high-dimensional variable by a smaller one that preserves its relationship with the other. It is well known that either $X$ (or $Y)$ can be replaced by its minimal sufficient statistic about $Y$ (or $X)$ while preserving the mutual information. While intuitively reasonable, it is not obvious or straightforward that both variables can be replaced simultaneously. We demonstrate that this is in fact possible: the information $X$'s minimal sufficient statistic preserves about $Y$ is exactly the information that $Y$'s minimal sufficient statistic preserves about $X$. We call this procedure information trimming. As an important corollary, we consider the case where one variable is a stochastic process' past and the other its future. In this case, the mutual information is the channel transmission rate between the channel's effective states. That is, the past-future mutual information (the excess entropy) is the amount of information about the future that can be predicted using the past. Translating our result about minimal sufficient statistics, this is equivalent to the mutual information between the forward- and reverse-time causal states of computational mechanics. We close by discussing multivariate extensions to this use of minimal sufficient statistics.

Figure 1

Information diagram (I-diagram) for four random variables $X,W,Z$, and $Y$. Each is depicted as a stadium shape and the information atoms are obtained by forming all possible intersections. Individual atoms are identified with lowercase letters.

Figure 2

I-diagram for sufficient statistics: The vanishing information atoms implied by a sufficient statistic being a function of a random variable are labeled $0^{*}$. Those vanishing atoms implied by a sufficient statistic forming a Markov chain are marked with $0^{†}$.

Figure 3

Minimal I-diagram containing only nonvanishing atoms in Fig. 2.

Figure 4

Minimal I-diagram involving three variables and their minimal sufficient statistics. This differs from a standard three-variable I-diagram by the addition of three atoms: $\mathrm{H}\left[X\right|X\searrow YZ],\mathrm{H}[Y|Y\searrow XZ]$, and $\mathrm{H}\left[Z\right|Z\searrow XY]$.

Figure 5

Information trimming example: The distinction between $X=0$ and $X=2$ is irrelevant for the statistics of $Y$. Similarly, the distinction between $Y=0$ and $Y=1$ is irrelevant for the statistics of $X$. The informationally trimmed version sheds this 1.5 bit of information.