Past-future information bottleneck in dynamical systems

Biological systems need to process information in real time and must trade off accuracy of presentation and coding costs. Here we operationalize this trade-off and develop an information-theoretic framework that selectively extracts information of the input past that is predictive about the output future, obtaining a generalized eigenvalue problem. Thereby, we unravel the input history in terms of structural phase transitions corresponding to additional dimensions of a state space. We elucidate the relation to canonical correlation analysis and give a numerical example. Altogether, this work relates information-theoretic optimization to the joint problem of system identification and model reduction.

Figure 1

Information curve for the spring-mass system. The dynamics is given by the matrices in Eq. (16) and the following parameters: $m_1=5$; $m_2=\sqrt{15}$; $k_1=1$; $k_2=0.5$; $c=0$. This particular system has Hankel singular values ${\sigma }_1=0.3358$, ${\sigma }_2=0.3109$, ${\sigma }_3=0.2374$, and ${\sigma }_4=0.2103$. The reduced systems $H\left(\beta \right)$, shown as gray squares, all lie on the information curve as given by the Hankel singular values. Circles indicate critical beta values.