Principles of maximum entropy and maximum caliber in statistical physics

The variational principles called maximum entropy (MaxEnt) and maximum caliber (MaxCal) are reviewed. MaxEnt originated in the statistical physics of Boltzmann and Gibbs, as a theoretical tool for predicting the equilibrium states of thermal systems. Later, entropy maximization was also applied to matters of information, signal transmission, and image reconstruction. Recently, since the work of Shore and Johnson, MaxEnt has been regarded as a principle that is broader than either physics or information alone. MaxEnt is a procedure that ensures that inferences drawn from stochastic data satisfy basic self-consistency requirements. The different historical justifications for the entropy $S=-\sum _i{p}_i\log p_i$ and its corresponding variational principles are reviewed. As an illustration of the broadening purview of maximum entropy principles, maximum caliber, which is path entropy maximization applied to the trajectories of dynamical systems, is also reviewed. Examples are given in which maximum caliber is used to interpret dynamical fluctuations in biology and on the nanoscale, in single-molecule and few-particle systems such as molecular motors, chemical reactions, biological feedback circuits, and diffusion in microfluidics devices.

Figure 1

The rate distribution $\alpha (\tau )$ taken from pulse fluorimetry experiments on L-tryptophan - vs $\tau$ on a logarithmic scale at two emission wavelengths (390 nm solid line, 320 nm with circles). From 120.

Figure 2

One possible stochastic trajectory of a single ion-channel transitioning between open (conducting) and closed (nonconducting) states (145; 49).

Figure 3

Microfluidics experiment on few-particle to test MaxCal. Colloids corralled on one side of a gate begin to diffuse at time $t=0$ when the gate is opened. (a) Schematic of the microfluidics chip. (b) Particle diffusion. From 163. (c) The flux distribution function $\frac{1}{2}\ln [⟨(\Delta J{)}^2⟩]+\log [P(J)]$ vs $(J-⟨J⟩)/\sqrt{2⟨(\Delta J{)}^2⟩}$. (d) The average flux $⟨J⟩$ is shown as a function of $\Delta N=N_1-N_2$. (e) The second cumulant $⟨\Delta J^2⟩=⟨J^2⟩-⟨J{⟩}^2$ vs the total number of particles $N=N_1+N_2$. For (c), (d), and (e), circles show experimental data; solid line shows MaxCal theory.

Figure 4

A dual-laser-trap experiment in which a colloidal particle hops between two “sculpted” energy wells, back and forth, with observed time trajectories as shown. From 204.

Figure 5

The $x$ axes give the predicted second cumulants, covariance, and third cumulants from the MaxCal approach, based on the known first moments. The $y$ axes give the experimental values. The quantities are variance in $N_1$, $N_{1\to 2}$ [which in the figure from the original source (204) read $N_B$ instead of $N_1$ and $N_{ba}$ instead of $N_{1\to 2}$], covariance of $N_1{N}_{1\to 2}$ and third cumulant of $N_{1\to 2}$ from left to right. We define $N_1=N_{1\to 1}+N_{2\to 1}+N_{0\to 1}$, where $N_{0\to 1}={\delta }_{i_0,1}$. The dashed lines are the best linear fits; fitting parameters are inset. Each point represents one experimentally observed trajectory. Trajectories were 30 000 $\Delta t$ units long, and errors were calculated for around 600 trajectories. From 204.

Figure 6

(a) One example of a time trajectory for a three-state cycle. (b) A three-state cycle, showing the definitions of rates used in the text.

Figure 7

A plot of ${\delta }_s=⟨J^2{⟩}_c/⟨J{⟩}^2$ vs $r=[(\sigma +ϵ)/(\sigma -ϵ){]}^2$, where $s$ denotes the number of states in the cycle.

Figure 8

Rate of occurrence of firing patterns with the approximated pattern rate vs the observed pattern rate. The solid black line indicates strict agreement. The (darker) colored dots coincide with the prediction made from the MaxEnt model where correlations were used as constraints, model $P_2$. The other (lighter) colored dots coincide with those predictions made from the independent Ising model, model $P_1$, where only mean firing patterns of each neuron are used as constraints.

Figure 9

The genetic toggle switch. The DNA plasmid is shown with promoters $p_A$ and $p_B$ of genes $g_A$ and $g_B$ that, when transcribed, produce proteins $A$ and $B$, respectively. In this gene circuit, production of $A$ inhibits production of $B$. And production of $B$ inhibits $A$.

Figure 10

Time trajectories of the toggle switch. (a) Gillespie simulation (77) of the model using $d=0.005$, $k=0.1$, $f=100$, and $r=2$. This acts as our “experimental data,” with fully known underlying behavior. (b) MaxCal time trace of the model that we then extracted from the averages of the “experiment.” This figure only validates that the parameters extracted by MaxCal do give trajectories resembling those from which they are extracted. The main quantitative tests, described in text and in the following figures, are the inference of entire statistical distributions.

Figure 11

Prediction of distributions by MaxCal and comparison with synthetic data. (a) Distribution of particle numbers. (b) Distribution of dwell times. Black (darker) curves: Gillespie “real data” simulations. Colored (lighter) curves: model predicted typical trajectory using the parameters extracted by MaxCal using Eq. (130). From 146.