Unwinding the model manifold: Choosing similarity measures to remove local minima in sloppy dynamical systems

In this paper, we consider the problem of parameter sensitivity in models of complex dynamical systems through the lens of information geometry. We calculate the sensitivity of model behavior to variations in parameters. In most cases, models are sloppy, that is, exhibit an exponential hierarchy of parameter sensitivities. We propose a parameter classification scheme based on how the sensitivities scale at long observation times. We show that for oscillatory models, either with a limit cycle or a strange attractor, sensitivities can become arbitrarily large, which implies a high effective dimensionality on the model manifold. Sloppy models with a single fixed point have model manifolds with low effective dimensionality, previously described as a “hyper-ribbon.” In contrast, models with high effective dimensionality translate into multimodal fitting problems. We define a measure of curvature on the model manifold which we call the winding frequency that estimates the density of local minima in the model's parameter space. We then show how alternative choices of fitting metrics can “unwind” the model manifold and give low winding frequencies. This prescription translates the model manifold from one of high effective dimensionality into the hyper-ribbon structures observed elsewhere. This translation opens the door for applications of sloppy model analysis and model reduction methods developed for models with low effective dimensionality.

Figure 1

Model classes. [(a), (d), (g)]: Cross sections of $C\left(\theta \right)$ [Eq. (1)] for three prototype models (see Appendix pp1, models $A, D$, and $G$). Contrast the canyons in (a) with the ripples in (d) and the roughness in (g). [(b), (e), (h)]: Hessian eigenvalues as a function of sampling time (same models). Colors differentiate scaling behaviors at long times. [(c), (f), (i)]: Projections of the model manifold (same models). In (c), a three-ball in parameter space was mapped to the nearly one-dimensional region of prediction space shown (low effective dimensionality). By contrast, for (f) and (i) a single parameter was varied producing a one-dimensional (1D) (space-filling) curve in prediction space (high effective dimensionality). Note that in (i), the model goes through a bifurcation where the manifold begins to oscillate rapidly. The sampling required to see continuity is prohibitive, so the points plotted become scattered.

Figure 2

Eigenvalues of $H\left({\theta }_0\right)$ for the following models (see Appendix pp1 for details): $A$: sum of exponentials; $B$: rational polynomial; $C$: biological adaptation; $D$: FitzHugh-Nagumo; $E$: Hodgkin-Huxley; $F$: Wnt oscillator; $G$: Lorenz; $H$: Hindmarsh-Rose; $I$: damped, driven pendulum. $\{A,B,C\}$ are nonoscillatory models, $\{D,E,F\}$ are periodic, and $\{G,H,I\}$ are chaotic.

Figure 3

Illustration of winding frequency. The “s”-shaped curve represents a possible 1D cross section of a model manifold (obtained, for example, by varying just one parameter combination) in a simple 2D data space. Also shown are the tangent or velocity vector $(\partial \mathbf{y}/\partial {\theta }^{\mu })v^{\mu }$ (sum over $\mu$ implied), the tangent circle with radius $R$, and the winding frequency $\omega$ defined in Eq. (15).

Figure 4

Winding frequencies along Hessian eigendirections for the models from Fig. 2, ordered from left to right by magnitude of the corresponding eigenvalue. “Stiff” refers to eigendirections with large eigenvalues, while “sloppy” refers to eigendirections with small eigenvalues. The black dashed line at $\omega =2\pi$ roughly distinguishes low from high winding frequencies.

Figure 5

Cost $C$ for the model $y\left(t\right)=Acos\left(\omega t\right)$, treating $A$ and $\omega$ as parameters. The cost has been rescaled to make the local minima apparent.

Figure 6

Decoupling amplitude from phase. (a) Signal vs time for two signals with mismatched amplitude and frequency; their difference is indicated by the shading between the curves. The mismatch in frequency causes a large difference $\delta y$ when the two signals are out of phase ($t\approx 1.8$) but little or no difference when they are in phase ($t\approx 3.5$). (b) Signal vs phase for the same signals. The difference-at-constant-phase $\delta \tilde{y}$ is consistent throughout (see Sec. 4a2). (c) Phase vs time for the two signals. The difference in phase $\delta \varphi$ simply grows linearly (see Sec. 4a2). (d) Analytic representation in the complex plane of the black point marked in the other three panels (see Sec. 4a1).

Figure 7

Cost decomposition of the model $y\left(t\right)=Acos\left(\omega t\right)$. (a) Same as Fig. 5. (b) $\tilde{y}\left(\varphi \right)=Acos\left(\varphi \right)$ is insensitive to changes in $\omega$ and varies linearly with $A$, resulting in a quadratic dependence of $C^{\tilde{y}\left(\varphi \right)}$ on $A$ only. (c) $\varphi \left(t\right)=\omega t$ is insensitive to changes in $A$ and varies linearly with $\omega$, resulting in a quadratic dependence of $C^{\varphi \left(t\right)}$ on $\omega$ only. (d) Cost using Eq. (27). The ripples in (a) have been replaced with a quadratic basin.

Figure 8

Effects of alternative metrics on cost surfaces (first column), winding frequencies (second column), and manifolds (third column), (a)–(c) using analytic signal (AS) and (d)–(f) using kernel density estimation (KDE). Compare (a),(c),(d), and (f) with Figs. 1, 1, 1, and 1, respectively. Panels (b) and (e) show both the winding frequencies shown previously in Fig. 4 (“w/o ___”) and the winding frequencies that result when using our metrics (“using ___”) for comparison. Note that long time series are needed to achieve these results; see Appendix pp5 for details.

Figure 9

Phases obtained when implementing the Hilbert transform numerically on the model $y(t,\theta )=Acos\left(\omega t\right)$, for $\omega$ ranging from $2\pi$ to $4\pi$. The effects of the Gibbs phenomenon can be seen near the ends for some values of $\omega$.

Figure 10

Propagation of uncertainty. (a) Data (blue) simulated from the model $y\left(t\right)=Acos\left(\omega t\right)$ (red) by adding uniform Gaussian noise. Error bars indicate uncertainty. (b) Data (blue) plotted as a function of phase compared with $y\left(\varphi \right)=Acos\left(\varphi \right)$ (red). Error bars indicate the uncertainties obtained using Eqs. (C14) and (C20). (c) Phase (blue), obtained for each data point using Eq. (19), compared with $\varphi \left(t\right)=\omega t$ (red). Error bars indicate the uncertainties obtained using Eq. (C14).

Figure 11

Magnitude of the gradient of the FitzHugh-Nagumo cost $|\mathbf{\nabla }{C}^{\varphi }\left(\theta \right)|$. The magnitude of the gradient has only one minimum, indicating that the cost cross section shown in Fig. 8 has a single minimum. The minimum of $|\mathbf{\nabla }{C}^{\varphi }\left(\theta \right)|$ shown is not quite zero because the actual minimum of Fig. 8 is between the grid points where $|\mathbf{\nabla }{C}^{\varphi }\left(\theta \right)|$ has been calculated. Note that in the upper corners of the plot, there is a phase transition to nonoscillatory behavior, where the methods of Sec. 4a cannot be applied effectively. The sharp apparent dropoff is due to such choices as having our algorithms return zeros rather than throw errors for these regions.

Figure 12

FitzHugh-Nagumo manifold projection. $A$ was calculated using about 24 time points per cycle in the original time series; $B$ was calculated using twice the time sampling of $A$.

Figure 13

FitzHugh-Nagumo amplitude oscillations. Colors are the same as in Fig. 12, with dark or light indicating the value of the parameter $a$. As the peak moves between sampled time points, the amplitude appears to oscillate.

Figure 14

Lorenz cost. As the number of sampled time points grows, the noise in the cost dies away. When fit to a parabola, the MAE between the parabola and the cost cross section shown is 0.0088 for $T=80$ and 0.0013 for $T=800$ (about a sevenfold reduction in noise).