Why are Nonlinear Fits to Data so Challenging?

Fitting model parameters to experimental data is a common yet often challenging task, especially if the model contains many parameters. Typically, algorithms get lost in regions of parameter space in which the model is unresponsive to changes in parameters, and one is left to make adjustments by hand. We explain this difficulty by interpreting the fitting process as a generalized interpolation procedure. By considering the manifold of all model predictions in data space, we find that cross sections have a hierarchy of widths and are typically very narrow. Algorithms become stuck as they move near the boundaries. We observe that the model manifold, in addition to being tightly bounded, has low extrinsic curvature, leading to the use of geodesics in the fitting process. We improve the convergence of the Levenberg-Marquardt algorithm by adding geodesic acceleration to the usual step.

Figure 1

The model manifold for the two-exponential problem, with $y_i$ evaluated at $t=1/3$, 1, and 3. Boundaries exist when ${\theta }_{\mu }=0,\infty$ and when ${\theta }_1={\theta }_2$. (The ribbonlike structure of Fig. 2 emerges only in higher dimensions.)

Figure 2

Top: Two views of the cross section of the model manifold for an infinite sum of exponentials $F_{\mathbf{A},\theta }(t)=\sum _n{A}_n\exp (-{\theta }_{n}t)$ with $A_n\ge 0$, given by fixing $F(0)=1$ and $F(1)=1/e$. Bottom: The range of allowed fits (gray) is strongly reduced by fixing the output at $t=1/2$ to the midpoint of its range (black).

Figure 3

Geodesic cross-sectional widths of an eight-dimensional model manifold along the eigendirections of the metric from some central point, together with the square root of the eigenvalues (singular values of the Jacobian), the inverse extrinsic (geodesic) curvature $K$ [24], and the inverse geodesic parameter-effects curvature $K^P$ [2, 3, 10]. Notice the hierarchy of these data-space distances—the widths and singular values each spanning around 4 orders of magnitude and the curvatures covering 8 orders of magnitude. Note also that the extrinsic curvatures are 3 orders of magnitude smaller than the parameter-effects curvature.

Figure 4

Geodesics can be used to construct polar coordinates on $\mathcal{M}$ (top). In these new geodesic coordinates, the cost contours are nearly perfect, isotropic circles (bottom).