Estimability and dependency analysis of model parameters based on delay coordinates

In data-driven system identification, values of parameters and not observed variables of a given model of a dynamical system are estimated from measured time series. We address the question of estimability and redundancy of parameters and variables, that is, whether unique results can be expected for the estimates or whether, for example, different combinations of parameter values would provide the same measured output. This question is answered by analyzing the null space of the linearized delay coordinates map. Examples with zero-dimensional, one-dimensional, and two-dimensional null spaces are presented employing the Hindmarsh-Rose model, the Colpitts oscillator, and the Rössler system.

Figure 1

Illustration of different orientations of a one-dimensional null space.

Figure 2

Illustration of different bases of a two-dimensional null space. ${\mathbf{v}}_{\text{B}}^{\left(1\right)}$ and ${\mathbf{v}}_{\text{B}}^{\left(2\right)}$ denote the orthonormal basis vectors obtained with the SVD, while ${\tilde{\mathbf{v}}}_{\text{B}}^{\left(1\right)}$ and ${\tilde{\mathbf{v}}}_{\text{B}}^{\left(2\right)}$ are the nonorthogonal basis vectors with vanishing components computed using Eq. (22).

Figure 3

Histograms (color coded) of the normalized singular values ${\sigma }_i/{\sigma }_1$, Eq. (11) (computed with delay reconstruction dimension $K=20$; $log$ denotes the logarithm with base 10), of $D\mathbf{G}$ computed with respect to all variables ($D=3$) and parameters ($N_{\text{p}}=4$) of the Colpitts oscillator, Eq. (37), where $x_1$ is assumed to be measured, Eq. (38). For each $\tau$ the singular values ${\sigma }_i$ are computed for ${10}^4$ points on the attractor using $\mathbf{p}=(5,0.08,0.7,6.3)$. The smallest normalized singular values ${\sigma }_7/{\sigma }_1$ is of magnitude smaller than ${10}^{-15}$ for (almost) all states and $\tau$ and, hence, is numerically very close to zero, indicating a one-dimensional null space of $D\mathbf{G}$, Eq. (13).

Figure 4

Histograms (color coded) of the components $v_{\text{B}}\left(w_i\right)$ of a basis vector spanning the one-dimensional (see Fig. 3) null space of $D\mathbf{G}$ (computed with delay reconstruction dimension $K=20$) computed with respect to all variables ($D=3$) and parameters ($N_{\text{p}}=4$) of the Colpitts oscillator, Eq. (37), where $x_1$ is assumed to be measured, Eq. (38). For each $\tau$ the components $v_{\text{B}}\left(w_i\right)$ are computed for ${10}^4$ points from the attractor using $\mathbf{p}=(5,0.08,0.7,6.3)$. Only $v_{\text{B}}\left(x_1\right)$ and $v_{\text{B}}\left(p_3\right)$ are within the interval $[-{10}^{-10},{10}^{-10}]$ for (almost) all states and delay times $\tau$ and numerically very close to zero, indicating that only $x_1$ and $p_3$ are locally estimable.

Figure 5

Histograms (color coded) of the measure of uncertainty $\nu \left(w_i\right)$, Eq. (33) (computed with delay reconstruction dimension $K=20$; $log$ denotes the logarithm with base 10), of all variables ($D=3$) and parameters $N_{\text{p}}=4$ of the Colpitts oscillator where $x_1$ is assumed to be measured, Eq. (38). For each delay time $\tau$ the measure of uncertainty $\nu \left(w_i\right)$ [see Eq. (33)] is computed for ${10}^4$ points on the attractor using $\mathbf{p}=(5,0.08,0.7,6.3)$. On average, $\nu \left(w_i\right)$ is quite large for $x_2, x_3, p_1, p_2$, and $p_4$, and much smaller for $x_1$ and $p_3$, indicating a good local estimability only for $x_1$ and $p_3$.

Figure 6

State and parameter estimation of the Colpitts oscillator Eq. (37) via measurement function $h\left(\mathbf{x}\right(t\left)\right)$, Eq. (38), from the data time series $\left\{\eta \left(t_n\right)\right\}$, Eq. (40). (a) Output of $h\left(\mathbf{x}\right(t\left)\right)$ (blue line) matches the data $\left\{\eta \left(t_n\right)\right\}$ (light green line). (b),(d),(e) Estimated model variables $x_i$ (blue line) and the true solutions $z_i, i=1,2,3$, (red dashed line) used to generate $\left\{\eta \left(t_n\right)\right\}$. (c) Zoomed version of (b). Only $x_1$ matches its true solution $z_1$. Parameters are estimated to $\mathbf{p}=(5573.7,1.62·{10}^{-4},0.700,2.79)$. That is, only $p_3$ was estimated to the value used to generate the data.

Figure 7

Histograms (color coded) of the correlation coefficients $\rho (w_i,w_j)$, Eq. (34) (computed with reconstruction dimension of $K=20$), of all variables and the parameters $p_1, p_2$, and $p_3$ of the Colpitts oscillator, Eq. (37), where $x_1$ is assumed to be measured, Eq. (38). For each delay time $\tau$ the coefficients $\rho (w_i,w_j)$ are computed for ${10}^4$ points on the attractor. In all plots it is $-1<\rho (w_i,w_j)<1$ for many states and $\tau$ indicating not very strong local dependencies between the quantities.

Figure 8

Similar to Fig. 5, but the measure of uncertainty, $\nu \left(w_i\right)$, was only computed for $x_1, x_2, x_3, p_1, p_2$, and $p_3$ (without $p_4$). For all quantities $\nu \left(w_i\right)$ is very small (on average), indicating that states and parameters can be estimated correctly. $log$ denotes the logarithm with base 10.

Figure 9

State and parameter estimation (using the algorithm from Sec. 3) of the Colpitts oscillator, Eq. (37), from a clean $x_1$ time series, Eq. (38). (a)–(c) To generate the data, the parameter values $p_1, p_2, p_3$, indicated by the red dashed lines, and $p_4=6.3$ [black vertical line in (a)–(d)] were used. For each fixed $p_4$ the other parameters (blue dots) and the model variables were estimated (concept of profile likelihood [39, 40]). (a),(b) The local dependency of $p_1$ and $p_2$ on $p_4$ around $p_4=6.3$ is consistent with the dependency analysis (see Fig. 4) since $\mathrm{\Delta }{p}_4>0\Rightarrow \mathrm{\Delta }{p}_1,\mathrm{\Delta }{p}_2<0$ and vice versa. (c) $p_3$ is estimated correctly independent of $p_4$ (consistent with the dependency analysis). (d) Cost function $C$, Eq. (36), is very small at the estimated solutions, indicating a relatively flat valley in the minimum.

Figure 10

Histograms (color coded) of the normalized singular values ${\sigma }_i/{\sigma }_1$, Eq. (11) (computed with delay reconstruction dimension $K=50$; $log$ denotes the logarithm with base 10), using all variables ($D=3$) and parameters ($N_{\text{p}}=7$) of the Rössler-model, Eq. (41), where $x_2$ is assumed to be measured, Eq. (42). For each delay time $\tau$ the normalized singular values ${\sigma }_i/{\sigma }_1$ of $D\mathbf{G}$ are computed for 1000 states from the attractor. The two smallest normalized singular values ${\sigma }_9/{\sigma }_1$ and ${\sigma }_{10}/{\sigma }_1$ are of magnitude smaller than ${10}^{-15}$ for (almost) all states and $\tau$ and, hence, are numerically very close to zero. This indicates a two-dimensional null space of $D\mathbf{G}$ and, therefore, the existence of redundant variables or parameters.

Figure 11

Histograms (color coded) of the normalized singular values ${\sigma }_i/{\sigma }_1$ of the square matrix ${\mathbf{V}}_{\text{B},1}$, Eq. (44), using all variables ($D=3$) and parameters ($N_{\text{p}}=7$) of the Rössler model, Eq. (41) (based on the same states, delay times $\tau$, etc., as in Fig. 10; $log$ denotes the logarithm with base 10). Since for all states and $\tau$ the smallest normalized singular values ${\sigma }_2/{\sigma }_1$ are relatively large, ${\mathbf{V}}_{\text{B},1}$ is not singular and has, therefore, a unique inverse.

Figure 12

Histograms (color coded) of the components of the first basis vector ${\tilde{\mathbf{v}}}_{\text{B}}^{\left(1\right)}$ (logarithm with base 10), Eq. (46), of the two-dimensional null space of $D\mathbf{G}$ using all variables ($D=3$) and parameters ($N_{\text{p}}=7$) of the Rössler model, Eq. (41) (based on the same states, delay times $\tau$, etc., as in Fig. 10). (c),(e),(h) Only the components associated with $x_3, p_2$, and $p_5$ are nonzero for all states and $\tau$. (a),(b),(d),(f),(g),(i),(j) All other components are within the interval $[-{10}^{-10},{10}^{-10}]$ for (almost) all states and delay times $\tau$ and, hence, numerically very close to zero.

Figure 13

As Fig 12, but histograms of the components of ${\tilde{\mathbf{v}}}_{\text{B}}^{\left(2\right)}$, Eq. (46), are shown. (a),(c),(d),(e),(f),(j) Only the components associated with $x_1, x_3, p_1, p_2, p_3$, and $p_7$ are nonzero for all states and $\tau$. (b),(c),(g),(h),(i) All other components are within the interval $[-{10}^{-10},{10}^{-10}]$ and, hence, numerically very close to zero for (almost) all states and delay times $\tau$.

Figure 14

Histograms (color coded) of the normalized singular values ${\sigma }_i/{\sigma }_1$, Eq. (11) (computed with delay reconstruction dimension $K=20$; $log$ denotes the logarithm with base 10), using all variables ($D=3$) and parameters $N_{\text{p}}=5$ of the HR model, Eq. (51), where $x_1$ is assumed to be measured, Eq. (52). For each $\tau$ the singular values ${\sigma }_i$ are computed for ${10}^4$ points on the attractor using $\mathbf{p}=(3,5,0.004,3.19,0.25)$. The smallest normalized singular value ${\sigma }_8/{\sigma }_1$ is of magnitude ${\sigma }_8/{\sigma }_1\approx {10}^{-7}$ and, hence, numerically greater than zero.

Figure 15

Histograms (color coded) of the measure of uncertainty $\nu \left(w_i\right)$, Eq. (33) (computed with delay reconstruction dimension $K=20$; $log$ denotes the logarithm with base 10), of all $D=3$ variables and all $N_{\text{p}}=5$ parameters for the HR-model, Eq. (51), where $x_1$ is assumed to be measured, Eq. (52). For each delay time $\tau , \nu \left(w_i\right)$ are computed for ${10}^4$ points on the attractor using $\mathbf{p}=(3,5,0.004,3.19,0.25)$. (a)–(f) Uncertainties of $x_1, x_2, x_3, p_1, p_2$, and $p_3$ converge (on average) to comparatively small values for larger $\tau$. (g),(h) Uncertainties of $p_4$ and $p_5$ converge (on average) to comparatively larger values, indicating a worse local estimability compared to the other quantities.

Figure 16

Histograms (color coded) of the correlation coefficients ${\rho }_{ij}$ (computed with $K=20$) of all variables and parameters of the HR model, Eq. (51), where $x_1$ is assumed to be measured, Eq. (52). For each $\tau$ the correlation matrix $\mathbf{\rho }$, Eq. (34) is computed for ${10}^4$ points on the attractor using $\mathbf{p}=(3,5,0.004,3.19,0.25)$. (a)–(a1) The quantities are not very correlated, indicating that they should be estimable from a $x_1$ time series. (b1) There is a strong, but not perfect, negative correlation between $p_4$ and $p_5$.

Figure 17

State and parameter estimation (using the method from Sec. 3) of all $D=3$ variables and all $N_{\text{p}}=5$ parameters of the HR-model, Eq. (51), from a noisy $x_1$ time series $\left\{\eta \left(t_n\right)\right\}$, Eq. (53), using the measurement function $h\left(\mathbf{x}\right)$, Eq. (52). (a) Output $h\left(\mathbf{x}\right)$ (blue line) matches the data $\left\{\eta \left(t_n\right)\right\}$ (green circles). (b),(c),(d) Estimated model variables $x_i$ (blue lines) match the true solutions $z_i$ (red dashed lines; used to generate $\left\{\eta \left(t_n\right)\right\}$). Parameters are estimated to $\mathbf{p}=(3.00,4.99,0.00402,4.19,0.191)$. That is, $p_1, p_2$, and $p_3$ are estimated to values similar to those used to generate the data. $p_4$ and $p_5$ are estimated to much less accurate values.

Figure 18

State and parameter estimation (using algorithm in Sec. 3) of the HR model, Eq. (51), from a clean $x_1$ time series. (a)–(d) The parameter values shown by the red dashed lines were used, beside $p_4=3.19$ [black vertical line in (a)–(e)], to generate the data. For each fixed $p_4$ all other parameters were estimated (blue dots) beside the model variables (profile likelihood approach [39, 40]). That $p_5$ is estimated to a too-small value, if $p_4$ is too large, is consistent with the negative correlation between both parameters; see Fig. 16(b1). Nevertheless, $p_1, p_2$, and $p_3$ are estimated (almost) correctly to values used to generate the data, (almost) independent of the value of $p_4$. (e) Although cost function $C$, Eq. (36), is very small for all $p_4, C$ exhibits a minimum around $p_4\approx 3.19$ (used to generate the data).