Classification framework for partially observed dynamical systems

We present a general framework for classifying partially observed dynamical systems based on the idea of learning in the model space. In contrast to the existing approaches using point estimates of model parameters to represent individual data items, we employ posterior distributions over model parameters, thus taking into account in a principled manner the uncertainty due to both the generative (observational and/or dynamic noise) and observation (sampling in time) processes. We evaluate the framework on two test beds: a biological pathway model and a stochastic double-well system. Crucially, we show that the classification performance is not impaired when the model structure used for inferring posterior distributions is much more simple than the observation-generating model structure, provided the reduced-complexity inferential model structure captures the essential characteristics needed for the given classification task.

Figure 1

(a) Schematical representation of three nested GnRH signaling model structures ($M1, M2$, and $M3$). The signaling pathway in these model structures is highlighted by the flow of red, blue, and green arrows, respectively. Each model structure is comprised of GnRH signal as the driving input, GSU time series as the measurable output, and one to three compartments along its signaling pathway. (b) Two classes of GnRH model parameter pairs (${K_d}_{T{F}_1}, {K_d}_{T{F}_2}$) that determine subject-specific functional relationship between GnRH pulse frequency and the integrated GSU measure: (1) class of normal subjects with bell-shaped frequency-response relationship (blue diamonds) and (2) class of abnormal subjects with simple frequency dependency of the GSU measure (red disks). These two classes are separated by two straight lines in the log-log parameter space.

Figure 2

(a) Equilibrium probability distribution of states $x$ for four example stochastic double-well systems with $(d,\kappa ,a)=(1.0,1.0,0.1)$ (red solid curve), $(d,\kappa ,a)=(1.3,1.5,-0.1)$ (blue dash-dot curve), $(d,\kappa ,a)=(1.0,1.5,0)$ (red dashed curve), and $(d,\kappa ,a)=(1.2,1.5,0)$ (blue dotted curve). (b) The same as in (a) but for stochastic multiwell systems.

Figure 3

(a) The observed trajectories of ten example stochastic double-well systems from each of two clusters in Fig. 2 (red curves with empty circles vs blue curves with filled circles). The range of each subpanel's vertical axis is scaled to $[-2.5,+2.5]$. The intersample interval (ISI) is 0.5, and the variance ${\sigma }^2$ of Gaussian distributed observation noise is 0.04. (b) The same as in (a) but for ${\sigma }^2=0.36$.

Figure 4

Classification performance as function of log kernel width (i.e. ${log}_{10}\rho$) using Learning in Model Space (LiMS) method to classify Partially Observed GnRH Models. Panel (a): The inter-sample intervals ($ISI$) is fixed 75 while the noise standard deviation varies across the set $\{0.1,0.03,0.01,0.005,0.001\}$ (red squares, blue circles, magenta diamonds, green triangles, and black left triangles, respectively). All observations were sampled on a fixed regular grid over a 7.5h time window. Panel (b): The standard deviation of observation noises is fixed to 0.3 while the $ISI$ value varies across the set $\{15,30,45,75,90\}$ (red squares, blue circles, magenta diamonds, green triangles, and black left triangles, respectively). The observation times are random and the $ISI$-values given are the expected value. Values of the kernel width hyper-parameter for the LiMS classifier were taken from $\{0.0001,0.0005,0.001,0.005,0.01,0.05,0.1,0.5,1,5,10,50\}.$ Note that the kernel width hyper-parameter $\rho$ of LiMS and KME classifier is introduced in Eq. (19) as the scale parameter of Gaussian kernels.

Figure 5

As same as in Fig. 4 but for classification performance as function of log tempering parameter (i.e., ${log}_2\alpha$) using probability product kernel (PPK) method. The tempering PPK hyperparameter took values from $\{1/32,1/16,1/8,1/4,1/2,1,2,4,8\}$. Note that the tempering hyperparameter $\alpha$ of PPK classifier is introduced in Eq. (32).

Figure 6

Same as in Fig. 4 but for kernel mean embedding (KME) classifier.

Figure 7

Relationship between classification performance and model uncertainty measured by average posterior entropy. Each data point corresponds to one of 15 data sets in group 1, group 2, and group 3. The $x$- and $y$ coordinate of each $\diamond , \square$, and $\circ$ point display the average posterior entropy and the accuracy for LiMS, PPK, KME classifiers, respectively.

Figure 8

Scatter plot of the mean and standard deviation pairs $({\mu }_y,{\gamma }_y)$ computed for the time series $\left\{y_t\right\}$ observed (1) from different tasks [from left to right: task 1, task 2, and task 3] and (2) with different ($\sigma , \text{ISI}$) settings [from top to bottom: $(0.2,0.5)$, and $(0.6,0.5)]$. The data points in the scatter plots from class 1 and class 0 are displayed as red diamonds and blue circles, respectively.

Figure 9

Classification performance as function of log kernel width (i.e., ${log}_{10}\rho$) using LiMS classifier to classify partially stochastic double-well systems for different tasks (from top to bottom: task 1–task 3) and for different observation settings [left: $(\sigma ,\text{ISI})=(0.3,0.5), (0.4,0.5)$, and $(0.6,0.5)$ with red squares, blue circles, and black diamonds, respectively; right: $(\sigma ,\text{ISI})=(0.3,0.5), (0.3,1.0)$, and $(0.3,1.25)$ with red squares, blue circles, and black diamonds, respectively)].

Figure 10

The same as in Fig. 9 but for classification performance as function of log tempering parameter (i.e., ${log}_2\alpha$) using probability product kernel (PPK) method.

Figure 11

The same as in Fig. 9 but for KME classifiers.

Figure 12

The same as in Fig. 9 but for bKLR classifiers.