Model-based evaluation of scientific impact indicators

Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the advantage of avoiding the biases present in real databases, and it allows us to assess which aspects of the model dynamics and which traits of individual researchers a particular indicator actually reflects. We find that the simple average citation count of the authored papers performs well in capturing the intrinsic scientific ability of researchers, regardless of the length of their career. On the other hand, when productivity complements ability in the evaluation process, the notorious $h$ and $g$ indices reveal their potential, yet their normalized variants do not always yield a fair comparison between researchers at different career stages. Notably, the use of logarithmic units for citation counts allows us to build simple indicators with performance equal to that of $h$ and $g$. Our analysis may provide useful hints for a proper use of bibliometric indicators. Additionally, our framework can be extended by including other aspects of the scientific production process and citation dynamics, with the potential to become a standard tool for the assessment of impact metrics.

Figure 1

Functional fits of the MAS data presented in the text: (a) the number of publications for researchers, and (b) the number of coauthors for papers. The fit functions are $\mathcal{G}\left(k\right)=3.48/exp[-{(lnk+3.5)}^2/5.9]$ and $\mathcal{H}\left(d\right)=19.7d/(100+d^{4.6})$, respectively. In the log-log scale, the coefficients of determination ($R^2$) are 0.92 and 0.89, respectively. For the coauthor distribution, the peak at $d=50$ coauthors is due to large-scale collaborations in particle physics and astrophysics, whose members are only partially covered in our data. The underlying Microsoft Academic Search (MAS) data that we present here were collected using the API of the service to obtain unique IDs for the authors of papers published by the American Physical Society (APS) in the years 1893–2009; this was successful for 71% of the APS papers. Excluding self-citations, the resulting data comprise 2 427 367 citations among 326 586 papers authored by 244 538 researchers. Thanks to having unique author IDs, the use of MAS data avoids the common name disambiguation problem in bibliometric data [14], which is vital for the analysis of coauthorship patterns [15].

Figure 2

Comparison of statistical features in the MAS data and in the artificial model: (a) distribution of researcher productivity $k$, (b) distribution of paper number of coauthors $d$, and (c) distribution of paper citation count $c$. In the log-log scale, the coefficients of determination ($R^2$) are 0.92, 0.85, and 0.85, respectively. The shaded areas visualize the variable's standard deviation observed in 100 model realizations. The dependence of paper citation count on paper appearance time with and without applying the stationary normalization value ${\mathrm{\Omega }}_{\infty }$ is shown in panel (d).

Figure 3

The effect of parameter variations on the paper citation distribution: (a) baseline author ability $a_0$ (when $\theta =4$), and (b) paper lifetime $\theta$ (when $a_0=1$). The left panel shows that introducing a value $a_0>0$ helps to make the distribution more concave: while the use of $a_0$ is not essential, $a_0>0$ actually improves the agreement between empirical and model citation distribution. The right panel shows that tuning paper lifetime can be used to calibrate the simulated citation distribution against the real one: in the log-log scale, the coefficient of determination ($R^2$) increases from 0.60 for $\theta =2$ years to 0.85 for $\theta =4$ years. The stationary value for the normalization constant ${\mathrm{\Omega }}_{\infty }$ is $1.33\times {10}^4$ in the basic setting $a_0=1$ and $\theta =4$. For the variations considered here, ${\mathrm{\Omega }}_{\infty }$ values are $1.06\times {10}^4$ and $1.62\times {10}^4$ for $a_0=0$ and 2, respectively, and $6.30\times {10}^3$ and $2.17\times {10}^4$ for $\theta =2$ and 8, respectively.

Figure 4

Comparison of the mean precision values achieved by research impact indicators with respect to different ground truth assumptions: (a) author ability, (b) average fitness of authored papers, (c) author ability times activity, (d) total fitness of authored papers. The horizontal dotted line marks the performance of the best metric (which is typed with bold letters); the error bars show three times the standard error of the mean.