Dependence of exponents on text length versus finite-size scaling for word-frequency distributions

Some authors have recently argued that a finite-size scaling law for the text-length dependence of word-frequency distributions cannot be conceptually valid. Here we give solid quantitative evidence for the validity of this scaling law, using both careful statistical tests and analytical arguments based on the generalized central-limit theorem applied to the moments of the distribution (and obtaining a novel derivation of Heaps' law as a by-product). We also find that the picture of word-frequency distributions with power-law exponents that decrease with text length [X. Yan and P. Minnhagen, Physica A 444, 828 (2016)] does not stand with rigorous statistical analysis. Instead, we show that the distributions are perfectly described by power-law tails with stable exponents, whose values are close to 2, in agreement with the classical Zipf's law. Some misconceptions about scaling are also clarified.

Figure 1

Total number of words $N_L$ with a relative frequency greater than or equal to $k/L$, for varying $L=L_{\mathrm{tot}}/n$, with $L_{\mathrm{tot}}$ the length of the complete text. We have taken the same books as in Ref. [23], (a) Moby-Dick and (b) Harry Potter, exactly reproducing Figs. 2 and 2 in Ref. [23] but also including some additional values of $n$. Deviations from the scaling law are always in the regime of very low frequencies, as expected due to discreteness effects (which are due to the fact that word tokens are discrete).

Figure 2

(a) Probability mass function $D_L\left(k\right)$ of the absolute frequency $k$, for varying subsets of length $L=L_{\mathrm{tot}}/n$ of Harry Potter, displaying a seeming change of shape. (b) The same curves, but plotting $D_L\left(k\right)L{V}_L$ versus $k/L$, as proposed in Ref. [14] and stated in Eq. (2). All curves collapse into a single, length-independent scaling function $g(k/L)$, in agreement with Eq. (2). Note the excellent data collapse: even the deviations for small $k$ are negligible in this case. This is at odds with Eq. (1): a length-dependent exponent in $D_L\left(k\right)$, as proposed by Yan and Minnhagen, is not compatible with the data collapse shown here.

Figure 3

Exponents of the power-law tails as a function of the text length, for seven texts in English, Spanish, and French. Fits are performed as in Refs. [8, 27], with $k_{\mathrm{cut}}$ restricted to the last two decades of the distribution, and accepted for the $k_{\mathrm{cut}}$ that yields a $p$ value larger than 0.20 (calculated from 1000 Monte Carlo simulations). Error bars correspond to one standard deviation. Types are defined at the word-lemma-tag level; more details on text processing are given in Ref. [8]. Note how for $L>3000$ all exponents are below 2.2, and they become quite stable in the range $6000<L<2\times {10}^6$.

Figure 4

$P$ values of two-sample Kolmogorov-Smirnov tests for word-frequency distributions rescaled as explained in the text. One of the samples has text length $L$ and the other $L^{\prime }$, ranging from $L_{\mathrm{tot}}/50$ to $L_{\mathrm{tot}}$. One of the data sets shown corresponds to testing the first fragment of length $L$ with the same fragment of length $L^{\prime }$; for the other data set other fragments are chosen. The texts are the same seven used in the previous figure. $P$ values below 0.05 should lead to the rejection of the scaling hypothesis with a 0.95 confidence level.

Figure 5

Relative error $[N_{L_{\mathrm{tot}}}(\ge k^{\prime })-N_L(\ge k)]/N_L(\ge k)$ of the approximation to $N_L(\ge k)$ given by the scaling of the distribution of the whole text $N_{L_{\mathrm{tot}}}(\ge k^{\prime })$, with $k^{\prime }=k{L}_{\mathrm{tot}}/L$, for Harry Potter. Symbols correspond to the same values of $n$ as in the previous figures.

Figure 6

Linear relation between the second moment of the distribution of word frequencies and the ratio of the squared text length to the vocabulary size, for diverse texts used in the previous figures [8]. A line with a linear coefficient 0.01 is shown for comparison.

Figure 7

Rescaled size distribution $D_L\left(k\right)$ as a function of the rescaled size $k$ for a critical Galton-Watson branching process with a binomial offspring distribution (with a maximum of two offspring) and with different values of the maximum number of generations $L$. A finite-size scaling law, of the form $D_L\left(k\right)=g(k/L^2)/L^3$, holds, but only for $k>10$, roughly. Curiously, when the offspring distribution is geometric (for which the size distribution coincides with the escape time of a random walker moving between an absorbing and a reflecting boundary [21]), the deviation takes the opposite sign (see Ref. [32]). But beyond the smallest values of $k$, the rescaled distributions coincide and universality shows up [44].