Text mixing shapes the anatomy of rank-frequency distributions

Natural languages are full of rules and exceptions. One of the most famous quantitative rules is Zipf's law, which states that the frequency of occurrence of a word is approximately inversely proportional to its rank. Though this “law” of ranks has been found to hold across disparate texts and forms of data, analyses of increasingly large corpora since the late 1990s have revealed the existence of two scaling regimes. These regimes have thus far been explained by a hypothesis suggesting a separability of languages into core and noncore lexica. Here we present and defend an alternative hypothesis that the two scaling regimes result from the act of aggregating texts. We observe that text mixing leads to an effective decay of word introduction, which we show provides accurate predictions of the location and severity of breaks in scaling. Upon examining large corpora from 10 languages in the Project Gutenberg eBooks collection, we find emphatic empirical support for the universality of our claim.

Figure 1

(a) An idealization (black points) of a rank-frequency distribution (gray points) for a single text (The Complete Historical Romances of Georg Ebers) from the English eBooks collection. Idealization is defined by a pure power law of scaling $1-N/M$ where $N$ is the number of unique words and $M$ is the total volume of words and $N/M$ is an estimate for $\alpha$ [red (gray) dashed line, see Sec. 4]. (b) The mixtures of all texts (gray points) and their idealizations (black points) from the English eBooks collection. Note that neither mixture results in a pure power law such as Zipf's [$\theta =1$, red (gray) dashed line].

Figure 2

Top: For each of the 10 deciles of the English distribution of text sizes, we measure the parameters $b, \gamma , N_{\text{avg}}$, and $\theta$ from 50-book sample corpora. Each cloud represents $1000$ sample corpora from deciles 1–10 [low to high, where red to blue (left to right) indicates increasing decile and fading to green or yellow (gray) indicates increasing density]. The line $b=N_{\text{avg}}$ is also presented (dashed line, main axis) and shows that $b$ increases with decile for all but the most extreme (${10}^{\text{th}}$) decile. Main axes insets show parameter variation across deciles for both $b$ and $N_{\text{avg}}$ (left); $\gamma$ and $\theta$ (right), where we note that Zipf's parameter, $\theta$, is the only one that exhibits signs of stationarity. Bottom: Box plots providing a more detailed look at the 10 deciles of the distribution of text sizes. For clarity we have separated the plots for deciles 1–9 from the ${10}^{\text{th}}$. This highlights the extreme nature of the later deciles (most notably the ${10}^{\text{th}}$), where the presence of poorly refined texts throw off estimates of $N_{\text{avg}}$, which we also note corresponds to the roll over in the distributions off of the $b=N_{\text{avg}}$ axis above.

Figure 3

A concrete example of the text mixing effect, where we consider two passages ($T_1$ and $T_2$) as separate texts that are then mixed. The similarity of word use between these excerpts provides an excellent example for understanding the differences between the growing text, where we count new word appearances ($n_m$) with the awareness of previous texts, and the memoryless text, where we count word appearances ($N_m$) as new with each initial appearance in each text. Note that both ${\alpha }_m$ and $A_m$ are simply the quotients of $n_m$ and $N_m$ with $m$ (respectively) and that their quotient (${\alpha }_m/A_m$) is equivalent to $n/N$ and is not equal to 1 only when texts are mixed.

Figure 4

Box plots of the base-10 logarithm vocabulary sizes of the texts contained in the 10 eBooks corpora studied. Center bars indicate means and whiskers extend to most extremal values up to $1.5$ times the interquartile-range length, whereupon more extremal values are plotted as points designated “outliers.”

Figure 5

The results of text mixing experiments for the largest, English corpus from the eBooks collection. The main axes show the empirical, normalized rank-frequency distribution (black points), $p$, and the model determined by text mixing [green (gray) points], $\widehat{p}$. The measured lower and upper exponents, $\gamma$ and $\theta$, are depicted in the lower right and upper left, respectively, with triangles indicating the measured slopes. We also present gray boxes in the main axes to highlight the different mixing regimes, marked by $N_{\text{char}}, N_{\text{min}}, N_{\text{avg}}$, and $N_{\text{max}}$ (see Sec. 4 and Table 1 for complete descriptions of these quantities). The lower left inset shows the pointwise squared error ${[p\left(r\right)-\widehat{p}\left(r\right)]}^2$, whose sum is minimized in the transformation of $\alpha$ into $\widehat{p}$. The upper right inset shows the untransformed rate of word introduction, $\alpha$ (black points), and the decay exponent $\mu$, which is depicted by the regressed slope [green (gray) dashed line].

Figure 6

The results of text mixing experiments for the nine smaller corpora analyzed. All insets, color coding, and labels are consistent with those from the larger, English presentation in Fig. 5, whose caption possesses full descriptions of all axes and plotted data.

Figure 7

Text mixing results for a single-author corpus. Here $\alpha$ was measured for differing refinements of the Egyptological fiction compendium-text The Complete Historical Romances of Georg Ebers into subtexts. All insets, color coding, and labels are consistent with those from the English presentation in Fig. 5, whose caption possesses full descriptions of all axes and plotted data. Left: Each series is considered a separate text. Middle: Each volume of each series is considered a separate text. Right: Each word (the extremal refinement, see Sec. 4) in the compendium is considered a separate text. Note that in the upper right insets, $\alpha$ decreases overall with each refinement (as by definition it must) and that there appears to be an optimal refinement for producing a text mixing model, likely close to the scale of volumes.