Explaining Zipf's law via a mental lexicon

Zipf's law is the major regularity of statistical linguistics that has served as a prototype for rank-frequency relations and scaling laws in natural sciences. Here we show that Zipf's law—together with its applicability for a single text and its generalizations to high and low frequencies including hapax legomena—can be derived from assuming that the words are drawn into the text with random probabilities. Their a priori density relates, via the Bayesian statistics, to the mental lexicon of the author who produced the text.

Figure 1

Frequency $f_r$ vs rank for the text TF; see Table  and (1). Red (upper, straight) line: the Zipf curve $f_r=0.168r^{-1.032}$. Arrows indicate on the validity range of the Zipf's law. Blue (lower, curved) line: the solution of (13) and (14) for $c=0.168$ and $n=2067$. It coincides with the generalized Zipf law (23) for $r>r_{\min }=36$. The stepwise behavior of $f_r$ for $r>r_{\max }$ refers to hapax legomena.

Figure 2

Main figure: Upper (green) curve: the generalized Zipf's curve (23) written in the scaling form $\mathbf{t}=c[{\mathbf{r}}^{-1}-1]$, where $\mathbf{t}=f_{r}n$, $\mathbf{r}=r/n$, and $c=0.2$. Squares: solutions of (13), (14), and (20) for $c=0.2$. Lower (red) curve: The same as for the upper curve but for $c=0.1$. Rounds: Solutions of (13), (14), and (20) for $c=0.1$. It is seen that the generalized Zipf's curve coincide with the actual solutions. Inset: Solid curve: The same as for solid curves in the main figure but for $c=1.0$. Dots: Solutions of (13), (14), and (20) for $c=1.0$. Now the generalized Zipf's curve does not coincide with the actual solution.