The Classical Model of Type-Token Systems Compared with Items from the Standardized Project Gutenberg Corpus

1. Introduction

Type-token systems consist of “types” of object, whose individual occurrences are referred to as “tokens”. In a biological habitat, types might represent species and tokens individual organisms [1,2]. Type-token models have been applied to galactic superclusters [3], the popularity of musical works [4], the sizes of software components [5] and the statistics of written and spoken texts (including dialogues) [6]. In the latter case, unique lexical units (lemmas, wordforms, bi- or trigrams) form types and their specific instances form tokens. The early work of Zipf [7,8], Mandelbrot [9] and Simon [10] have identified underlying power laws, especially the two laws of Zipf and Heaps’ law, whose exponents are supposedly linked by a series of mathematical relationships. We shall call this the “classical model”.

Here we compare this classical model with data extracted from the Standardized Project Gutenberg Corpus (SPGS) [11], which comprises nearly 60,000 works of literature, mostly English, spanning many genres and historical periods. For the purposes of this work, we define types as complete wordforms, regardless of common stems. It thus extends our two earlier two papers [12,13] with a larger and more consistent dataset and demonstrates the ranges of applicability of the classical equations.

2. The Classical Model

Although Zipf’s first law (the “types” law) was named for George Kingsley Zipf following his famous treatise of 1948 [8], it was known at least 35 years before this when it was observed in the distribution of city-sizes [14]. It links the frequency $f$ of a type to its “rank” $r$ ($r=1$ being the most frequent type, $r=2$ the second most frequent etc.):

$$f\left(r\right)\propto {\displaystyle \frac{1}{r^{\alpha }}}$$

(1)

Where $\alpha$ (the “Zipf alpha index”) is typically in the region of 1. Figure 1 shows the log-log frequency vs. rank distributions for two typical English texts, showing that $\alpha$ changes significantly between different ranges of $r$. Nevertheless, both exhibit a “middle range” ($100\lesssim r\lesssim 1000$) over which α is approximately constant. The reduced log-log slope for small $r$ is embraced by the Zipf-Mandelbrot law $f\left(r\right)\propto {\displaystyle \frac{1}{{\left(r+m\right)}^{\alpha }}}$ [9] where $m$ is an additional constant, which is asymptotically equivalent to (1) when $r\gg m$.

If word type selection were statically independent, with a probability distribution $\propto 1/r^{\alpha \left(r\right)}$, normalization would require that $\alpha \left(r\right)>1$ at the upper extremity of $r$, though $\alpha \le 1$ may exist in other regions. Indeed, from an aggregate frequency/rank plot of over 2,606 English books, Montemurro [15] observed $\alpha \approx 2.3$ for $r>\mathrm{10,000}$, an effect also seen in the BNC corpus [16] and the Gutenberg corpus for $r>\mathrm{100,000}$ [17]. It has been suggested that the switchover between low and high $\alpha$ domains may correspond to the transition from a common kernel lexicon to vocabularies used for specific communications; the latter limited only by the capacity of the human brain [16]. The graphs of Figure 1 relate to individual 100,000-word texts, where discrete plateaux in their upper tails (representing $f=\mathrm{1,2},3$… etc. [18]) are difficult to interpret in terms of (1). However, there is a strong suggestion of downward “droop” and hence increasing $\alpha$ with increasing $r$.

Zipf’s second (or “size”) law relates the token frequency $f$ to the number of types $n\left(f\right)$ exhibiting that frequency (i.e. the widths of the low frequency plateaux in Figure 1):

$$n\left(f\right)\propto {\displaystyle \frac{1}{f^{\beta }}}$$

(2)

where the $\beta$, the “Zipf beta index”, is typically between 1 and 2. Like the first law, it predates Zipf’s work and was noted by Corbet in a 1941 a study of Malayan lepidoptera [19]. Figure 2 shows Corbet’s data, indicating $\beta \approx 0.91$ across $1\le f\le 10$, an abnormally low value compared with that of the more typical distribution of the King James Bible also shown ($\beta \approx 1.36$). Simon [10] noted that for both words and biological genera, ${\displaystyle \frac{n\left(1\right)}{n\left(2\right)}}\approx 3$ ($\beta \approx 1.58$), $n\left(1\right)$ comprising approximately half the token population. We note from Figure 2 that while the log-log slopes are approximately constant at low frequencies, extrapolating the law for $f>10$ mostly overestimates the data, suggesting a general log-log convexity or an increasing $\beta$ with increasing frequency. Many authors talk of a“low frequency cut-off $a$ (the law only valid for $f\ge a$), estimated 6, 8 and 51 for different English novels [20]. In view of this, one might suppose that Zipf’s second law is only truly valid in the noisy discretized part of the graph, and the seemingly smooth section is in fact the anomaly.

The third power relationship is Heaps’ (or Herdan’s) law, which is an observed sublinear increase in the total number of types $v\left(t\right)$ as the total number of tokens t increases [21]

$$v\left(t\right)\propto t^{\lambda }$$

(3)

where the constant $\lambda$, the “Heaps index”, lies between zero and one. Figure 3 shows how this law holds roughly across at least two decades for two randomly selected documents (one English, the other Finnish).

Although an unbounded increase (both of hapaxes and total types) suggests an infinite accessible vocabulary, this cannot literally be true: a 20-year-old English speaker knows approximately 42,000-word lemmas [22] while the OED of 2020 has 171,476 [13]. Estimating that English lemmas have around four common variations and allowing for proper names and spelling alternatives, the maximum English vocabulary could contain around a million unique wordforms. However, no English text even approaches this: the entire King James Bible has only 12,143 unique wordforms and the aggregate corpus analyzed by Montemurro [15] contains no more than 448,359. The latter comprises 2,626 volumes, each with its own unique jargon and terminology; one cannot suppose that this comes close to exhausting all possible wordforms, especially as language constantly evolves with the introduction of new words (neologisms). Despite this, $v\left(t\right)$ vs. $t$ saturation is seen in ideogrammic languages like Chinese where the dictionary of semantic characters does have a finite limit [23].

It is often claimed (e.g. [24]) that (2) and (3) are inevitable consequences of (1) with the two indices inextricably related

$$\beta =1+{\displaystyle \frac{1}{\alpha }}.$$

(4)

Similar relationships exist between the Zipf $\alpha$ and Heaps indices:

$$\lambda ={\displaystyle \frac{1}{\alpha }}$$

(5)

and the Zipf $\beta$ and Heaps indices:

$$\lambda =\beta -1.$$

(6)

Under certain assumptions and approximations, these equations are reasonably self-consistent. Lü et al. [25] (amongst others) take (1) as the most fundamental law, describing the selection probability for different types, each characterized by an underlying “rank” $r$ (not necessarily its rank within any given sample). If this is approximated by a continuous probability density function

$$f\left(r\right)={\eta r}^{-\alpha }$$

(7)

(where $\eta$ is a constant) then the frequency range $\delta f$ associated with a small number $\delta r$ of contiguous ranks is given by

$$\delta f=\eta \left[r^{-\alpha }-{\left(r+\delta r\right)}^{-\alpha }\right]\approx \eta \alpha r^{-\alpha -1}\delta r.$$

(8)

Solving (7) for $r$ and substituting into (8) gives ${\displaystyle \frac{\delta r}{\delta f}}\approx {\displaystyle \frac{1}{\alpha }}{\eta }^{{\displaystyle \frac{1}{\alpha }}}{f}^{-\left(1+{\displaystyle \frac{1}{\alpha }}\right)}$, which approximates the number of types (ranks) associated with a single frequency $f$ ($\delta f=1$). We thus obtain expressions (2) and (4). Some workers (e.g. [3,26]) have taken the opposite approach, making (2) the fundamental law and inferring (1) and (4), though the analysis is largely equivalent. As for Heaps’ law (3), several quite rigorous proofs exist (e.g. [27]) but the following provides a useful intuition [25]: loosely interpreting $v\left(t\right)$ as the rank at which $f=1$, we note that (1) can be rewritten $f\left(r\right)={\displaystyle \frac{v^{\alpha }}{r^{\alpha }}}$. Since the token population $t$ must be approximately the integral of this function across $r=1..v$ we find that

$$t\approx {\int }_1^{v}f\left(r\right)dr={\displaystyle \frac{v^{\alpha }-v}{\alpha -1}}.$$

(9)

Although this is not analytically solvable for $v$, if $v^{\alpha }\gg v$ then $v\left(t\right)\approx {\left(\alpha -1\right)}^{{\displaystyle \frac{1}{\alpha }}}{t}^{{\displaystyle \frac{1}{\alpha }}}$ which gives (3) and (5). Finally (6) is obtained by a simple substitution of (5) into (4).

The most glaring assumption so far is that the continuous probability distribution (7) provides an adequate approximation for what is actually a discrete stochastic process. Discrete analysis based on fixed $\alpha$ and unlimited word types [28] shows that although (3) and (5) are valid, (2) and (4) are asymptotically true only for large $f$ [13,29]. At smaller frequencies, $\beta$ is predicted to increase with decreasing $f$, contradicting a general observation that $\beta$ decreases slightly as $f$ approaches 1 [13] (mirroring the low frequency “droop” observed in Figure 1). This effect can be duplicated in the model by imposing an upper limit on the number of accessible types (a “closed vocabulary”), but this is clearly an artificial “fudge” [13]. Furthermore, when this model is optimised to fit a random selection of texts, the resulting low frequency $\beta$ exhibits a greater variability than the corresponding directly measured value. When extended beyond the sample to which it was optimised, the model must inevitably predict a rapid decrease in the numbers of low-ranking types as potential new word-types are exhausted. This clearly contradicts the common observation that the numbers of hapax legomena (types represented by only one token) always increase without saturation, as required by Heaps’ law.

A second assumption is that word selection is an independent random process governed by a statical probability distribution. Common sense tells us this cannot be true, since the sample space for each successive word is generally dictated by its predecessors [30]. There are also empirical reasons to be suspicious: during population growth, observed ranks change much more frequently than the model would predict, and their relative frequencies do not always converge towards stable values [24]. Tria et al. [17] point out that (5) is only true under random word selection when $\alpha >1$, and $\alpha \le 1$ would require $\lambda \approx 1$.

There have been many attempts to develop all three laws from an underlying dynamical framework in which the type-selection probability is not static (e.g. [26,31]). These are mostly variants of the Pólya urn model, in which the probability of a type’s future selection increases in proportion to its past selection; the so-called “rich get richer” principle. As early as 1955 Simon [10] showed that if the selection probability of a word type with frequency $f$ is proportional to $n\left(f\right)$, and the generation probability of new word types is constant, then an approximation to Zipf’s second law emerges. This idea has been developed and expanded over several decades, recently by Tria et al. [17] who suggested that the appearance of a new type triggers expansion into an “adjacent possible”, introducing further types not hitherto accessible.

We note these models for the sake of completeness but do not attempt to expand upon them in this paper. We concentrate instead on the equations themselves and their relationship with observation.

3. Materials and Methods

As previously noted, $\alpha$ and $\beta$ are not a truly constant but vary between different sections of their respective distributions. To investigate the variation of $\beta$ with frequency we define the seven frequency ranges shown in Figure 4, overlaid upon a typical size/frequency distribution (the Project Gutenberg item PG10, the King James Bible). While the log-log slope ${\beta }_1$ over Range 1 (frequencies 1-10) is clearly perceived, the same cannot be said of Range 7 (frequencies 64-640) where the underlying continuous curve is obscured by discretization and random noise. Nevertheless, the Maximum Likelihood (ML) method [32] gives optimum ${\beta }_1\dots {\beta }_7$ for Ranges 1 to 7 respectively. Previous observations (see Section 2) suggest a degree of log-log convexity, meaning that (2) will not be strictly valid even within an individual range: nevertheless, if we assume that it is approximately valid then the optimized ${\beta }_s$ may be considered representative of range $s$ and compared with the values obtained for other ranges.

The following procedure is used to select a “clean” subset of this SPGC, allowing general statistics to be studied with minimal disturbance from unrepresentative and anomalous phenomena.

1. To eliminate document length as an interfering variable, all items with less than 100,000 word tokens are rejected. The remainder are trimmed to exactly 100,000 tokens each.

2. The item PG4656 “Checkmates for Four Pieces” by W.B. Fishburne is removed. (This consists almost entirely of chess notation, which is not amenable to iterative process of optimizing maximum likelihood).

3. While most of the remaining data follow a clear “main sequence” (a term we borrow from star classification [33]), anomalies still exist in the $\beta$ data. To classify these anomalies consistently we define $k_{i,j}={\prod }_{s=1}^{R}K\left({\beta }_{s,i},{\beta }_{s,j}\right)$ where $K\left(x,y\right)=e^{-{{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{x-y}{\sigma }}\right)}^2}$ is a Gaussian “support” function with standard deviation $\sigma$ and ${\beta }_{s,i}$ is the Zipf index computed for item $i$ over the frequency range $s=1..R$ ($R$ in this case being 7). For item $i$, the average support from the other $n-1$ items $k_i={\displaystyle \frac{1}{n-1}}{\sum }_{j=1}^n{k}_{ij}{1}_{i\ne j}$, where $n$ is the total number of items.

4. If $k_i<\epsilon$ (where $\epsilon$ is some arbitrary threshold) then item$i$ is classified as anomalous.

Figure 5 shows the cumulative distribution of $k_i$ for four possible values of $\sigma$. The classification clearly depends on $\sigma$ as well as $\epsilon$, and while the choice is ultimately subjective, we use $\sigma =0.15$ since it creates a clear dichotomy at $\epsilon =0.00001$.

Of the 57,713-item SPGC, approximately 16% survive this winnowing process and are included in our main study. Table 1 lists the seven anomalies identified in stage 3, along with the seven highest-scoring items.

Figure 6 shows some example plots of $\beta$ for different frequency ranges. We note that while some anomalies appear close to the non-anomalous data, this is an illusion of perspective in 7-dimensional space; the rightmost column of Table 1 shows the shortest Euclidean distance of any anomaly to its nearest neighbour is 0.594, while for the seven highest scoring items this is at least an order of magnitude lower.

This procedure only removes anomalies manifested in the $\beta$-data, when the full 100,000 tokens of each item are analyzed. Remaining anomalies appearing in the $\alpha$ and $\lambda$ indices are considered as and when they arise.

4. Observations on the Zipf $\mathit{\beta }$ Indices

Figure 7(a) shows the resulting mean values of ${\beta }_1\dots {\beta }_7$ together with the corresponding upper and lower quartiles. While statistical variation is large, the mean $\beta$ shows a gradually slowing increase with increasing frequency, peaking at a little under 2 before falling very slightly at the highest range 7. This agrees with the long-established log-log convexity of $\beta$ vs. $f$, $\beta$ being smallest for the smallest frequencies (though never becoming negative as a closed vocabulary model would require [13]). Figure 6(b) shows the corresponding KS-significance [34], a measure of how plausibly the distributions follow the power law (1 being the most plausible and 0 the least). Median values and upper and lower quartiles are also shown. Note that when the frequency is low, the median is at 0.7 with an almost maximal interquartile range. As frequency increases, the median rises to a maximum of about 0.9 while the interquartile range narrows. For higher frequencies still, the median falls very slightly with a steady interquartile range; we may suppose that in this region most items are above the “low frequency cut-off” (as defined by Corral et al. [20]). For the combined range 1-640 (ranges 1-7 combined) only 10% of items have KS-significance exceeding 0.01, and the interquartile range cannot meaningfully be plotted on this scale. Zipf’s second law is therefore approximately valid across the individual ranges but generally does not apply across the entire range of frequencies.

Table 2 shows a matrix of the Pearson correlation coefficients between $\beta$-values computed across different frequency ranges: adjacent (overlapping) ranges have a strong positive correlation, while those far apart show a very weak negative correlation. For example, a higher than average ${\beta }_7$ tends accompany a lower than average ${\beta }_1$, which suggests that that beneath the noise, size/frequency distributions differ not so much in their absolute slopes but in their respective log-log convexities.

5. Zipf $\mathit{\alpha }$ Indices in the Middle Range

It has long been recognized that Zipf’s frequency/rank law is most accurate in the “middle range”, with major deviations appearing for high and low ranks [24]. However, it is by no means obvious where the boundaries of this “middle range” lie, nor whether a common definition can be applied to different items. Qualitative examination suggests that many items have an approximately log-log linear range (characterizable by a constant $\alpha$ between ranks 100 and 1000, although many exceptions can be found. We therefore apply the Kolmogorov-Smirnov (KS) test [34] to the ML frequency vs. rank best-fits for ranks 100-1000, for all the non-anomalous items identified in Section 2. Figure 8 shows the cumulative distribution of KS-significance, along with highlighted items whose frequency/rank distributions are shown in Figure 9. We see that for KS significances greater than about 0.01 the graph has tolerable log-log linearity, whereas lower KS values are indicative of an increasing log-log convexity.

Somewhat arbitrarily (based on a qualitative assessment of “what looks right”) we choose a boundary value of 0.03 to classify items into “low” and “high” KS-significance groups, the former comprising the lowest decile. Figure 10 shows the cumulative distributions for mid-range $\alpha$ for these two groups, indicating that the low KS-significance $\alpha$-values are on average significantly higher than those of the low KS-significance group. (We recognise however that $\alpha$-values obtained with a low KS-significance may not be entirely meaningful, since linear spacing of ranks creates a bias towards the upper end of the middle range, where the log-log slope is higher.)

6. Alpha vs. Beta

Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 show the indices ${\beta }_1\dots {\beta }_7$ plotted against the corresponding mid-range $\alpha$ for low and high KS significance items, along with the theoretical curve predicted by (4). The cumulative distributions of $\beta$ are also shown. Figure 18 summarizes the correlation coefficient magnitudes for all seven graphs. Our initial qualitative observations are as follows:

1. For the low frequency ranges the correlation coefficient magnitude is very low. It rises close to unity for range 6, where the frequencies correspond roughly to the middle range ranks across which $\alpha$ was computed. It then falls again in range 7. (See Figure 18).

2. Agreement with (4) generally improves as the frequencies increase, the best fit being for Range 5 (Figure 15). Here items with higher KS-significance are distributed almost symmetrically around the theoretical line, while items with lower KS-significance exist in separate clusters above and below. There is a hint of this behaviour for ranges 4 and 7, though it is curiously it is absent for range 6 (Figure 16).

3. Throughout all frequency ranges there is a clear difference between the mean $\beta$ for high and low KS-significance. While in the low frequency ranges, low KS-significance items have a lower average $\beta$, the difference narrows with increasing frequency. The difference eventually reverses for range 7 (Figure 17), though here there is an intersection of the two cumulative distributions created by the clustering above and below the model.

4. For range 1 (and to some extent 2) there is a distinct cluster of high-$\beta$ points, also characterized by a narrower $\alpha$-range than the main population. These points are numerically quite close to the theoretical curve, though they show no obvious indication of following it. Nearly all these items belong to the high KS-significance group and nearly all of them are Finnish; we therefore refer to this feature as the “Finnish cluster”. (The main cluster centred around $\beta =1.6$ is dominated by English items.)

5. For range 5 there is a distinct “filament” of data points exhibiting low $\beta$, all of which belong to the low KS-significance group.

6.1. The “Finnish Cluster”

Figure 11 shows a prominent cluster of high ${\beta }_1$ items, nearly all of which are Finnish, which belong exclusively to the high KS-significance group. (Finnish items do not appear anywhere else in this graph.) For analytical purposes, we define this cluster as containing all items whose indices fall within the rectangle $2<{\beta }_1<2.2$, $0.95<\alpha <1.1$ (shown on Figure 10) and plot the cluster size as a function of a KS-significance limit between 0.01 and 1 (see Figure 19). We find the cluster only exists for KS-significances above 0.048, beyond which its relative size increases approximately linearly with the increasing logarithm of the KS-significance.

Comparison between the Finnish cluster and the main (mostly English) population suggests significant differences between the statistical properties of languages. English is well known for its limited inflexion and agglutination, while Finnish is heavily agglutinated and inflected (more so even than German), giving it a much larger variety of wordforms. Our results also suggest that compared to English, Finnish has a closer adherence to Zipf’s first law (evidenced by the high KS-significance) and a significantly higher $\beta$-index for lower frequencies. We intend to investigate this further, along with the statistical differences between other languages.

6.2. The Middle Range and the “Low Beta Filament”

Here we examine the filament of low ${\beta }_5$, low KS-significance data points seen in Figure 15 (and to a lesser extent 16). In this range the frequencies correspond most closely to the middle range of ranks across which the $\alpha$-indices were measured. This is evidenced by the close adherence (at least amongst the high KS-significance data) to the model (4). To obtain a more precise correspondence we calculate the average upper and lower frequencies corresponding to the ranks 100 and 1000 across the entire dataset and Figure 20(a) and (b) show the $\beta$ values recomputed for this range. The “low beta filament” is visible in Figure 20(a), though maybe not quite so clearly as in Figure 15. This filament comprises items of the lowest KS-significance, including those less than ${10}^{-10}$ highlighted in Figure 20(c). It includes many editions of the CIA World Factbook and other works of reference. Meanwhile the highest KS-significant data in Figure 20(d) are those in closest agreement with (4).

7. Vocabulary Growth and Heaps’ Law

So far we have applied Zipf’s first and second laws to documents with a fixed length of 100,000 word tokens. Meanwhile Heaps’ law (3) describes the growth of documents and is theoretically linked to Zipf’s laws by (5) and (6). An optimum Heaps’ index was obtained for each item by iteratively adjusting $\lambda$ to minimize the mean square difference between (3) and the measured vocabulary profile across 100 intervals of 1000 tokens each. Figure 21 shows the cumulative frequency distributions for optimized Heaps’ indices obtained for items with high and low KS-significance. Our first observation is that for the low KS-significant items, the average Heaps index is significantly lower than for the high KS-significant items, with a larger standard deviation.

Figure 22, Figure 23, Figure 24, Figure 25, Figure 26, Figure 27 and Figure 28 show $\lambda$ plotted against the $\beta$ indices for the seven frequency ranges, compared with the prediction of (6). (We again separate items into the high and low KS-significance groups.) For range 1 (Figure 22) the data in both groups are strongly correlated. The bulk of points agree roughly with the theoretical curve (6), although items in the “Finnish cluster” (which have a considerably higher than average $\lambda$) lie well above the theoretical prediction. Figure 29 summarises the correlation coefficients. As the frequencies increase, the correlation coefficient falls as correspondence with (6) is gradually lost. This makes sense, since $\lambda$ is associated with the rate at which new types (types with frequency 1) appear and would be mostly associated with the lowest frequency statistics.

However, vocabulary growth is also retarded by the reappearance of existing types, so one would expect some correlation with higher frequency statistics. To investigate this, we propose a simplified model to represent the frequency-rank distribution and obtain the corresponding profile of vocabulary growth. To represent the imperfect adherence to Zipf’s first law we propose an $\alpha$-value which changes abruptly at some rank $r=\rho$, from a high-frequency value ${\alpha }_{hf}$ to a low-frequency value ${\alpha }_{lf}$:

$$f\left(r\right)=\left\{\begin{array}{cc}{\displaystyle \frac{v^{{\alpha }_{lf}}}{{\rho }^{{\alpha }_{lf}-{\alpha }_{hf}}{r}^{{\alpha }_{hf}}}};& r<\rho \\ {\displaystyle \frac{v^{{\alpha }_{lf}}}{r^{{\alpha }_{lf}}}};& r\ge \rho \end{array}\right..$$

(10)

Following (4) we define high and low frequency $\beta$ indices, ${\beta }_{hf}=1+{\displaystyle \frac{1}{{\alpha }_{hf}}}$ and ${\beta }_{lf}=1+{\displaystyle \frac{1}{{\alpha }_{lf}}}$ respectively. Log-log convexity requires that ${\alpha }_{hf}<{\alpha }_{lf}$ so ${\beta }_{hf}>{\beta }_{lf}$.

Since high- and low-frequency $\beta$ have no meaningful correlation (see Table 2) we generate ${\beta }_{hf}$ and ${\beta }_{lf}$ as independent Gaussian random variables. Substituting (10) into (9) we obtain the corresponding expression for the token population $t$:

$$t={\displaystyle \frac{v^{{\alpha }_{lf}}}{{\rho }^{{\alpha }_{lf}-{\alpha }_{hf}}}}{\displaystyle \frac{{\rho }^{1-{\alpha }_{hf}}-1}{1-{\alpha }_{hf}}}+v^{{\alpha }_{lf}}{\displaystyle \frac{v^{1-{\alpha }_{lf}}-{\rho }^{1-{\alpha }_{lf}}}{1-{\alpha }_{lf}}}$$

(11)

allowing $v$ to be computed iteratively for any value of $t$. From the profile of $v$ vs. $t$ (using 100 steps of 1000 tokens) optimal values of $\lambda$ obtained by minimizing the mean square error. While this model is not precise, it nevertheless provides an insight into the observed behaviour of real texts. We generate values of ${\beta }_{hf}$ and ${\beta }_{lf}$ with means 1.9 and 1.6 respectively (corresponding roughly to the middle values in Figure 11 and Figure 17), with both standard deviations set to 0.15. Combinations for which ${\beta }_{hf}-{\beta }_{lf}<0.15$ are rejected and recomputed to ensure a minimum degree of log-log convexity. The transitional rank $\rho$ is set to 1000, to correspond with the upper limit of the middle range as defined in Section 5.

Figure 30 shows scattergrams of (a) ${\beta }_{lf}$ and (b) ${\beta }_{hf}$ plotted against the optimised Heaps index $\lambda$: while the latter shows little correlation with $\lambda$ (as in the experiment), the former is strongly correlated and approximately follows (6) (though deviating somewhat for large $\lambda$). Statistical scatter is far less than observed in Figure 22, though this is to be expected with a simplified model with far fewer variables than the reality. The parallelism between the base of the data clusters in Figure 25 and Figure 26 is reproduced in Figure 30(b) where it is clearly related to log-log convexity (i.e. ${\beta }_{hf}<{\beta }_{lf}$; note that if ${\beta }_{hf}={\beta }_{lf}$ then (11) would become identical to (9) and all points would lie on the theoretical line).

8. Discussions and Conclusions

This paper examines the agreement between data taken from the Standardized Project Gutenberg Corpus (SPGC) and the classical type-token equations used by (amongst many others) Kornai [24] and Lü et al. [25]. Items selected for study are all truncated to exactly 100,000 word tokens, and the Zipf $\beta$ indices are computed across seven logarithmically frequency ranges. Outliers are identified in terms of their proximity to other data-points by means of a Gaussian support function and eliminated from the main study. Our findings can be summarised as follows.

1. Outlying items identified in the $\beta$-data are all dictionaries, while items with the highest support are dominated by religious works.

2. Although wide statistical variations exist, the average $\beta$ index generally increases with increasing frequency, corresponding to a log-log convexity of the types vs. frequency distribution.

3. The $\beta$ indices measured for closely overlapping frequency ranges show a strong positive correlation, while those for frequency ranges widely separated show a weak negative correlation. (For example, if $\beta$ is above average in the frequency range 1 to 10, then it is likely to be below average in the range 64-640,)

4. Adherence to Zipf’s first law across the middle range (defined as the interval between ranks 100 to 1000 inclusive) may be characterised by the Kolmogorov-Smirnoff (KS) significance. Items with a high KS-significance in this region tend to have a slightly lower Zipf $\alpha$ index and a higher $\beta$ index across all but the highest frequency range.

5. When the frequency range used to compute $\beta$ corresponds roughly to the previously defined middle range (Figure 15), the points roughly follow the classical equation (4). Of these, items with the highest KS-significance follow the equation most closely (Figure 20).

6. Two notably anomalous phenomena appear in the plots of $\beta$ vs. $\alpha$. Firstly, for range 1 (frequencies 1 to 10) there exists a distinct secondary cluster of points, all with high $\beta$ and all of which have high KS-significance (>0.048). Nearly all of these are Finnish. Secondly for frequencies corresponding to the middle range we note a “filament” of low-$\beta$ points, all of which have especially low KS-significance (${<10}^{-10}$). All of the latter are reference books.

7. For the lowest frequency range, the $\beta$ index correlates strongly with the optimised Heaps index $\lambda$ in a manner roughly consistent with the equation (6). (The Finnish items noted in point 6 are an exception.) This correlation gradually disappears as the frequencies increase.

8. A simplified model based upon an abrupt change in $\alpha$ and $\beta$ between high and low frequencies (to create a convex distribution) reproduces the basic features observed in the data. We note that the Heaps’ index is largely dependent on the statistics of the lowest-frequency types and is affected only weakly by the high-frequency indices.

While all these points merit further study, we are particularly fascinated by the dichotomy between the Finnish items and the main body of data. We are furthermore interested to discover if other languages have similar peculiarities and what mathematical models can be used to describe their behaviour. The SPGC has a paucity of Finnish: our selection (based on document length and $\beta$-statistics alone) contains only 165 Finnish items, compared to 7,932 items in English. We therefore intend to compile a much larger corpus of items in Finnish and other languages in order to investigate further.