The meta book and size-dependent properties of written language

Sebastian Bernhardsson¹, Luis Enrique Correa da Rocha and Petter Minnhagen

Department of Physics, Umeå University, 901 87 Umeå, Sweden

¹ Author to whom any correspondence should be addressed.

E-mail: sebbeb@tp.umu.se

Received 10 September 2009
Published 10 December 2009

Contents

• 1. Introduction
• 2. The meta book concept
• 3. Size dependence of the word-frequency distribution (wfd)
• 4. Size dependence in real books
• 5. Conclusions
• Acknowledgment
• Appendix
• References

1. Introduction

The development of spoken and written language is one of the major transitions in evolution [1]. It has given us the advantage of easily and efficiently transferring information between individuals and even between generations. It could be argued that it is clear why language evolved in general, but it is harder to explain its structure. The structure of language has been studied from as early as the Iron Age in India and is, to this day, a popular subject.

The field was boosted after George Kingsley Zipf, approximately 75 years ago, discovered an empirical law (Zipf's law) [2] describing a seemingly universal property of written language. It states that the number of occurrences of a word in a sufficiently long written text falls off as 1/r, where r is the occurrence–rank of a word (the smaller the rank, the more occurrences) [2]–[6]. This in turn means that the normalized word-frequency distribution (wfd) follows the expression P(k)1/k², where P(k) is the probability of finding a word that appears k times in a text [6]. This empirical law is generally believed to represent a universal property of the wfd, and has inspired the development of several models reproducing this structure [7, 8]. However, empirically, one typically finds that the wfd follows a power-law distribution with an exponent smaller than 2 [9, 10]. It was also reported in [10] that the exponent (commonly denoted as γ) for a power-law description of the wfd seems to change with the length of a text, rather than being constant.

Another property is the number of different (unique) words, N, as a function of the total number of words in a book, M (in this context, a book is a sequence of words, where words are defined as collections of letters separated by spaces). The conventional way of describing this relation is by using Heap's law [11], that states that NM^α, where 0 < α < 1 is a constant.

In this paper we present, and support with evidence, a meta book concept that is an abstract picture of how an author writes a text. We suggest a systematic text-length dependence for the wfd which is directly connected to an extended Heap's law with an α changing from 1 to 0 as the text length is increased from M = 1 to infinity.

2. The meta book concept

We start by studying the above-mentioned property, N(M). Figure 1(a) shows this curve for three different authors (Hardy, Melville and Lawrence). We have created very large books by attaching novels together, in order to extend the range of book sizes (see the appendix for a full list of books). The curve shows new words being added at a decreasing rate, which means that N grows more slowly than linear (α < 1) [6, 10]. Also, for a real book, N(M = 1) = 1, which means that the proportionality constant in Heap's law should in fact be one. So, if N = M^α, then α = lnN/lnM. Note that α(M) = lnN/lnM is the average slope of the N(M) curve up to a given point in a log–log scale, which differs from the local slope given by α(M) = dlnN/dlnM. The representation N = M^α(M) means that each point on the N(M) curve is, in log–log scale, identified as the endpoint of a straight line from the origin, where the slope of the line is given by α(M) (see figure 1(b)). This is of course a completely general representation of a single-valued function. However, the present case of word frequencies constrains the values of α: α(M = 1) = 1, and the limit N(M) = N_max corresponds to the limit α(M)1/lnM, so that α(M) is for word frequencies a monotonically decreasing function from 1 towards 1/lnM for large M. The quantity α(M) is plotted in the inset of figure 1 and the data show that α is decreasing as a function of the size, ruling out the possibility of accurately describing the N(M) curve using a constant α.

When the length of a text is increased, the number of different words is also increased. However, the average usage of a specific word is not constant, but increases as well. That is, we tend to repeat words more when writing a longer text. One might argue that this is because we have a limited vocabulary and when writing more words the probability of repeating an already-used word increases. But, at the same time, a contradictory argument could be made that the scenery and plot, described for example in a novel, are often broader in a longer text, leading to a wider use of one's vocabulary. There is probably some truth in both statements, but the empirical data seem to suggest that the dependence of N on M reflects a more general property of an author's language.

For every size of text, the average occurrence for a word can be calculated as k = M/N. This means that the N(M) curve can be converted into a curve for the average frequency as a function of M, . This curve is shown in figures 2(a)–(c) for the three different authors. Each point represents a real book or collection of books, and the curves represent the k_M curve for the full collection of books for each author (i.e. the same data as in figure 1). The data are plotted as 1/k as a function of 1/M in order to explore the asymptotic behavior as M reaches infinity.

Figure 2. Evidence in favor of the meta book concept. (a)–(c) Average frequency for a word as a function of the size of the book (M) plotted as for the three authors. The long-dashed, short-dashed and dotted curves correspond to the N(M) curve as for the biggest collection of books by each respective author. The k for a small book is close to the same as for a section (of the same size) of the bigger book. (d)–(f) Word-frequency distribution for an nth part (triangles) of the full collection of books (filled circles) compared to a small book (open circles) of the same size as the nth part. The wfd is approximately the same for a small book as for a section (of the same size) of a big book.

The overlap between the line and the points means that the average frequency of a word (and consequently also N) in a short story is to a good approximation the same as for a section of equal length from a larger text written by the same author. Note that the texts have to be written by the same author since the overlap would not be nearly as good if books by Lawrence were compared against Melville's curve.

In figures 2(d)–(f), we extract sections from a very large book and compare the results to those from a much smaller book (with a size difference of a factor n). The figures show the wfd for an nth part (averaged over 200 sections) of the full collection and for a short story by the same author. The distribution for the full collection is also included for comparison. The overlap between the short story and the section of the large book implies that the wfd for a text can be recreated by taking a section of a larger book written by the same author. It does not matter if we pull out half of a book of size M, or a quarter of a book of size 2M.

These findings lead us towards the meta book concept. The writing of a text can be described by a process where the author pulls a piece of text out of a large mother book (the meta book) and puts it down on paper. This meta book is an imaginary infinite book that gives a representation of the word-frequency characteristics of everything that a certain author could ever think of writing. This has nothing to do with semantics and the actual meaning of what is written, but rather concerns the extent of the vocabulary, the level and type of education, and the personal preferences of an author. The fact that people have such different backgrounds, together with the seemingly different behavior of the function N(M) for the different authors, opens up the speculation that everyone has a personal and unique meta book, in which case it can be seen as an author's fingerprint.

Yet another, more obvious, property is the frequency of the most common word, k_max. When dividing a book in half, k_max should also be cut in half. This linear relation between k_max and M is shown in figure 3 to be in agreement with the real data, which is consistent with the meta book concept. This follows because the most common word is most likely a `filling word' (e.g. `the'), which would be evenly distributed throughout the text (e.g. every 20th word or so).

So far we have been sectioning books into smaller sizes, according to the meta book concept. But what happens if we go in the other direction and extrapolate to larger sizes? What could the meta book look like? In the next section, we obtain the size dependences for the parameter values of the wfd in terms of α and present the asymptotic limit of α = 0.

3. Size dependence of the word-frequency distribution (wfd)

To find the size dependence of the wfd, we note that there is a simple relation between wfd and k. If k (which is directly related to the N(M) curve, and thus to α) is changing with size, wfd also has to change in some way (e.g. smaller cut-off or changed slope). We also know that the tail of the distribution must be regulated in such a way that the maximum frequency does not look abnormal (e.g. 90% of all the words are the same), but is consistent with figure 3. Given a functional form, what kind of relationship between the functional parameters is needed to balance these requirements? The requirements mentioned can be summarized in three basic assumptions supported by our previous analyses:

The second assumption reflects the statement that if the most common word used by an author is `the', then a text of length 2M should have twice as many `the's as a text of length M. This statement can be expressed in terms of the cumulative normalized wfd, defined as , which is the fraction of different words with a frequency larger or equal to k '. Thus means that there should be only one word with a frequency larger than or equal to . In other words, if a data set is created by drawing N random numbers from a theoretical and continuous function, P(k), then one would have, on average, one word appearing with a frequency larger than . This word, with frequency k_max, would then be the most common word in the text. So, is a theoretical limit, while k_max is the actual frequency of the most common word. Since the distribution P(k) is a rapidly decreasing function for large k, the most common word always appears with a frequency very close to ( ). It follows that k_maxM, which means that is a valid assumption to a good approximation.

The first two assumptions can be expressed in the continuum approximation as two integral equations

The third assumption is based on the notion that this functional form fits well to empirical data [10, 12]. The basic assumption made in the present context is that the power law with an exponential gives the correct large k behavior and that A and γ vary slowly with M.

Next, we explore the consequences of the three basic assumptions, but first the normalization condition is investigated. From equation (1) we obtain

(For our purpose, it is enough to use the convenient approximation for the leading b dependence. A more precise expansion can be obtained through the relation to the ordinary (Γ(a)) and incomplete (Γ(a,b)) gamma function .) This means that as long as γ > 1 and M is sufficiently large, we have no explicit M dependence for A. That is, if γ is constant or varies slowly enough, we can treat A as constant.

The next step is to evaluate equation (2) by inserting equation (1)

According to equations (5) and (2), γ > 2 means that the average usage of a word is independent of the size of the book, so that M/N = const and consequently NM (α = 1). That is, the number of different words grows linearly with the size of the book. Solving for γ in this case gives . This is also the analytic solution for the Simon model [7], where a text grows linearly as with preferential repetition. Here we instead arrive at this result from the assumed functional form, without introducing any type of growth or preferential element.

However, the crucial point is that if γ < 2, then M^1–αM^β(2–γ) and α = 1–β(2–γ), or

Thus, we have a relationship between γ and α, so the power-law exponent is determined by the rate at which new words are introduced.

The second assumption (equation (3)), with γ > 1, gives the relation

The last case in equation (7) (β < 1) can be disregarded as impossible since γ needs to be smaller than one for the integral to be positive, which means that α is also negative. This would give a book where the number of different words decreases as a function of the total number of words. However, the case of β≥1, together with equation (3), gives the relation 1/NM^–αM^1–γ, and consequently α = γ–1, or

Finally, substituting equation (8) into equation (6) locks down the value of β to be one, and the wfd (given the previously assumed form) becomes

for large M.

Note that if α goes to zero as M goes to infinity, then γ will move infinitely close to one, and this should be true for all authors. Nevertheless, different authors might reach this point in different ways. Taking the limit M going to infinity for equation (9) (b₀/M, α(M)→0) then gives us the functional form of the wfd for an infinite book

In practice though, b₀/M and α(M) will never be exactly zero.

So far, we have shown that the meta book concept is supported by empirical data. We have also derived an expression for the size dependence of the parameters of the wfd, given a functional form. These are in some sense two independent findings that are connected through the exponent α. Next we show that the derived expression for the wfd (equation (9)) is consistent with the real data and that the process of extracting sections from a large book recreates the observed size dependence in α.

4. Size dependence in real books

To validate the assumption that α approaches zero as M increases, we need to fit the real data to an appropriate functional form. This functional form needs to satisfy two constraints: (i) α(M) should be a monotonically decreasing function with the asymptotic limit for large M equal to zero; (ii) N = M^α(M) should be a monotonically increasing function (by definition the number of unique words never decreases). These constraints result in the condition

where the equality gives the solution 1/α(M) = ulnM, where u is an arbitrary constant. In order to parametrize α(M), we introduce an additional parameter, v, giving the expression 1/α(M) = ulnM + v, which obeys the inequality in equation (11) if v > 0. The final parametrization to describe α is then

The limiting value for N, given equation (12), is lim_M→∞N = lim_M→∞M^α(M) = e^1/u. Note that this parametrization is a generalization of Heap's law (α = const if u = 0). We obtain a good fit for this parametrization for all three authors, as shown in figure 4, where we ignore the first 2×10⁵ words because we are interested in the large M behavior. However, the resulting fit for N(M) = M^α(M) is very reasonable also for small M.

The main point is not to identify the exact extrapolation behavior for each author, but to show that the behaviors are all in accordance with the suggested functional form of α(M), telling us that the empirical data are consistent with α going to zero.

The three assumptions in the previous section lead to the specific form of the wfd in terms of α(M) (equation (9)). In figure 5 this result is compared to the real data for two authors (columns), and for each author, three different book sizes (rows). Since A is a normalization constant and α(M) = lnN/lnM, there is essentially only one free parameter, b₀. This parameter is a characteristic of the author and according to the above analysis is independent of the length of the text. In other words, once the author's characteristic b₀ is determined, then the parameter b for a text of length M by the same author is given by b = b₀/M. The agreement suggests that the analysis leading to equation (9) is indeed valid.

The empirical data seem consistent with the size dependence derived for the wfd with b = b₀/M and γ = 1 + α(M). But what causes the peculiar form of α(M)? Our suggestion is that the actual sectioning of a book is responsible for creating such a structure. This can be tested by applying the meta book concept to a large hypothetical book.

The actual process of pulling a section out of a book can be described analytically by a combinatorial transformation, provided one assumes that the words in a book are uniformly distributed [10]. For instance, if the word `the' exists k ' times in a book, then the probability of getting k `the's when taking half (n = 1/2) of that book is given by the binomial distribution. This can be generalized for any n (equation (13)) and is called the random book transformation (RBT) [6, 10]. This transformation describes how the wfd changes when a section of size M is extracted from a bigger book of size M '

where n = M ' /M, C is the normalization constant and A_kk ' is the triangular matrix with the elements

To analyze the behavior of the RBT, we start with the theoretical wfd for the entire Hardy corpus from figure 5(a) (P_M '(k) = A exp(–0.000019k)/k^1.732) and transform it down to smaller sizes, calculating the average frequency for each size, M, according to the formula

where P_M(k) is given by equation (13).

In figure 6, k_M is plotted in a log–log scale for the data created by the RBT, as circles, and the full line represents the real data for the Hardy corpus (same data as the line in figure 2(a)). The dotted line shows the corresponding analytic result k = M^2–γ = M^1–α = M^0.268 (γ = 1.732) for a constant α and γ. The figure shows the similar behavior of the RBT and the real data.

Figure 6. Average frequency of a word, k, as a function of the total number of words, M. The line shows the real data of the full collection by Hardy and the circles show the result obtained from the RBT starting at the wfd for the full Hardy in figure 5(a), i.e. PM '(k) = Aexp(–0.000019k)/k1.732. The dotted line corresponds to the analytic solution k = M2–γ = M1–α = M0.268, for a constant γ = 1.732.

5. Conclusions

In this paper, we have discussed the text-length dependence of the wfd of single authors. Evidence is presented for a systematic decrease in the power-law index γ of the wfd, from γ≈2 for short novels to the infinite book size limit with γ = 1. This systematic change is linked to the text-length dependence of the number of unique words N as a function of the total number of words M.

We have shown empirically that the size dependence of the wfd (and also N and k as a function of M) display very similar behavior to sectioning a large book. It was also demonstrated, through the use of the random book transformation, that the same process can reproduce the observed decrease of α. This led us to introduce the concept of a meta book, which is an imaginary book of infinite length written by an author, as a description of this behavior. Furthermore, the meta book should have a wfd close to P(k) = A/k. The meta book should contain all the statistical properties of a real text, related to the specific writing style of an author, which are then transferred to the real book when extracted from this meta book. It is important to remember that this is an abstract description, and novels (or text sections in novels) written by a single author, of length M, are on average characterized by P_M(k). One may also note that the meta book is a holistic concept, which implies that any text of any length written by the author carries information about the total extent of the author's vocabulary; the P_M(k) average for a text section of size M is independent of the total size M ' of the book.

It is interesting to compare this to the related phenomenon of family name distribution where the γ = 1 limit is realizable [15, 16]. In this case, M corresponds to the number of inhabitants of a country or town, N to the number of different family names, and P(k) to the corresponding frequency distribution of family names. For a country like the USA or a city like Berlin, P(k)k^–γ with γ≈2 [13, 14]. However, for Vietnam γ≊1.4 [16] and for Korea γ≊1 [15]. This decrease of γ is correlated with a corresponding decrease of α in NM^α. Thus, the less the number of family names increases with the size of the population, the lower is γ, until the limiting case γ = 1 and α = 0 is reached. For Korea the empirical finding is NlnM , which indeed corresponds to α = 0. In fact, the relation between the exponents γ = 1 + α was also achieved in [15] for family names, suggesting that the relation between P_M(k) and N(M) is more general than suggested here, and could hold for different kinds of systems.

Acknowledgment

This work was supported by the Swedish Research Council through contract 50412501.

Appendix

The empirical data used in this article consists of books written by three authors: Thomas Hardy, Herman Melville and David Herbert Lawrence (see table 1 for a complete list of books). All books are taken from the online book catalog, `Project Gutenberg' (http://www.gutenberg.org/catalog/). In order to estimate the behavior of very large books, we combined a collection of books by each author, simply adding them together one after the other. Averages have been obtained by employing periodic boundary conditions and using different starting points in the book. This is a valid procedure since the words, to a large extent, are uniformly distributed throughout the book and, statistically speaking, there is no such thing as a beginning or an end [10]. This method gives a considerable reduction of statistical fluctuations.

When presenting the wfd, we use a log₂ binning where the size of the bins follows the formula S_i = 2^i–1.

References

[1]

Smith J M and Szathmry E 1995 The Major Transitions in Evolution  (New York: Oxford University Press) 

[2]

Zipf G 1932 Selective Studies and the Principle of Relative Frequency in Language  (Cambridge, MA: Harvard University Press) 

[3]

Zipf G 1935 The Psycho-Biology of Language: an Introduction to Dynamic Philology  (Boston, MA: Mifflin) 

[4]

Zipf G 1949 Human Bevavior and the Principle of Least Effort  (Reading, MA: Addison-Wesley) 

[5]

Mitzenmacher M 2003 Internet Math. 1 226 
CrossRef

[6]

Baayen R H 2001 Word Frequency Distributions  (Dordrecht: Kluwer) 
CrossRef

[7]

Simon H 1955 Biometrika 42 425 

[8]

Mandelbrot B 1953 An Informational Theory of the Statistical Structure of Languages  (Woburn, MA: Butterworth) 

[9]

Cancho R F and Sola R V 2001 J. Quant. Linguist. 8 165 
CrossRef

[10]

Bernhardsson S, Rocha L E C da and Minnhagen P 2010 Physica A 389 330–341 
CrossRef

[11]

Heaps H S 1978 Information Retrieval: Computational and Theoretical Aspects  (New York: Academic Press) 

[12]

Clauset A, Shalizi C R and Newman M E J 2009 SIAM Rev. 51 661–703 
CrossRef

[13]

Newman M E J 2005 Contemp. Phys. 46 323 
CrossRef

[14]

Zanette D H and Marunbia S C 2001 Physica A 295 1 
CrossRef

[15]

Kim B J and Park S M 2005 Physica A 347 683–94 
CrossRef

[16]

Baek S K, Kiet H A T and Kim B J 2007 Phys. Rev. E 76 046113 
CrossRef