Identifying missing dictionary entries with frequency-conserving context models

In an effort to better understand meaning from natural language texts, we explore methods aimed at organizing lexical objects into contexts. A number of these methods for organization fall into a family defined by word ordering. Unlike demographic or spatial partitions of data, these collocation models are of special importance for their universal applicability. While we are interested here in text and have framed our treatment appropriately, our work is potentially applicable to other areas of research (e.g., speech, genomics, and mobility patterns) where one has ordered categorical data (e.g., sounds, genes, and locations). Our approach focuses on the phrase (whether word or larger) as the primary meaning-bearing lexical unit and object of study. To do so, we employ our previously developed framework for generating word-conserving phrase-frequency data. Upon training our model with the Wiktionary, an extensive, online, collaborative, and open-source dictionary that contains over $100000$ phrasal definitions, we develop highly effective filters for the identification of meaningful, missing phrase entries. With our predictions we then engage the editorial community of the Wiktionary and propose short lists of potential missing entries for definition, developing a breakthrough, lexical extraction technique and expanding our knowledge of the defined English lexicon of phrases.

Figure 1

Example showing the sharing of contexts by similar phrases. Suppose that our text consists of the two phrases, “in the contrary” and “on the contrary,” that each occurs once, and that the latter has definition $\left(D=1\right)$ while the former does not. In this event, we see that the three shared contexts “$☆☆☆$”, “$☆☆\text{contrary}$,” and “$☆\text{the}\text{contrary}$” present elevated likelihood $\overline{D}$ values, indicating that the phrase “in the contrary” may have meaning and be worthy of definition.

Figure 2

With data taken from the Twitter corpus, we present (tenfold) cross-validation results for the filtration procedures. For each of the lengths 2, 3, 4, and 5, we show the ROC curves, comparing true and false positive rates for both the likelihood filters (black) and the frequency filters (gray). There we see increased performance in the likelihood classifiers (except possibly for length 5), which is reflected in the AUCs (where an AUC of 1 indicates a perfect classifier). In the insets we also monitor the average number of missing entries discovered as a function of the number of entries proposed, for each length. There the horizontal dotted lines indicate the average numbers of missing entries discovered for both the likelihood filters (black) and the frequency filters (gray) when short lists of 20 phrases were taken (red dotted vertical lines). From this we see an indication that even the 5-gram likelihood filter is effective at detecting missing entries in short lists, while the frequency filter is not.

Figure 3

With data taken from the NYT corpus, we present (tenfold) cross-validation results for the filtration procedures. For each of the lengths 2, 3, 4, and 5, we show the ROC curves, comparing true and false positive rates for both the likelihood filters (black) and the frequency filters (gray). There we see increased performance in the likelihood classifiers (except possibly for length 5), which is reflected in the AUCs (where an AUC of 1 indicates a perfect classifier). In the insets we also monitor the average number of missing entries discovered as a function of the number of entries proposed, for each length. There the horizontal dotted lines indicate the average numbers of missing entries discovered for both the likelihood filters (black) and the frequency filters (gray) when short lists of 20 phrases were taken (red dotted vertical lines). From this we see an indication that even the 5-gram likelihood filter is effective at detecting missing entries in short lists, while the frequency filter is not.

Figure 4

With data taken from the lyrics corpus, we present (tenfold) cross-validation results for the filtration procedures. For each of the lengths 2, 3, 4, and 5, we show the ROC curves, comparing true and false positive rates for both the likelihood filters (black) and the frequency filters (gray). There we see increased performance in the likelihood classifiers, which is reflected in the AUCs (where an AUC of 1 indicates a perfect classifier). In the insets we also monitor the average number of missing entries discovered as a function of the number of entries proposed, for each length. There the horizontal dotted lines indicate the average numbers of missing entries discovered for both the likelihood filters (black) and the frequency filters (gray), when short lists of 20 phrases were taken (red dotted vertical lines). Here we can see that it may have been advantageous to construct slightly longer 3- and 4-gram lists.

Figure 5

With data taken from the Wikipedia corpus, we present (tenfold) cross-validation results for the filtration procedures. For each of the lengths 2, 3, 4, and 5, we show the ROC curves, comparing true and false positive rates for both the likelihood filters (black) and the frequency filters (gray). There we see increased performance in the likelihood classifiers, which is reflected in the AUCs (where an AUC of 1 indicates a perfect classifier). In the insets we also monitor the average number of missing entries discovered as a function of the number of entries proposed, for each length. There the horizontal dotted lines indicate the average numbers of missing entries discovered for both the likelihood filters (black) and the frequency filters (gray) when short lists of 20 phrases were taken (red dotted vertical lines). Here we can see that it may have been advantageous to construct slightly longer 3- and 4-gram lists.

Figure 6

With data taken from the eBooks corpus, we present (tenfold) cross-validation results for the filtration procedures. For each of the lengths 2, 3, 4, and 5, we show the ROC curves, comparing true and false positive rates for both the likelihood filters (black) and the frequency filters (gray). There we see increased performance in the likelihood classifiers, which is reflected in the AUCs (where an AUC of 1 indicates a perfect classifier). In the insets we also monitor the average number of missing entries discovered as a function of the number of entries proposed, for each length. There the horizontal dotted lines indicate the average numbers of missing entries discovered for both the likelihood filters (black) and the frequency filters (gray) when short lists of 20 phrases were taken (red dotted vertical lines). Here we can see that the power of the 4-gram model does not show itself until longer lists are considered.