Page 1 of 19Free preview · 1–5

About this document

Published in the Journal of Machine Learning Research in February 2003, this paper presents a neural language model that jointly learns distributed word representations and probabilities of word sequences. These representations enable generalization to unseen sequences and help address the curse of dimensionality. The paper describes the model architecture, training procedures, and parallel implementations. Experiments on the Brown and Associated Press News corpora demonstrate lower perplexity than the compared n-gram models and benefits from longer contexts. It also discusses energy-based extensions and future research directions.

Free preview text

The first 5 pages, shown as text so search engines and readers can find this document.

Page 1

Journal of Machine Learning Research 3 (2003) 1137–1155 Submitted 4/02; Published 2/03 A Neural Probabilistic Language Model Yoshua Bengio BENGIOY@IRO.UMONTREAL.CA Réjean Ducharme DUCHARME@IRO.UMONTREAL.CA Pascal Vincent VINCENTP@IRO.UMONTREAL.CA Christian Jauvin JAUVINC@IRO.UMONTREAL.CA Département d’Informatique et Recherche Opérationnelle Centre de Recherche Mathématiques Université de Montréal, Montréal, Québec, Canada Editors: Jaz Kandola, Thomas Hofmann, Tomaso Poggio and John Shawe-Taylor Abstract A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. Keywords: Statistical language modeling, artificial neural networks, distributed representation, curse of dimensionality 1. Introduction A fundamental problem that makes language modeling and other learning problems difficult is the curse of dimensionality. It is particularly obvious in the case when one wants to model the joint distribution between many discrete random variables (such as words in a sentence, or discrete at- tributes in a data-mining task). For example, if one wants to model the joint distribution of 10 consecutive words in a natural language with a vocabulary V of size 100,000, there are potentially 100 00010 − 1 = 1050 − 1 free parameters. When modeling continuous variables, we obtain gen- eralization more easily (e.g. with smooth classes of functions like multi-layer neural networks or Gaussian mixture models) because the function to be learned can be expected to have some lo- cal smoothness properties. For discrete spaces, the generalization structure is not as obvious: any change of these discrete variables may have a drastic impact on the value of the function to be esti- c©2003 Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin.

Page 2

BENGIO, DUCHARME, VINCENT AND JAUVIN mated, and when the number of values that each discrete variable can take is large, most observed objects are almost maximally far from each other in hamming distance. A useful way to visualize how different learning algorithms generalize, inspired from the view of non-parametric density estimation, is to think of how probability mass that is initially concentrated on the training points (e.g., training sentences) is distributed in a larger volume, usually in some form of neighborhood around the training points. In high dimensions, it is crucial to distribute probability mass where it matters rather than uniformly in all directions around each training point. We will show in this paper that the way in which the approach proposed here generalizes is fundamentally different from the way in which previous state-of-the-art statistical language modeling approaches are generalizing. A statistical model of language can be represented by the conditional probability of the next word given all the previous ones, since ˆP(wT 1 ) = T ∏ t=1 ˆP(wt |wt−1 1 ), where wt is the t-th word, and writing sub-sequence w j i = (wi, wi+1, · · · , w j−1, w j). Such statisti- cal language models have already been found useful in many technological applications involving natural language, such as speech recognition, language translation, and information retrieval. Im- provements in statistical language models could thus have a significant impact on such applications. When building statistical models of natural language, one considerably reduces the difficulty of this modeling problem by taking advantage of word order, and the fact that temporally closer words in the word sequence are statistically more dependent. Thus, n-gram models construct ta- bles of conditional probabilities for the next word, for each one of a large number of contexts, i.e. combinations of the last n − 1 words: ˆP(wt|wt−1 1 ) ≈ ˆP(wt|wt−1 t−n+1). We only consider those combinations of successive words that actually occur in the training cor- pus, or that occur frequently enough. What happens when a new combination of n words appears that was not seen in the training corpus? We do not want to assign zero probability to such cases, because such new combinations are likely to occur, and they will occur even more frequently for larger context sizes. A simple answer is to look at the probability predicted using a smaller context size, as done in back-off trigram models (Katz, 1987) or in smoothed (or interpolated) trigram mod- els (Jelinek and Mercer, 1980). So, in such models, how is generalization basically obtained from sequences of words seen in the training corpus to new sequences of words? A way to understand how this happens is to think about a generative model corresponding to these interpolated or back- off n-gram models. Essentially, a new sequence of words is generated by “gluing” very short and overlapping pieces of length 1, 2 ... or up to n words that have been seen frequently in the training data. The rules for obtaining the probability of the next piece are implicit in the particulars of the back-off or interpolated n-gram algorithm. Typically researchers have used n = 3, i.e. trigrams, and obtained state-of-the-art results, but see Goodman (2001) for how combining many tricks can yield to substantial improvements. Obviously there is much more information in the sequence that immediately precedes the word to predict than just the identity of the previous couple of words. There are at least two characteristics in this approach which beg to be improved upon, and that we 1138

Page 3

A NEURAL PROBABILISTIC LANGUAGE MODEL will focus on in this paper. First, it is not taking into account contexts farther than 1 or 2 words,1 second it is not taking into account the “similarity” between words. For example, having seen the sentence “The cat is walking in the bedroom” in the training corpus should help us gener- alize to make the sentence “A dog was running in a room” almost as likely, simply because “dog” and “cat” (resp. “the” and “a”, “room” and “bedroom”, etc...) have similar semantic and grammatical roles. There are many approaches that have been proposed to address these two issues, and we will briefly explain in Section 1.2 the relations between the approach proposed here and some of these earlier approaches. We will first discuss what is the basic idea of the proposed approach. A more formal presentation will follow in Section 2, using an implementation of these ideas that relies on shared-parameter multi-layer neural networks. Another contribution of this paper concerns the challenge of training such very large neural networks (with millions of parameters) for very large data sets (with millions or tens of millions of examples). Finally, an important contribution of this paper is to show that training such large-scale model is expensive but feasible, scales to large contexts, and yields good comparative results (Section 4). Many operations in this paper are in matrix notation, with lower case v denoting a column vector and v′ its transpose, A j the j-th row of a matrix A, and x.y = x′y. 1.1 Fighting the Curse of Dimensionality with Distributed Representations In a nutshell, the idea of the proposed approach can be summarized as follows: 1. associate with each word in the vocabulary a distributed word feature vector (a real- valued vector in Rm), 2. express the joint probability function of word sequences in terms of the feature vectors of these words in the sequence, and 3. learn simultaneously the word feature vectors and the parameters of that probability function. The feature vector represents different aspects of the word: each word is associated with a point in a vector space. The number of features (e.g. m =30, 60 or 100 in the experiments) is much smaller than the size of the vocabulary (e.g. 17,000). The probability function is expressed as a product of conditional probabilities of the next word given the previous ones, (e.g. using a multi- layer neural network to predict the next word given the previous ones, in the experiments). This function has parameters that can be iteratively tuned in order to maximize the log-likelihood of the training data or a regularized criterion, e.g. by adding a weight decay penalty.2 The feature vectors associated with each word are learned, but they could be initialized using prior knowledge of semantic features. Why does it work? In the previous example, if we knew that dog and cat played simi- lar roles (semantically and syntactically), and similarly for (the,a), (bedroom,room), (is,was), 1. n-grams with n up to 5 (i.e. 4 words of context) have been reported, though, but due to data scarcity, most predictions are made with a much shorter context. 2. Like in ridge regression, the squared norm of the parameters is penalized. 1139

Page 4

BENGIO, DUCHARME, VINCENT AND JAUVIN (running,walking), we could naturally generalize (i.e. transfer probability mass) from The cat is walking in the bedroom to A dog was running in a room and likewise to The cat is running in a room A dog is walking in a bedroom The dog was walking in the room ... and many other combinations. In the proposed model, it will so generalize because “similar” words are expected to have a similar feature vector, and because the probability function is a smooth function of these feature values, a small change in the features will induce a small change in the probability. Therefore, the presence of only one of the above sentences in the training data will in- crease the probability, not only of that sentence, but also of its combinatorial number of “neighbors” in sentence space (as represented by sequences of feature vectors). 1.2 Relation to Previous Work The idea of using neural networks to model high-dimensional discrete distributions has already been found useful to learn the joint probability of Z1 · · · Zn, a set of random variables where each is possibly of a different nature (Bengio and Bengio, 2000a,b). In that model, the joint probability is decomposed as a product of conditional probabilities ˆP(Z1 = z1, · · · , Zn = zn) = ∏ i ˆP(Zi = zi|gi(Zi−1 = zi−1, Zi−2 = zi−2, · · · , Z1 = z1)), where g(.) is a function represented by a neural network with a special left-to-right architecture, with the i-th output block gi() computing parameters for expressing the conditional distribution of Zi given the value of the previous Z’s, in some arbitrary order. Experiments on four UCI data sets show this approach to work comparatively very well (Bengio and Bengio, 2000a,b). Here we must deal with data of variable length, like sentences, so the above approach must be adapted. Another important difference is that here, all the Zi (word at i-th position), refer to the same type of object (a word). The model proposed here therefore introduces a sharing of parameters across time – the same gi is used across time – that is, and across input words at different positions. It is a successful large- scale application of the same idea, along with the (old) idea of learning a distributed representation for symbolic data, that was advocated in the early days of connectionism (Hinton, 1986, Elman, 1990). More recently, Hinton’s approach was improved and successfully demonstrated on learning several symbolic relations (Paccanaro and Hinton, 2000). The idea of using neural networks for language modeling is not new either (e.g. Miikkulainen and Dyer, 1991). In contrast, here we push this idea to a large scale, and concentrate on learning a statistical model of the distribution of word sequences, rather than learning the role of words in a sentence. The approach proposed here is also related to previous proposals of character-based text compression using neural networks to predict the probability of the next character (Schmidhuber, 1996). The idea of using a neural network for language modeling has also been independently proposed by Xu and Rudnicky (2000), although experiments are with networks without hidden units and a single input word, which limit the model to essentially capturing unigram and bigram statistics. The idea of discovering some similarities between words to obtain generalization from training sequences to new sequences is not new. For example, it is exploited in approaches that are based on learning a clustering of the words (Brown et al., 1992, Pereira et al., 1993, Niesler et al., 1998, Baker 1140

Page 5

A NEURAL PROBABILISTIC LANGUAGE MODEL and McCallum, 1998): each word is associated deterministically or probabilistically with a discrete class, and words in the same class are similar in some respect. In the model proposed here, instead of characterizing the similarity with a discrete random or deterministic variable (which corresponds to a soft or hard partition of the set of words), we use a continuous real-vector for each word, i.e. a learned distributed feature vector, to represent similarity between words. The experimental comparisons in this paper include results obtained with class-based n-grams (Brown et al., 1992, Ney and Kneser, 1993, Niesler et al., 1998). The idea of using a vector-space representation for words has been well exploited in the area of information retrieval (for example see work by Schutze, 1993), where feature vectors for words are learned on the basis of their probability of co-occurring in the same documents (Latent Semantic Indexing, see Deerwester et al., 1990). An important difference is that here we look for a repre- sentation for words that is helpful in representing compactly the probability distribution of word sequences from natural language text. Experiments suggest that learning jointly the representation (word features) and the model is very useful. We tried (unsuccessfully) using as fixed word features for each word w the first principal components of the co-occurrence frequencies of w with the words occurring in text around the occurrence of w. This is similar to what has been done with documents for information retrieval with LSI. The idea of using a continuous representation for words has how- ever been exploited successfully by Bellegarda (1997) in the context of an n-gram based statistical language model, using LSI to dynamically identify the topic of discourse. The idea of a vector-space representation for symbols in the context of neural networks has also previously been framed in terms of a parameter sharing layer, (e.g. Riis and Krogh, 1996) for secondary structure prediction, and for text-to-speech mapping (Jensen and Riis, 2000). 2. A Neural Model The training set is a sequence w1 · · · wT of words wt ∈ V , where the vocabulary V is a large but finite set. The objective is to learn a good model f (wt , · · · , wt−n+1) = ˆP(wt|wt−1 1 ), in the sense that it gives high out-of-sample likelihood. Below, we report the geometric average of 1/ ˆP(wt|wt−1 1 ), also known as perplexity, which is also the exponential of the average negative log-likelihood. The only constraint on the model is that for any choice of wt−1 1 , ∑|V | i=1 f (i, wt−1, · · · , wt−n+1) = 1, with f > 0. By the product of these conditional probabilities, one obtains a model of the joint probability of sequences of words. We decompose the function f (wt , · · · , wt−n+1) = ˆP(wt|wt−1 1 ) in two parts: 1. A mapping C from any element i of V to a real vector C(i) ∈ Rm. It represents the distributed feature vectors associated with each word in the vocabulary. In practice, C is represented by a |V | × m matrix of free parameters. 2. The probability function over words, expressed with C: a function g maps an input sequence of feature vectors for words in context, (C(wt−n+1), · · · ,C(wt−1)), to a conditional probability distribution over words in V for the next word wt. The output of g is a vector whose i-th element estimates the probability ˆP(wt = i|wt−1 1 ) as in Figure 1. f (i, wt−1, · · · , wt−n+1) = g(i,C(wt−1), · · · ,C(wt−n+1)) The function f is a composition of these two mappings (C and g), with C being shared across all the words in the context. With each of these two parts are associated some parameters. The 1141

Tags

Doxxly editorial guide

A Neural Probabilistic Language Model: summary, contents and background

This guide helps you understand the scope of A Neural Probabilistic Language Model before or alongside reading the original document. It is based on the catalogue record, credited creator, source, classification and topics shown on this page. It does not replace the original work and does not invent claims about passages that Doxxly cannot verify.

Summary and overview

A Neural Probabilistic Language Model is a 19-page documentcredited to Juno W. Bellweather. Published in the Journal of Machine Learning Research in February 2003, this paper presents a neural language model that jointly learns distributed word representations and probabilities of word sequences. These representations enable generalization to unseen sequences and help address the curse of dimensionality. The paper describes the model architecture, training procedures, and parallel implementations. Experiments on the Brown and Associated Press News corpora demonstrate lower perplexity than the compared n-gram models and benefits from longer contexts. It also discusses energy-based extensions and future research directions. It appears in the Science & Engineering collection, which groups it with other material that readers may use for comparison and further study.

The item can be approached in several ways. A reader looking for a quick orientation can begin with the free five-page preview, inspect the page structure and use the metadata to judge relevance. Someone undertaking deeper research can work through the complete file, search its indexed text, record useful page references and compare its treatment of Statistical language modeling, Artificial neural networks, Distributed representations, Curse of dimensionality, Natural language processing, Parallel computing, N-gram models with the related resources linked below.

Rather than reducing the work to a single claim, this page treats it as a source that should be read in context. The title, creator, publication source, licence and format all help explain what kind of evidence it may offer. Readers should distinguish the original author's statements from Doxxly's catalogue description, and should cite the original edition when using the material in academic, professional or published work.

Key topics and reader takeaways

The catalogue connects this document with Statistical language modeling, Artificial neural networks, Distributed representations, Curse of dimensionality, Natural language processing, Parallel computing, N-gram models. These terms offer practical entry points rather than a substitute for close reading. Start by identifying the definitions, examples, arguments, tables or visual explanations attached to each topic. Then note where the creator draws connections, makes distinctions or moves from background material to a conclusion, recommendation, calculation or narrative development.

A useful reading method is to separate description from interpretation. First record what the item actually presents: names, headings, figures, formulas, quotations or stated propositions. Next consider why those details matter within the whole work. Finally compare them with another source from the same subject area. This process helps turn a long digital file into evidence that can be evaluated rather than merely skimmed.

Table of contents and reading guide

The following is an editorial content guide based on the verified catalogue topics. It is not represented as the original author's formal table of contents. Use it to organise notes while moving through the 19 pages in the player.

  1. Part 1

    Statistical language modeling

    Use this section of the reading guide to identify how Statistical language modeling appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  2. Part 2

    Artificial neural networks

    Use this section of the reading guide to identify how Artificial neural networks appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  3. Part 3

    Distributed representations

    Use this section of the reading guide to identify how Distributed representations appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  4. Part 4

    Curse of dimensionality

    Use this section of the reading guide to identify how Curse of dimensionality appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  5. Part 5

    Natural language processing

    Use this section of the reading guide to identify how Natural language processing appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  6. Part 6

    Parallel computing

    Use this section of the reading guide to identify how Parallel computing appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

  7. Part 7

    N-gram models

    Use this section of the reading guide to identify how N-gram models appears in A Neural Probabilistic Language Model, note important terms, and connect the subject to the wider purpose of the work.

When a heading in the original differs from this guide, follow the original heading in your citation. Page numbers can also vary between editions. Doxxly's direct page field is useful for navigation, but formal references should include the title, creator, source and edition details available in the file itself.

About Juno W. Bellweather

Doxxly credits Juno W. Bellweather as the author or creator of A Neural Probabilistic Language Model. This page does not add an unverified biography, job title or institutional affiliation. The attribution is preserved so readers can distinguish the creator from the hosting library and follow the original record when an external author link is available.

Author context matters because a work reflects a particular purpose, audience and point of view. Before relying on the material, consider what the creator set out to explain, when the source was produced and whether later research or revised guidance should also be consulted. For literature and philosophy, compare translations and editions; for technical material, check whether a newer version has superseded the one shown here.

About the source organisation

The recorded source for this item is jmlr.org. Doxxly provides access and discovery tools but does not claim to be the original publisher unless explicitly stated. Source links, creator attribution and the recorded Publicly available file collected for editorial review licence are displayed to help readers assess provenance and permitted use.

Provenance is especially important for reports, standards, presentations and working spreadsheets. Check the original source for amendments, supporting files and current policy. A trustworthy research workflow keeps the source URL with any notes or downloaded copy, verifies important figures against the original, and avoids treating an archived document as automatically current.

Visit jmlr.org

How to read and use this document

Begin with the title and opening pages, then use the guide above to decide whether to read sequentially or move to a particular topic. The Doxxly player provides previous and next navigation, a direct page field, zoom steps, fit-to-width and fit-to-page modes, rotation and fullscreen. These controls are useful when a page contains small type, a wide table, an image, a scanned layout or a dense sequence of notes.

The first five pages are open to every visitor. A plan unlocks the complete item and full-document search. Search is best used for distinctive names, technical terms or short phrases; broad words can produce too many possible contexts. When you find a useful passage, read the surrounding pages so that a sentence, chart or formula is not separated from its explanation.

If downloading is enabled, members can obtain the original PDF file. Keep the original format when formulas, slide sequencing, pagination or layout carries meaning. Respect the Publicly available file collected for editorial review terms displayed on this page, and use the rights-report link if you believe the attribution or availability needs review.

Research questions to consider

As you read A Neural Probabilistic Language Model, ask which audience the creator addresses, what problem or idea organises the work, what evidence is presented, and which assumptions remain unstated. Note whether important terms are defined consistently and whether examples represent a broader pattern or a narrow case. For quantitative material, record units, time periods, formulas and data sources. For interpretive material, separate the creator's language from your own summary.

A second set of questions concerns relevance today. Which parts remain durable, which depend on their historical or institutional setting, and what would need confirmation from a newer source? Following the tags and the Science & Engineering collection can reveal useful comparisons. Agreement between sources can strengthen confidence, while disagreement can point to changing definitions, methods, evidence or priorities.

Finally, decide what you need from the item: background knowledge, a quotation, a worked example, a model, a visual explanation or a starting bibliography. That purpose should guide the depth of reading and the notes you keep. Doxxly helps locate and navigate the source; careful interpretation remains the reader's responsibility.

Explore more Science & Engineering resources

Frequently asked questions about A Neural Probabilistic Language Model

Practical answers about reading, downloading and understanding this document on Doxxly.

More in Science & Engineering