Report

GENERATING TEXT WITH RECURRENT NEURAL NETWORKS Ilya Sutskever, James Martens, and Geoffrey Hinton, ICML 2011 2013-4-1 Institute of Electronics, NCTU 指導教授: 王聖智 S. J. Wang 學生 : 陳冠廷 K. T. Chen 2 Outline • Introduction • Motivation • What is the RNN? • Why do we choose RNN to solve this problem? • How to train RNN? • Contribution • Character-Level language modeling • The multiplicative RNN • The Experiments • Discussion 3 Outline • Introduction • Motivation • What is the RNN? • Why do we choose RNN to solve this problem? • How to train RNN? • Contribution • Character-Level language modeling • The multiplicative RNN • The Experiments • Discussion 4 Movation • Read some sentences and then try to predict next character. Easter is a Christian festival and holiday celebrating the resurrection of Jesus Christ.. ? Easter is a Christian festival and holiday celebrating the resurrection of Jesus Christ on the third day after his crucifixion at Calvary as described in the New Testament. 5 Recurrent neural networks • A recurrent neural network (RNN) is a class of neural network where connections between units form a directed cycle Feed-forward neural network output Recurrent neural network output hidden hidden input input 6 Why do we choose RNNs? • RNNs are suitable to deal with sequential data.(memory) • RNNs are neural network in time predictions ℎ ℎ ℎ ℎℎ ℎℎ ℎ ℎ hiddens ℎ inputs t -1 t t +1 time 7 How to train RNN? • Backpropagation through time (BPTT) • The gradient is easy to compute with backpropagation. • RNNs learn by minimizing the training error. predictions ℎ ℎ ℎ ℎℎ ℎℎ ℎ ℎ hiddens ℎ inputs t -1 t t +1 time 9 RNNs are hard to train • They can be volatile and can exhibit long-range sensitivity to small parameter perturbations. • The Butterfly Effect • The “vanishing gradient problem” makes gradient descent ineffective. outputs hiddens inputs time 10 How to overcome vanishing gradient? • Long-short term memory.(LSTM) • Modify the architecture of neural network. data write keep read • Hessian-Free optimizer. (James Martens et al. 2011.) • Base on the Newton’s method + conjugate gradient algorithm • Echo State network. • Only learn the hidden-output weighted. 11 Outline • Introduction • Motivation • What is the RNN? • Why do we choose RNN to solve this problem? • How to train RNN? • Contribution • Character-Level language modeling • The multiplicative RNN • The Experiments • Discussion 12 Character-Level language modeling • The RNN observes a sequence of characters. • The target output at each time step is defined as the input character at the next time-step. o “Hell” “Hello” target “Hel” l “He” l “H” e H e l l o Hidden state stores relevant Information. 13 The standard RNN ℎℎ ℎ = tanh(ℎ + ℎℎ ℎ−1 + ℎ ) ……. ……. = ℎ ℎ + ℎ H …… character: 1-of-86 Softmax Predict distribution for next character. • The current input is transformed via the visible-to- hidden weight matrix ℎ ,and then contributes additively to the input for the current hidden state. 14 Some motivation from model a tree i ..fixi ..fix e ..fixe n .fixin • Each node is a hidden state vector. The next character must transform this to a new node. • The next hidden state needs to depend on the conjunction of the current character and the current hidden representation. 15 The Multiplicative RNN • They tried several neural network architectures and found the “Multiplicative-RNN” (MRNN) to be more effective than the regular RNN The weight matrix is chosen by the current character Current input character ℎ = tanh(ℎ + ℎℎ = ℎ ℎ + ℎ−1 + ℎ ) 16 The Multiplicative RNN • Naïve implementation : assign a matrix to each character • This requires a lot of parameters. (86*1500*1500) • This could make the net overfit. • Difficult to parallelize on a GPU • Factorize the matrices of each character • Fewer parameters • Easier to parallelize 17 The Multiplicative RNN • We can get groups a and b to interact multiplicatively by using “factors” f Group c Group a = ( )( ) = ( )( ) Scalar coefficient Group b Outer product transition matrix with rank 1 = 18 The Multiplicative RNN = ……. ℎ H …… character: 1-of-86 ……. f ……. 1500 hidden units 1500 hidden units Predict distribution for next character. Each factor f defines a rank one matrix, 19 The Multiplicative RNN Output ℎ ℎ ℎ ℎ ℎ ℎ ℎ t ℎ ℎ ℎ ℎ t-1 ℎ ℎ t+1 ℎ ℎ ℎ t +2 Input characters Time 20 The Multiplicative RNN:Key advantages • The MRNN combines conjunction of contexts and characters more easily: Predict “i,e,_” Predict “n” fix fixi i • The MRNN has two nonlinearities per timestep,whick make its dynamics even ricker and more powerful. 21 Outline • Introduction • Motivation • What is the RNN? • Why do we choose RNN to solve this problem? • How to train RNN? • Contribution • Character-Level language modeling • The multiplicative RNN • The Experiments • Discussion 22 The Experiments • Training on three large datasets • ~1GB of the English Wikipedia • ~1GB of articles from New York Times • ~100MB of JMLR and AISTATS paper • Compare with the Sequence Memorizer (Wood et al.) and PAQ (Mahoney et al.) 23 Training on subsequences millions long This is an extremely long string of text…………………………………………………………………………. 250 This is an extre ……… his is an extrem…….. is is an extreme……... The subsequences s is an extremel……... is an extremely……... is an extremely ....... s an extremely l....... ….. Compute the gradient and the curvature on subset of the subsequences. Use a different subset at each iteration 24 Parallelization • Use HF optimizer to evaluate the gradient and curvature on large minibatches of data. Data GPU GPU + gradient + curvature GPU GPU GPU GPU GPU GPU 25 The architecture of model • Use 1500 hidden units and 1500 multiplicative factors on 250- long sequences. ...... 1500 predicetions ...... …. …. …. …. …. …. …. …. ...... hiddens input 500 • Arguably the largest and deepest neural network ever trained. 26 Demo • The MRNN extracts “higher level information”, stores it for many timesteps ,and uses it to make a prediction. • Parentheses sensitivity • (Wlching et al. 2005) the latter has received numerical testimony without much • • • • • • deeply grow (Wlching, Wulungching, Alching, Blching, Clching et al." 2076) and Jill Abbas, The Scriptures reported that Achsia and employed a the sequence memoizer (Wood et al McWhitt), now called "The Fair Savings.'"" interpreted a critic. In t Wlching ethics, like virtual locations. The signature tolerator is necessary to en Wlching et al., or Australia Christi and an undergraduate work in over knowledge, inc They often talk about examples as of January 19, . The "Hall Your Way" (NRF film) and OSCIP Her image was fixed through an archipelago's go after Carol^^'s first century, but simply to 27 Outline • Introduction • Motivation • What is the RNN? • Why do we choose RNN to solve this problem? • How to train RNN? • Contribution • Character-Level language modeling • The multiplicative RNN • The Experiments • Discussion 28 Discussion • The MRNN model generated text contains very few non- words. (e.g., “cryptoliation”, “homosomalist”). This let MRNN can deal with real words that it didn’t see in the training set. • If they have more computational power, they could train much bigger MRNNs with millions of units and billions of connections. 29 Reference • Generating Text with Recurrent Neural Networks, Ilya Sutskever, James Martens, and Geoffrey Hinton, ICML 2011 • Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style, GrahamW. Taylor, Geoffrey E. Hinton • Coursera : Neural Networks for Machine Learning ,Geoffrey Hinton • http://www.cs.toronto.edu/~ilya/rnn.html