{"id":23577,"date":"2020-06-15T15:28:00","date_gmt":"2020-06-15T22:28:00","guid":{"rendered":"https:\/\/www.springboard.com\/blog\/?p=23577"},"modified":"2023-06-28T00:33:50","modified_gmt":"2023-06-28T07:33:50","slug":"predictive-text-generation","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-science\/predictive-text-generation\/","title":{"rendered":"Deep Learning Project Ideas: Text Generation Using Recurrent Neural Networks (RNNs) and Transformers in NLP"},"content":{"rendered":"\n

When I began typing the title of this article, \u201ctext generation using recurrent n\u2026\u201d, the tool I\u2019m typing on, Google Docs, began automatically completing my sentences. In this case, it accurately suggested recurrent neural networks! If you\u2019ve ever used Gmail compose or even Google search, this wouldn\u2019t surprise you, because the predictive text has been in vogue for a very long time, and has widespread use across industries.<\/p>\n\n\n\n

In this blog post, Springboard\u2019s machine learning mentor Raghav Bali inspires many deep learning project ideas and walks you through creating your own predictive text generation model with recurrent neural networks and transformers. Raghav is a senior data scientist at the UnitedHealth Group, where he designs and implements machine learning, AI and deep learning-based solutions for healthcare and insurance. Before United Health, he has also worked at American Express and Intel, building enterprise-level intelligent solutions.<\/p>\n\n\n\n

For further reading, check out data scientist job description here<\/a> and learn more about data science<\/a>.<\/em><\/p>\n\n\n\n

Understanding Language to Build Recurrent Neural Networks<\/h2>\n\n\n\n

For humans, language is an integral part of existence. We use language to communicate thoughts and ideas every single day. We instinctively understand what language is. Before getting machines to understand them, let\u2019s first define \u2018Language\u2019 in abstract terms, is a collection of alphabets, used in specific settings\/context, to create words\/vocabulary, following a set of rules\/grammar. <\/p>\n\n\n\n

It takes us, humans, many years of learning to communicate in a language. This is simply because languages are complex and constantly evolving. Rules of spelling and grammar aren\u2019t standard, there are multiple ways to communicate the same thing, and the same word might mean different things in different contexts. For example, take the sentence, \u201cThe bowler made a batsman duck.\u201d What does this mean? Does it mean that the bowler made the batsman bend to avoid being hit by the ball? Or that the bowler got the batsman\u2019s wicket at zero? Or that he converted him into a quacking duck?<\/p>\n\n\n\n

Let\u2019s see another example. \u201cThe stolen painting was found by the tree.\u201d Did the tree find the painting? Or was the painting left near the tree?<\/p>\n\n\n\n

As people, we understand communication through a complex process which takes into account several tangible and intangible aspects. Sarcasm, lingo, hashtags, etc. are processed by the human brain easily, which can be difficult for machines to replicate. But we have to try. Today, let\u2019s see a deep learning project idea, based on which you can build complex models to improve on existing systems.<\/p>\n\n\n

Get To Know Other\tData Science Students<\/h4>
\"Karen<\/a>

Karen Masterson<\/p>

Data Analyst at Verizon Digital Media Services<\/p><\/div>

<\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Brandon<\/a>

Brandon Beidel<\/p>

Senior Data Scientist at Red Ventures<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Jonathan<\/a>

Jonathan Orr<\/p>

Data Scientist at Carlisle & Company<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div><\/div><\/div>\n\n\n\n

Understanding Recurrent Neural Networks<\/h2>\n\n\n\n

A recurrent neural network (RNN) is an upgraded version of the neural network, where connections between nodes are treated as sequential signals. <\/p>\n\n\n\n

Take the visual below, for instance. In this case, you\u2019ll notice that the input for h2 is not just x2, but also y1, which is the output of the previous action. We use this for natural language processing and text generation applications because typically language is a sequence. When people speak, words take meaning based on previous words, and sentences take meaning from previous sentences. With RNNs, the context follows.<\/p>\n\n\n\n

\"predictive<\/figure>\n\n\n\n

How to Build a Language Model<\/h2>\n\n\n\n

We can build language models across many levels \u2014 word-level, phrase-level or what we are going to do today, which is corrector-level. Basically, a corrector is a set of alphabets, punctuation, etc., which helps predict the next corrector. We are doing this primarily because corrector-level language modelling will give you a finite and manageable vocabulary. You can go forth and build word-level models on the same principles as well.<\/p>\n\n\n\n

We will use TensorFlow 2.0 with Keras<\/a> as the high-level library. We\u2019ll use gated recurrent units (GRUs), which are better at managing long-range dependencies and gradient problems. We are taking the book \u2018Adventures of Sherlock Holmes\u2019 from the Gutenberg library as our input dataset.<\/p>\n\n\n\n

Step 1: Pre-processing<\/h4>\n\n\n\n