{"id":2776,"date":"2017-08-28T14:23:41","date_gmt":"2017-08-28T21:23:41","guid":{"rendered":"https:\/\/www.springboard.com\/?p=2776"},"modified":"2023-08-22T10:29:49","modified_gmt":"2023-08-22T17:29:49","slug":"introduction-word-embeddings","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-science\/introduction-word-embeddings\/","title":{"rendered":"An Introduction to Word Embeddings"},"content":{"rendered":"\n

Understanding Word Embeddings<\/h2>\n\n\n\n
\n\n\n\n

Part 1: Applications<\/strong><\/em><\/h3>\n\n\n\n

Written by Aaron Geelon So<\/span><\/strong><\/em><\/p>\n\n\n\n

If you already have a solid understanding of word embeddings and are well into your data science career<\/a>, skip ahead to the next part<\/a>!<\/em><\/p>\n\n\n\n

Human language is unreasonably effective<\/span> at describing how we relate to the world. With a few, short words, we can convey many ideas and actions with little ambiguity. Well, mostly<\/span>. <\/span><\/p>\n\n\n\n

          B<\/span>ecause we’re capable of seeing and describing so much complexity, a lot of structure is implicitly encoded into our language. It is no easy task for a computer (or a human, for that matter) to learn natural language<\/i>, for it entails understanding how we humans observe the world, if not understanding how to<\/i> observe the world. <\/span><\/p>\n\n\n\n

          <\/span>For the most part, computers can’t understand natural language. Our programs are still line-by-line instructions telling a computer what to do — they often miss nuance and context. How can you explain sarcasm to a machine? <\/span><\/p>\n\n\n\n

 In the area of data science<\/a>, NLP has its own importance.         <\/span>There’s good news though. There’s been some important breakthroughs in natural language processing<\/i> (NLP), the domain where researchers try to teach computers human language.<\/span><\/p>\n\n\n\n

          <\/span>Famously, in 2013 Google researchers (Mikolov 2013<\/em>) found a method that enabled a computer to learn relations between words such as:<\/span><\/p>\n\n\n\n

king<\/span>-<\/span>man<\/span>+<\/span>woman<\/span>\u2248<\/span>queen<\/span>.<\/span><\/pre>\n\n\n\n
president<\/span>-<\/span>power<\/span>\u2248<\/span>prime minister<\/span><\/pre>\n\n\n\n
Admittedly, this might be one of those specious relations. <\/span><\/p>\n\n\n\n
          <\/span>Joking aside, it’s worth studying word embeddings for at least two reasons. First, there are a lot of applications made possible by word embeddings. Second, we can learn from the way researchers approached the problem of deciphering natural language for machines. <\/span><\/p>\n\n\n\n
          <\/span>In Part 1 of this article series, let\u2019s take a look at the first of these reasons.<\/span><\/p>\n\n\n\n
Uses of Word Embeddings<\/span><\/strong><\/h4>\n\n\n\n
There’s no obvious<\/span> way to usefully compare two words unless we already know what they mean. The goal of word-embedding algorithms is, therefore, to embed words with meaning based on their similarity or relationship with other words. <\/b><\/span><\/p>\n\n\n\n
          <\/span>In practice, words are embedded into a real vector space, which comes with notions of distance and angle. We hope that these notions extend to the embedded words in meaningful ways, quantifying relations or similarity between different words. And empirically, they actually do!<\/span><\/p>\n\n\n\n
           <\/span>For example, the Google algorithm I mentioned above discovered certain nouns are singular\/plural or have gender (Mikolov 2013abc):<\/span><\/p>\n\n\n\n
\"relations<\/figure>\n\n\n\n
<\/p>\n\n\n\n
 They also found a country-capital relationship:<\/span><\/p>\n\n\n\n
\"country<\/figure>\n\n\n\n
<\/p>\n\n\n\n
           <\/span>And as further evidence that a word’s meaning can be implied from its relationships with other words, they actually found that the learned structure for one language often correlated to that of another language, perhaps suggesting the possibility for machine translation<\/span><\/a> through word embeddings (Mikolov 2013c):<\/span><\/p>\n\n\n\n
\"mt<\/figure>\n\n\n\n
<\/p>\n\n\n\n
          <\/span>They released their C code as the word2vec<\/span><\/a> package, and soon after, others adapted the algorithm for more programming languages. Notably, for gensim<\/span><\/a><\/span> (Python) and deeplearning4j<\/span><\/a> (Java).<\/span><\/p>\n\n\n\n
Today, many companies and data scientists<\/a><\/span> have found different ways to incorporate word2vec into their businesses and research. Spotify uses it to help provide music recommendations. Stitch Fix uses it to recommend clothing. Google is thought to use word2vec in RankBrain as part of its search algorithm. <\/span><\/p>\n\n\n
Get To Know Other\tData Science Students<\/h4>
\"Hastings<\/a>
Hastings Reeves<\/p>
Business Intelligence Analyst at Velocity Global<\/p><\/div>
<\/div>
Read Story<\/a><\/p><\/div><\/div>
\"Samuel<\/a>
Samuel Okoye<\/p>
IT Consultant at Kforce<\/p><\/div>
Read Story<\/a><\/p><\/div><\/div>
\"Bryan<\/a>
Bryan Dickinson<\/p>
Senior Marketing Analyst at REI<\/p><\/div>
Read Story<\/a><\/p><\/div><\/div><\/div><\/div>\n\n\n\n
\u00a0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>Other researchers are using word2vec for sentiment analysis, which attempts to identify the emotionality behind the words people use to communicate. For example, one Stanford research group<\/span><\/a> looked at how the same words in different Reddit communities take on different connotations. Here\u2019s an example with the word soft<\/i>:<\/span><\/p>\n\n\n\n
\"reddit<\/figure>\n\n\n\n
<\/p>\n\n\n\n
As you can see, the word “soft” has a negative connotation when you’re talking about sports (you might think of the term “soft players”) while they have a positive connotation when you’re talking about cartoons.<\/p>\n\n\n\n
 And here are more examples where the computer could analyze the emotional sentiment of the same words across different communities.<\/span><\/p>\n\n\n\n
\"reddit-spectrum<\/figure>\n\n\n\n
<\/p>\n\n\n\n
           <\/span>They can even apply the same method over time, following how the word terrific<\/i>, which meant horrific<\/i> for the majority of the 20th century, has come to essentially mean great<\/i> today.<\/span><\/p>\n\n\n\n
\"terrific<\/figure>\n\n\n\n
<\/p>\n\n\n\n
          <\/span>As a light-hearted example, one research group<\/span><\/span> used word2vec to help them determine whether a fact is surprising or not, so that they could automatically generate trivia facts.<\/span><\/p>\n\n\n\n
          <\/span>The successes of word2vec have also helped spur on other forms of word embedding\u2014WordRank<\/span><\/a>, Stanford’s GloVe<\/span><\/a>, and Facebook’s fastText<\/span><\/a>, to name a few major ones. <\/span><\/p>\n\n\n\n
These algorithms seek to improve on word2vec — they also look at texts through different units<\/i>: characters, subwords, words, phrases, sentences, documents, and perhaps even units of thought. As a result, they allow us to think about not just word similarity, but also sentence similarity and  <\/span>document similarity—like this paper did (Kusner 2015):<\/span><\/p>\n\n\n\n
\"wmd<\/figure>\n\n\n\n
<\/p>\n\n\n\n
          <\/span>Word embeddings transform human language meaningfully into a form conducive to numerical analysis<\/b>. In doing so, they allow computers to explore the wealth of knowledge encoded implicitly into our own ways of speaking. We’ve barely scratched the surface of that potential<\/strong>.<\/span><\/p>\n\n\n\n
          A<\/span>ny individual programmer or scholar can use these tools and contribute new knowledge. Many areas of research and industry that could benefit from NLP have yet to be explored. Word embeddings and neural language models<\/a> are powerful techniques. But perhaps the most powerful aspect of machine learning is its collaborative culture. Many, if not most, of the state-of-the-art methods, are open-source, along with their accompanying research.<\/span><\/p>\n\n\n\n
          <\/span>So, it’s there, if we want to take advantage. Now, the main obstacle is just ourselves. And…maybe an expensive GPU.<\/span><\/p>\n\n\n\n
\/\/<\/span><\/p>\n\n\n\n
For the theory behind word embeddings, see<\/span> Part 2<\/span><\/a>.<\/span><\/p>\n\n\n\n
References<\/span><\/p>\n\n\n\n
[Hamilton 2016<\/span><\/a>] Hamilton, William L., et al. \u201cInducing domain-specific sentiment lexicons from unlabeled corpora.\u201d arXiv preprint arXiv:1606.02820 (2016).<\/span><\/p>\n\n\n\n
[Kusner 2015<\/span><\/a>] Kusner, Matt, et al. \u201cFrom word embeddings to document distances.\u201d International Conference on Machine Learning. 2015.<\/span><\/p>\n\n\n\n
[Mikolov 2013a<\/span><\/a>] Mikolov, Tomas, Wen-tau Yih, and Geoffrey Zweig. \u201cLinguistic regularities in continuous space word representations.\u201d hlt-Naacl. Vol. 13. 2013.<\/span><\/p>\n\n\n\n
[Mikolov 2013b<\/span><\/a>] Mikolov, Tomas, et al. \u201cEfficient estimation of word representations in vector space.\u201d arXiv preprint arXiv:1301.3781 (2013).<\/span><\/p>\n\n\n\n
[Mikolov 2013c<\/span><\/a>] Mikolov, Tomas, et al. \u201cDistributed representations of words and phrases and their compositionality.\u201d Advances in neural information processing systems. 2013.<\/span><\/p>\n\n\n\n
[Mikolov 2013d<\/span><\/a>] Mikolov, Tomas, Quoc V. Le, and Ilya Sutskever. \u201cExploiting similarities among languages for machine translation.\u201d arXiv preprint arXiv:1309.4168 (2013).<\/span><\/p>\n\n\n\n
Since you\u2019re here\u2026
<\/strong>Curious about a career in data science? Experiment with our free data science learning path<\/a>, or join our Data Science Bootcamp<\/a>, where you\u2019ll get your tuition back if you don’t land a job after graduating. We\u2019re confident because our courses work \u2013 check out our student success stories<\/a> to get inspired.<\/p>\n","protected":false},"excerpt":{"rendered":"
Understanding Word Embeddings Part 1: Applications Written by Aaron Geelon So If you already have a solid understanding of word embeddings and are well into your data science career, skip ahead to the next part! Human language is unreasonably effective at describing how we relate to the world. With a few, short words, we can […]<\/p>\n","protected":false},"author":10,"featured_media":2792,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_eb_attr":"","_eb_data_table":"","footnotes":""},"categories":[67],"tags":[],"marketing_tags":[],"class_list":{"0":"post-2776","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-science"},"acf":[],"_links":{"self":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/2776"}],"collection":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/comments?post=2776"}],"version-history":[{"count":4,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/2776\/revisions"}],"predecessor-version":[{"id":49277,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/2776\/revisions\/49277"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media\/2792"}],"wp:attachment":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media?parent=2776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/categories?post=2776"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/tags?post=2776"},{"taxonomy":"marketing_tags","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/marketing_tags?post=2776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
         This method, called word embeddings<\/i>, has a lot of promise; it might even be able to reveal hidden structures in the world we see. Consider one relation it discovered<\/span><\/a>:<\/span><\/p>\n\n\n\n