word similarity using spacy

 Introduction:

 

In text mining algorithms, as well as nlp based data modeling, word similarity is a very common feature. Word similarity in nlp context refers to semantic similarity between two words, phrases or even two documents. We will discuss how to calculate word similarity using spacy library.

what is similarity in NLP and how is it calculated?

In NLP, lexical similarity refers between two texts refer to the degree by which the texts has same literal and semantic meaning. i.e. how much similar the texts mean; is calculated by similarity metrics in NLP.

There are many different ways to create word similarity features; but the core logic is mostly same in all cases. The core logic in all these cases is to create two representative vectors of the two items; using either universal vectors created from pre-trained models like word2vec, glove, fasttext, bart and others; or using the present document and using different methods like tf-idf match, pagerank procedures, etc. 

whatever process you use, in the end, the two vector representations are "compared" to produce a relevance of the two items. The most common procedure for comparison is cosine similarity, with less popular methods including considering different varieties of cosine similarity, correlation and other complex methods.

Generally, word similarity ranges from -1 to 1 or can be also normalized to 0 to 1. The lower values stand for low relevance; and as the relevance increases, the semantic similarity increases between the words as well. 

Now, let's see how does spacy solve this very common problem of calculating similarity between words/docs.

Similarity calculation using spacy:

Here onwards, I will assume that the readers know basic spacy techniques; and if you are not familiar with that please read this introduction to spacy and then continue.

First of all, let's just remember the fact that there are 3 types of word objects in spacy; (1) docs (2) tokens and (3) spans. Docs refer to doc objects made from texts analogous to paragraphs or full documents; while tokens refer to word like chunks which represents the most atomic parts of a doc. spans are continuous list of these tokens; i.e. analogue of a phrase.

Now each of these objects, doc, token and span has similarity() method, which allows us to calculate their similarity with any other type of text object. See the snippet below:

import spacy
nlp = spacy.load('en_core_web_sm')
text = 'the fat dog runs around the fountain.'
text2 = 'the little cat sleeps near the fountain.'
doc = nlp(text)
doc2 = nlp(text2)
token = doc[2]
token2 = doc2[2]
span = doc[:3]
span2 = doc2[:3]
print(doc.similarity(doc2))
#0.876536
print(doc.similarity(token2))
<stdin>:1: ModelsWarning: [w007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. 'en_core_web_sm', which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.
0.412095152223016

Note the warning. This warning says, that as we have loaded the small spacy model; therefore there is no real vector loaded and the similarity measure is created using ner and pos taggers and similar signs. The reason behind this is that to optimize the memory usage, spacy doesn't load any real word embedding for the vocabulary it uses when it loads the smaller models. Therefore, to use the actual vectors and get better accuracy, we need to load either the medium model i.e. en_core_web_md or the large model i.e. en_core_web_lg. The large model contains a very large vocabulary with unique vectors for more than a million words.

Now, on using the bigger model, one can also load the vectors directly. In such case the doc.vector, span.vector and token.vector attributes provide the vectors of 300 length vectors which are used inside spacy for calculating similarity. Look at the following example where we load large model en_core_web_lg and see the vector of a document.

import spacy
nlp = spacy.load('en_core_web_lg')
text = 'the fat sister was running towards the plate.'
doc = nlp(text)
print(doc.vector)

There are vectors for most of the common words. But for the uncommon and words for which there is no trained vector, they get assigned to the zero vector in this setting. i.e. it means that they are too uncommon and in sense of similarity, they are similar to no one.

There are a few more attributes, like token.vector_norm which gives you the L2 norm for normalizing your vector; token.has_vector which tells you whether there is real vector attached or not and so on. Also token.oov i.e. out of vocabulary tells you whether the current token is out of the assigned vocabulary of the model or not. 

There are ways to customize word vectors and use different methods to create these custom vectors for better usability. That is somewhat out of scope for this article. We will discuss that in another article. You can read it from the official site for further understanding.

A few questions:

(a) what is the length of spacy vectors?

spacy vectors for english words come in two bunchs, one from en_core_web_md with 50 length and another from en_core_web_lg with 300 length. The 300 length vectors are better in quality by few performance points, but the 50 length is smaller in size and easier to use.

(b) how spacy similarity is calculated?

internally, spacy similarity is a cosine similarity between the internal vectors for the document spacy creates. Here is the main line regarding this. 

return numpy.dot(self.vector, other.vector) / (self.vector_norm * other.vector_norm)

(c) How is the pretrained vectors created?

According to spacy documentation, spacy uses the word2vec technique on a specific corpus and have got their vectors. So in short, spacy vectors are similar to word2vec.

Conclusion:

So in conclusion, to calculate similarity using spacy for two text parts, you have to create docs out of them using nlp(text) and then use doc1.similarity(doc2) to get the similarity. So spacy really simplifies similarity calculation in this way. Thanks for reading! stay tuned for more awesome nlp articles.