What is topic modeling?

                               What is topic modeling?

                                     Written by: Shyambhu Mukherjee

Introduction:

Topic modeling is one of the famous natural language processing tasks. Topic modeling refers to assigning one or multiple topic to a set of document. In this article, we will discuss the basic, medium and advanced understanding of topic modeling and discuss multiple python libraries which will be used to do topic modeling.

Summary of the article:

In this article, first we will define what topic modeling and topic classification modeling are; what are their difference and how to do both as a overview. Then we will discuss topic modeling procedures, such as LSA, LDA; in details. Finally we will end the discussion with conclusion and further readings.

What is topic modeling?

Topic modeling is a type of statistical modeling tool which is used to assess what all abstract topics are being discussed in a set of documents. Topic modeling, by its construction solves the problem of creating topic in an unsupervised manner.  

        Generally the statistical approach is used by considering that each document talks about a few different topics, while each topics are generally denoted by a distribution of words. i.e. there is a two step structure assumed about the documents, such as a document = (topic1, topic2, topic3, etc) and then again topic1 = (w1,w2,...etc).

Clearly, the topic modeling is generally performed by counting words, their proportions and related indicators. We will describe two models, LSA ( latent semantic analysis) and LDA ( latent dirichlet analysis) based on the idea we just discussed. 

Topic modeling is therefore an unsupervised machine learning method, which is used to model topics out of an unlabelled data and it can work therefore without any training. For obvious reasons, topic modeling is therefore a rough beginning approach rather than being a sophisticated and end solution.

What is topic classification modeling?

While topic modeling is an unsupervised modeling, we need to train models to our custom topics for high end and more accurate systematic usage. For example, if you are building a classification model for detecting support tickets, in that case you may want to assign specific topics to the tickets rather than creating topics from the tickets like we do in topic modeling. Therefore, a topic classification modeling is more of a simple text classification modeling where we classify the text into multiple topics.

What is the difference between topic modeling and topic classification modeling?

As we described in the above paragraphs, clearly, topic classification is a supervised process, where as topic modeling is a unsupervised modeling effort. In general modeling effort, we start with topic modeling, then find out the topics coming up naturally. Finally we create human curated topics which we use for topic classification modeling.

The two different topic modeling techniques:

In this article, we will discuss the two main ways to do topic modeling. These two methods are:

(1) LSA: latent semantic analysis

(2) LDA: latent dirichlet analysis

Let's now discuss the basic ideas of these two methods.

(1) LSA:

LSA or latent semantic analysis, is a method using bag of word and term-document matrix to detect topic. The assumption behind LSA is that different topics have different distributions of words in them, as well as different topics have different distribution of the topics in them. 

To mathematically solve this idea, We first create a bag of words from the corpus. Then we create a word-document frequency matrix where in the rows we have words, and in the columns we have the different documents. A cell (i,j) of the matrix therefore denotes how many times the word w_i has occurred in the document D_j.

Now we use the above assumption in LSA to factor this matrix into three matrices, which are:

(1) word-topic frequency matrix: in this matrix, words are in row, topics in columns. each cell denotes the occurrence of a word in the respective column.

(2) topic importance matrix: this is a diagonal matrix, where i th diagonal element of the matrix denotes the importance of the topic i.

(3) topic-document matrix: in this matrix again, topics are in row, documents are in the column, each cell denotes the weight of a topic in the respective document.

Refer to this datacamp diagram to understand the structure:

Now, you may ask how this matrix factorization is done. In this case, we use something called SVD matrix factorization. Using SVD, we factorize a matrix M as follows:

M = UDV

where U is a unitary matrix, D is a diagonal matrix and V is again a unitary matrix. This ensures that the middle matrix is a diagonal. Now you may ask that why do we use this specific matrix factorization. 

The answer to that is that from linear algebra consideration, SVD changes a linear map into different basis representation so that it becomes D. For example, because U and V are unitary matrices, so we can write,

M = UDV => U^TMV^T = D

Which sort of can be interpreted that the term frequency matrix is basically represented by creating a basis made of topics ( which are linear combination of words) and then representing the map from words to matrix by another map from topic to words. 

And therefore, as original SVD represents the diagonal matrix as svd ( some sort of importance value) for the changed basis representations; which implies clearly that the diagonal matrix in this case represents the weights of the topics ( which are the new bases).

Now that we have both discussed the basic idea and the theoretical understanding of why the process makes sense, we will discuss LDA next.

(2) LDA:

LDA is a significant improvement from LSA in the context that LSA considers no probabilistic determination inside the document structures. In LDA, this is the major change. Before jumping into the concepts, we will cite few lines from the original LDA paper[link provided in reference].

"A significant step forward in this regard was made by Hofmann (1999), who presented the probabilistic LSI (pLSI) model, also known as the aspect model, as an alternative to LSI. The pLSIapproach, which we describe in detail in Section 4.3, models each word in a document as a sample from a mixture model, where the mixture components are multinomial random variables that can be viewed as representations of “topics.” Thus each word is generated from a single topic, and different words in a document may be generated from different topics. Each document is represented as a list of mixing proportions for these mixture components and thereby reduced to a probability distribution on a fixed set of topics. This distribution is the “reduced description” associated with the document."

After hofmann, still no proper probabilistic model is provided for documents. In LDA ( latent dirichlet allocation), we solve this issue. We will discuss the LDA's construction below:

Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words. LDA assumes the following generative process for each document w in a corpus D:

(1) Choose N which follows a poisson distribution with lambda = e.
(2) Choose theta which follows a dirichlet distribution with alpha = a.
(3) for each word w_n for n in N:
      (a) choose a topic z_n which follows Multinomial distribution with theta.
      (b) choose a word w_n which comes from p(w_n|z_n, beta), which is a multinomial probability conditioned on topic z_n.

From this, we can calculate the probability distribution of the unigrams theoretically dependent on alpha, beta. We get a distribution from the actual dataset too for the different unigrams. 

To finally solve the problem, we need to optimize alpha and beta for decreasing the kullback leibler divergence distance between the theoretical and the observed probability distributions. The math behind this optimization is complicated, so we are not discussing it in this brief article. For quenching your thirst of knowing that, read the original paper mentioned.

Conclusion:

In this article, we explored what is topic modeling and topic classification modeling; learnt their difference and explored the different procedures and their theoretical assumptions. In our next post, we will see the programmatic side of topic modeling and explore different libraries related to it. Stay tuned and subscribe to our blog for more such awesome contents.

References:

(1) topic modeling monkeylearn

(2) kaggle topic modeling dataset

(3) LDA original paper