Introduction to Support vector machine

                                       Introduction

SVM or support vector machine is one of the most dominant classification algorithms in the traditional machine learning. svm is a supervised classification algorithm which mainly works in a two-group classification algorithm; by projecting the features into a linear space; and then by drawing a hyperplane between the two classes thereby classifying. It is okay if you don't understand the terminologies I just used, as in the next few sections we will go through basic and advanced concepts of svm and by the end of this article, you will be confident to work with svm model.

The basics of SVM:

SVM is the abbreviation for support vector machine. In SVM is that in this algorithm, we try to find a hyperplane dividing the two classes of data with maximum distance between the support vectors of two classes. In case of SVM, we consider points which are near the hyperplane as support vectors; as we maximize the distance of each of the support vectors. So there are two main concepts:

(1) hyperplane: it is a decision boundary and a n-1 dimensional plane in a n dimensional linear feature space. This plane creates separation between two categories of data. 

(2) support vectors: these are data points which are nearest to hyperplane and the optimal hyperplane is chosen based on maximizing the distance to these hyperplane. 

Now, in the linear svm, the main objective function is:

which basically means that our goal is to both have minimum weights as well as to maximize the distance between the hyperplane and the feature points. The expression on the right basically stands for the distance between hyperplane and a feature point x.

So basically in a linear setting, we evaluate a weighted linear combination of the data points and then if its value is less than -1 we assign it to one category; while if its value is more than 1, we assign it to a separate category. The goal of the svm algorithm is generally to reach optimal weights such that the above mentioned objective function is optimized. And this gap between [-1,1] is basically the confidence of the algorithm.

So what happens when we introduce a new point?

Basically we calculate the weighted linear combination for this new data point. Given it falls in the -1 side of the hyperplane, it gets classified as the -1 class, while if the linear combination is more than 0, then it falls in the 1 side of the hyperplane and gets classified as 1 class.

So why do we need to minimize the absolute value of the weight?

If you understand correctly the above discussion; basically the ∑w_ix_i=2 is the portion which gives the algorithm confidence. So the more the weights, the lesser the confidence. This is why we need to minimize the weight.

so how does the weight gets updated?

The weight gets updated by taking partial derivative of the objective functions with respect to each n weights respectively associated to each features. Look at the formula which comes out from this.

 

                                          when there is no misclassification

                                   

                            when an example is mis-classified, we add it to the update

Hence this is again like the normal gradient descent type formula, and we keep updating the weights until they converge beyond basic tolerance value.

Now that we are clear with the basics, let's see how we can use SVM in modelling.

Using SVM with sklearn API:

So in work, you don't have to consider all these complicated theory to use SVM. scikit-learn has API for SVM. Here is the official link for the same. In this section, we will try out different types of SVM we can perform. You can check this kaggle notebook where I call different SVM methods and apply them on a sample dataset.

(1) SVC:

Here is a sample code on how to use SVC.

from sklearn.svm import SVC
clf = make_pipeline(StandardScaler(), SVC(gamma='auto'))
clf.fit(X_train_arr, Y_train_arr)

clearly, it follows the standard sklearn format of APIs where in the first line we have to define the object as we have done in this case with SVC(gamma = 'auto') and then we call the fit() method on the data. This fits a SVC ( support vector classifier) to the data. Linear refers to the kernel of the SVM. There is an option to choose the kernel in the SVC model. The default kernel which is used is 'rbf' kernel, which means that to calculate the distance with hyperplane, we use radial basis function instead of normal euclidean distance. We can also use polynomial kernels; which helps you create non-linear and higher degree decision boundary instead of normal straight hyperplane. 

Here is a sample code for using polynomial kernel:

clf = make_pipeline(StandardScaler(), SVC(kernel='poly',
                                          degree = 3,
                                          gamma='auto',
                                          class_weight = 'balanced'
                                          ))
clf.fit(X_train_arr, Y_train_arr)

Y_pred_valid = clf.predict(X_valid_arr)
print(classification_report(y_valid,Y_pred_valid))

Also note that we are using degree to specify what degree of polynomial to use. In case of using polynomial kernel, it is a good idea to fine tune this degree parameter. Also I will draw your attention to another parameter; which is class_weight. This is generally set to 'balanced' which provides inverse weight to each class based on its frequency, to resolve imbalance.

(2) NuSVC:

NuSVC is a variation of SVC algorithm where there is a parameter Nu, which we apply to control the number of support vector we use to calculate the cost function. The ability to control the Nu, let's user generalize the hyperplane and decreases chance of overfitting. If you are trying to fit NuSVC, make sure that you are fine tuning the nu parameter.

Here is a sample code on how to use NuSVC.

from sklearn.svm import NuSVC 

clf = make_pipeline(StandardScaler(), NuSVC(nu = 0.2, gamma = 'auto'))
clf.fit(X_train_arr, Y_train_arr)

Y_pred_valid = clf.predict(X_valid_arr)
print(classification_report(y_valid,Y_pred_valid))

So note that I have added nu = 0.2 in the NuSVC instantiation code. This value I obviously got after trying a number of values. Nu can range from 0 to 1; denoting what fraction of vectors I am letting the model use. 

Now, you may also note that I am using StandardScaler() in the pipeline always with SVC. This is important to use, as SVC minimizes the weight coefficient which is multiplied to the feature values. So that's why it is important that all the parameters are standardized and are therefore in the same range.

Now that you know how to use NuSVC also, we will quickly check the linearsvc.

(3) LinearSVC:

linearsvc is much similar to svc; but here a important point is that you need to set the dual parameter to false, so that your model converges within reasonable number of iterations. Other than that, don't use this model in case of a non-linear separation problem; as it literally fits a linear decision boundary.

clf = make_pipeline(StandardScaler(), LinearSVC(class_weight = 'balanced',
                                                dual = False,
                                                max_iter = 1000))
clf.fit(X_train_arr, Y_train_arr)

So as you can see, we have set the dual as False. If still there is a convergence issue, you will have to change your max_iter and increase it so that it can converge.

Conclusion and a practical note on svm:

So with this our discussion on SVM finishes. There are more advanced things as kernels, different distance functions, and how we can implement approximate calculations for calculating these kernels. But we are not going to discuss more into that in this article. And there are two reasons of this. First of all, these discussions again involve a lot around linear algebra, multivariate calculus and analysis. So these are mostly tough math. The second and more important reason is that you are probably never going to use it. If you have a problem where you can choose only SVM, then probably you can go on to read the reference articles, and learn all about it; but if you are allowed to choose other models, then probably you will end up finding some better and complex model. With that being said, try and use svm model as it is often fast and easy to tune, simple model and will come handy in quick modeling tasks with easily or linearly separable data. Please comment and share this article. If you liked this article, subscribe to get more of these directly in your mail. 

Thanks for reading!

References:

(1) math behind SVM from analytics vidhya

(2)picture of formulas are taken from: what are support vectors in svm model

(3) math behind svm part 2