Machine learning and statistics with python

 Introduction: One of the main application of spacy is to perform named entity recognition or ner activity with it.  Named entity recognition stands for the NLP task where we detect named entity or entities such as organization, person, location etc different entity names. Spacy provides built-in methods to detect named entity recognition. In this article, we will go through the basics of detecting the different ner and using the spacy ner features. What is NER and what are the different labels? NER or named entity recognition, is the procedure to detect named entities using natural language processing algorithms. In spacy, universal ner labels are detected by default. So there are the following ner in this ner label list: (1) PERSON: this represents any kind of person. This includes fictional and real people. fictional basically refers to cartoon, movie or book characters etc. (2) NORP: nationalities or religious or political groups. This refers to basically a type of co...

 Introduction:  In languages which are non-english and non-european as well, NLP work has progressed slowly in the last few decades because of the lesser number of scholars working on them as well as a lack of global interest in them. But now the time has changed and people from all over the world are collaborating on these lesser explored libraries and they are building resources for working on these languages with the same ease with that of english.  Pyarabic is a package created from such a similar effort which deals with the intricate details of the arabic language and helps processing all kinds of arabic texts. While trying to learn it, being from a non-arab background, I couldn't read lots of parts of the main readthedocs site and had to work my around it. So in this blog post, I will summarize my learnings in english language, so that you can learn it and use the package with much more ease than me. [Credit where credit is due: this article heavily uses the ac...

Introduction: I didn't start my career with java, but with python and a bit of C. Then years after, I am learning java for android projects. In this blog, I will point out how I am learning java and also a little bit of resources and information along the way to help you learn.  This is more for a person who already is familiar with the programming concepts but is getting accustomed with the java syntax. Also it only spans to very basic and if people like it then only I will proceed with further details. I have taken guidance and screenshots from the android course by rob percieval for this blog. If you want to learn android, do take his course. Thanks and lets begin. Initial observations:  (1) a java code needs to have a class to start with. Also each class has method or functions associated to a class. among these methods, it must have the main method. (2) void means it returns nothing. (3) static means some variable which doesn't change at all. it is only dependent on ...

Introduction: In this last post , we discussed the basics of pyspark. Now, in this post, we will discuss how to do machine learning in pyspark. We will discuss what are the main machine learning pipeline elements, and how to use them too. The content is taken from datacamp ditto, and the sole credit of writing the blocks go to datacamp pyspark course. I am merely compiling it together for you to go through fast and learn quickly, with the completed exercises and the full flow. Machine Learning Pipelines You'll step through every stage of the machine learning pipeline, from data intake to model evaluation. Let's get to it! At the core of the pyspark.ml module are the Transformer and Estimator classes. Almost every other class in the module behaves similarly to these two basic classes. Transformer classes have a .transform() method that takes a DataFrame and returns a new DataFrame; usually the original one with a new column appended. For example, you might use t...

 Introduction:  Pyspark is one of the first big data tools and one of the fastest too. In this article, we will discuss the introductory part of pyspark and share a lot of learning inspired from datacamp's course. The first step: The first step in using Spark is connecting to a cluster. In practice, the cluster will be hosted on a remote machine that's connected to all other nodes. There will be one computer, called the master that manages splitting up the data and the computations. The master is connected to the rest of the computers in the cluster, which are called worker . The master sends the workers data and calculations to run, and they send their results back to the master. Creating a connection to spark: Creating the connection is as simple as creating an instance of the SparkContext class. The class constructor takes a few optional arguments that allow you to specify the attributes of the cluster you're connecting to. An object holding all these att...