Machine learning and statistics with python

 Introduction: We have discussed different aspects of spacy in part 1 , part 2 and part 3 . Now, up to this point, we have used the pre-trained models. Now, in many cases, you may need to tweak or improve models; enter new categories in the tagger or entity for specific projects or tasks. In this part, we will discuss how to modify the neural network model or train our own models also; as well as different technical issues which arise in these cases. How does training works: 1. initialize the model using random weights, with nlp.begin_training.  2. predict a bunch of samples using the current model by nlp.update 3. Compare prediction with true labels, calculate change of weight based on those predictions and finally update the weights. 4. finally reiterate from 2. Refer to the following diagram for a better understanding: Now in practice, the goal is to create annotated data using prodigy, brat or easily using phraseMatcher feature of spacy. Using phraseMatcher we can quick...