Machine learning and statistics with python

Introduction Many times, you will want to search through a dataframe so many times that both writing the search queries using dataframe rules and programmatically the search operation becomes redundant. In such cases, you will want to create a dictionary from a pandas dataframe. Now, if you wonder how, then I have the solution. Read on to explore more. to_dict() method: Possible is that you want to turn the rows into values while keeping one specific column as index. In such a case, you have to use the method as follows df.set_index('id').to_dict() zip and dict method: You can use a simpler method and can get done in one line, if your goal is to turn one specific column into keys and one specific column into values, then you can use the zip and dict method. The details follows as below: data_dict = dict(zip(data[key_column],data[value_column])) Duplicate document problem: This is a case when you have multiple values for each key. In that case, it will become a bit...

Introduction: Hi guys! I have started working on my Kaggle profile recently. So, this is the first of many, Kaggle project discussions I am going to write in this blog. One of the very famous projects in Kaggle has been the house price prediction data. This data is often used by beginners like me for learning and demonstrating regression. But the reason I have used this data has been to mainly demonstrate the usefulness of a random forest model as a successor of my random forest theory post. If you have not worked on, or read about the random forest before this, please read the random forest for beginner post. In this link, I have curated the complete description for random forests' theory, algorithm, tuning details and every other specific thing. My stand Now, let's begin with the house price prediction data. The full jupyter notebook can be found in this Github link. The achieved score is 0.15411, with which I ranked 2863 in the leaderboard, therefore, it is not a ver...

Introduction: In day to day works of data analysts and scientists, you often need to write your results and present them in a nice and neat csv file or a excel file. In this article we will discuss how to write csv files from python scripts. It is important for not only data science but anyone who has to present data on a regular basis. The basic option: pandas.dataframe.to_csv attribute This option uses pandas library.If you are not introduced to pandas yet, read about pandas here first. When you have a data table in your environment which you want to save, then the best way is to put it in a dataframe version and then use the to_csv attribute of pandas dataframe to save it into a specific file. The normal use would look like: To clear the idea, the normal format to use to_csv on a dataframe is: dataframe.to_csv(destination_file_path_as_string,index = False) The above example depicts how you would save marks for 4 students. We first create a empty dataframe. Then we stor...

A/B testing significance checker: Complete your A/B testing now! Impressions in normal group Clicks in normal group Impressions in variation Clicks in variation Show me A/B test result! result This is a very basic template A/B testing tool for clicks impression scenario. We have used normality assumption and sattertheid approximation for the calculation of significance. If you want to re-enter values, refresh the page and enter the new values. Thanks! Sponsored Ads Learn a/b testing in python from Udemy

Introduction: I think I am eligible to answer this question as I am a student of ISI Bangalore and have cracked IIT also successfully. So, I stood on that path. See, in this place, you have to choose your decision carefully. First, ask yourself the following questions: how much do you love mathematics? what is your long term goal? Being a researcher or being a top-notch industrialist? These two questions are sorting questions. If you are not dedicated enough to mathematics, and love physics, programming, and other staff also more than or equal to mathematics, then you should go for IIT. Maybe you want to become a big industrialist, then you should go to IIT for sure. In IIT, you will be doing engineering and therefore will be exposed to a lot of subjects and will learn diverse things. Now, let's say you are a math nerd, you solved a lot of RMO and INMO level problems, want to know a lot of mathematics in your life, boss, you are welcome to ISI. Now, you are like ...

Introduction: I have been seeing people ask this question many times in quora as well as in conversations too. So I thought I will take time and I will answer this question outright in a blog post. So, two of my friends from ISI went through this exam this year. It depends on your background. I will suggest you prepare your statistics, probability, and general bachelor's level mathematics properly. For basic level probability, I will suggest you go through Introduction to probability by Sheldon ross; read the concepts, solve the examples and proceed. This book is good enough to prepare you for the probability questions you will face. Now, let’s come to the statistics part. For statistics, I will suggest reading through Casella Berger and CR Rao. See, these books are pretty rigorous, and maybe you can find it really hard for you. But, You can skip some of the hard theoretical parts, and follow the flow basically. That will give you a good knowledge of what can come. Now,...

Introduction: For a little more than a year now, I have been learning and practicing data science, and data science have thoroughly amazed me throughout this journey. But a question strikes real hard when an online coaching company asked me last month; "what is data science?" and I suddenly lost my words. Because, grasping a thing you daily work on, think about and try to become creative about, becomes abstract to you slowly, such that you can't grasp that anymore all in a few words. That is why, in this post, I will try to express my view on the small but big question, "what is data science?" We will discuss the following topics in this post: What is data science? What are the pillars of data science? Why do we need data science? What are the different data science roles? What are the different common data science problems in different industries How does a general data science problem get solved?: CRISP-DM What are the pre-requisites to be...

Introduction: Random forest is one of the highest used models in classical machine learning. Because of its robustness in high noisy data, and its much better ability to learn irregular patterns of data makes the random forest a worthy candidate for modeling in many fields like genomics and others. In this post, we will discuss: definition of a random forest What is the meaning of bagging What do you mean by ensemble model What is a decision tree The mathematics behind decision tree Training a Random forest Random forest packages Hyperparameter tuning Random forest How many trees should there be in Random forest Further study links definition of a random forest "Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest." --Leo Breiman Random forest is an ensemble model consisting of number of decision trees...

Calculate onpage now! basic calculator

Introduction: You have completed your linear regression fitting and prediction. But you want to know how to represent the accuracy of the linear regression. We discuss different accuracy metrics of linear regression in this post. (1) adjusted R-square and mean-squared-error:   In linear regression, the R-square is the measure of the accuracy of linear regression. The R-square can be from 0 to 1 i.e. it can also be interpreted as from 0 to 100%. Roughly speaking, R-square denotes the amount of variance in the data which is described by the linear regression. The more amount of variance gets described by the linear regression, the better the regression is. So, to describe how efficient a linear model-fitting has been, one can depend on how high the adjusted R-square percentage is.  While adjusted R-square is accuracy-measure from the statistical point of view, the more application point of view is the mean-squared-error (mse). Mean-squared-error is the mean of the...