Machine learning and statistics with python

 Introduction: One of the best methods of speeding up your python codes is turning it into a cython code. For turning a python script to cython, you need to save it with a pyx extension and then compile it using a setup file . For example, say you have a file named helloworld.pyx. To compile this, your code inside setup file will look like: from setuptools import setup from Cython.Build import cythonize setup ( ext_modules = cythonize ( "helloworld.pyx" ) ) Now, after doing this, you have to run this setup file using the following command which creates a .c and a .so file. $ python setup.py build_ext --inplace The Problem: So even after compiling, you may end up getting: importerror:... undefined symbol: _py_ZeroStruct when you try and import the compiled cython module.  Why does it happen? and the solution: the source of this problem, according to the stackoverflow is the mismatch between the python version of the compiled cython code and the python f...

 Introduction: Many times, as a data scientist, you will be working on cloud machines which are on either aws, gcp or something else. These machines are costly, and so when you run your codes on these machines, you are actively increasing the project spending. During my last session in aws, I encountered a moderately large program to run; which according to my estimation would take 48-72 hrs to run. And to decrease the time needed as well as optimize the operations, I took 2 steps. This post will briefly describe these processes. (1) cythonizing my codes: Let's face it. Python is SLOW. yes, python is slow and that's why most of the standard computation libraries are written on cython or c++ background. But being a python and pandas dependent data scientist, I write most of my codes in pure python. These codes, are very slow to run; when let's say compared to c++ or cython equivalent of the same codes. So the easiest way to reduce operation time; is to create cython li...