Web scrapping: things you will not know before getting your hand dirty

What is Web scrapping?

Web scrapping is a important part of data science. Data science is not only use and analysis of the already created, usable data; but also part of it is to search and create that data. Many times, you will come across a general question in data science field for which you may not find a ready made data in CSV or JSON. So, the solution is Web scrapping. 

Web scrapping means to scrap the web, i.e. programatically extracting information needed from one or more than one web page. Web scrapping also goes into detailed soft-wares named web crawlers or so called spiders.  

Web scrapping can be done in many languages. For this and the coming posts, I am going to use python for web scrapping. I will show examples where the easiest approach for web scrapping works and where it does not. Also, on later posts, we will explore a bit complex methods and modules to do web scrapping.

Introduction to methods:

There are broadly two methods to do web scraping. One is using simple http request to access the sites and then mine them. The second one, far more complicated, is using the automation tool selenium; which is mainly used for web application testing.

(1) First process, at least for python, you can scrap the webpage using a library called "BeautifulSoup". This is a very useful library and mainly used for web scraping. I will not go into a detailed documentation kind of description of the same, but if one is interested, one can find here. 

Now, I will provide code snippets and describe different scenarios about web scraping. First, we will go on to scrap reddit.com; which is, being kind enough, lets you scrap the page. 

here, if you notice the code throughly; you can see that, I have used the urllib library of python 3 here. Please be aware to use the python 3 version libraries in your code if you are using python 3; otherwise, in case of python 2, this library is something different which I am not going to mention.
urllib.request is the python package used to send requests to the sites for access. Other libraries are out of scope for this current post. But, if you want to work with more on these url requesting topic, you must definitely check out urllib modules.

Now, here, I have created a variable named page_url, where I have put the website url. Now, putting this variable as the parameter for urllib.request.urlopen() opens the corresponding webpage, and then saves the source in the variable to which the function returns.

Now, we will use BeautifulSoup module. Note that, this has to be imported from the bs4 module further.
In beautifulsoup, there are multiple html parsers available; i.e. which are specifically suitable for HTML5, XML files etc. Here, we are going to use the html.parser to parse the downloaded html file.

Now, this is important to understand that, although the webpage source gets downloaded in the page variable in the above code; but it is not at all easy to find any link or text out of the html and CSS mess which will get downloaded in that.
Also, there occurs number of issues with the webpages when it detects a machine request; but more on that later; specifically, we will discuss the different problems before getting into  selenium, to show the importance of selenium in avoiding all those problems.

Whatever, so, in the next cell, we parse through the downloaded page source using BeautifulSoup and create a soup object out of it.
To see the source with proper indentation; you can write:
print(soup.prettify())
which basically prettifies the source code and then prints it in a well manner.

This, soup, is a object where each divs and tags are children of their parent classes and therefore the whole html-css code forms a tree like object , which is the soup.
As web scraping is too deep and if one starts to show examples of doing the work then one will have to write a book; so I will stay in the scope of the post, and will keep it brief.

We will show, how to find basic tags and then at the end of this post, we will show a piece of code, which can take user input and sum up a paragraph on that, scraping the wikipedia.

Searching tags and classes:

(1)

For searching specific tags, i.e. breaks (</br>) , paragraphs (<p>) , divs (<div>) etc, you have to use the function soup.find() and  soup.find_all().

i.e. observe the following code snippet:

Notice the level at which the html code can look cryptic when it comes for a commercial website. The above code, searches for 'a' in the tags, and therefore, it effectively finds out all the <a href> references.

(2)
For finding something specific, you can use the find function; i.e.
soup.find(attr={#something})

this  will give you a result of one such attributed class or tag from the html source code.

(3)
To find some tag/class with specific mentions like a div with specific class name "Ih30" or something like that, you have to use the same type of find function but then the format changes to soup.find_all(class_name,{attr: specific name }) where the attribute will be inside that class named tag.
Look at the code below to understand one such implementation:

(4)
To find the text written in between the opening part and the closing part of a tag, you can use the following format:
soup.tag.text.strip() which will give you the text inside the tags. Without using the strip(), you would get teh tags attached to the text, which is of no good.

Also, there are multiple ways of parsing from one element to the next in the tree structure of soup. But we will not get into that.

We have explored the surface of use of beautifulsoup; and how to find tags and classes.

The paragraph creator

Now, I will show a small application to create a paragraph about anything out of wikipedia:
first see the code below on your own and try to figure out.

I hope that uptil now, the first four lines are clear from above description. Now, the text variable basically is a list of all the paragraph tags. Then we initiate a empty string.
Now, as we have discussed earlier, in each loop, the num refers to one paragraph tag, and the loop, at each go, takes one para tag, strip it from the html additions, and then we add it with the string. At end of each looping, we add three empty lines.
In this above code snippet, we have hard coded the website "https://en.wikipedia.org/wiki/Pattern". In a better code, we will want to search just by putting the keyword.
This, in turn, is still in development for me.

Conclusion:

For scraping websites, you will need good understanding of HTML, CSS and basic javascript; and then using BeautifulSoup and urllib library of python, one can really download source of the webpage, search for tags, texts, images and links and therefore properly derive data from it. 

In the next post, we will discuss about problems with this simplistic approach and therefore extend to selenium. The post exploring selenium is here. 

Thanks for reading!