Posts
Introduction:
In languages which are non-english and non-european as well, NLP work has progressed slowly in the last few decades because of the lesser number of scholars working on them as well as a lack of global interest in them. But now the time has changed and people from all over the world are collaborating on these lesser explored libraries and they are building resources for working on these languages with the same ease with that of english.
Pyarabic is a package created from such a similar effort which deals with the intricate details of the arabic language and helps processing all kinds of arabic texts. While trying to learn it, being from a non-arab background, I couldn't read lots of parts of the main readthedocs site and had to work my around it. So in this blog post, I will summarize my learnings in english language, so that you can learn it and use the package with much more ease than me.
[Credit where credit is due: this article heavily uses the actual documentation provided in the reference link in the end. I am very much grateful to the authors for being able to work on arab nlp.]
Features:
First things first; lets get our basics known. The pyarabic package is written by taha zerrouki (http://tahadz.com/). Here are the following features mentioned for this library:
- Arabic letters classification
- Text tokenization into words or sentences
- Strip Harakat ( all, except Shadda, tatweel, last_haraka)
- Sperate and join Letters and Harakat
- Reduce tashkeel
- Mesure tashkeel similarity ( Harakats, fully or partially vocalized, similarity with a template)
- Letters normalization ( Ligatures and Hamza)
- Numbers to words
- Extract numerical phrases
- Pre-vocalization of numerical phrases
- Unshiping texts
So, basically all the basic NLP stuff. But to know more into what these tasks stand for, we have to know what harakat, tashkeel and all other terms stand for.
Harakat/tashkeel:
Harakat translates to short vowel in arab. There are different vowels such as shadda, tanween etc which sit on the arabic letters and change their meaning.
Cases:
There are no upper case or lower cases in arabic.
How to tokenize arabic text?
I think in NLP, the most bare minimum thing to achieve is to do tokenization of texts. For arabic, you can simply do the following:
from pyarabic import araby
text_tokenized = araby.tokenize(arabic_text)
and simply the text_tokenized will have the list of tokens.
The tokenize function also allows different morph ( transformation and modification of the text) and conditions. The morph and conditions take both strings of function names which are built in to the pyarabic package or callable/lambda functions directly passed to be used for tokenization. Let's explore it with 2 examples from each.
(a) example of built in functions being passed:
(b) example of callable being used:
Also, one can use the sentence_tokenize method to break down a paragraph or so into the different sentences. Checkout the below example for the same:
There are other utility functions related to tokenization; but you can check the documentation or figure out using simple python code yourself only. Next lets check how to handle numbers in arabic using this package.
how to extract numbers and deal with numbers:
There is a specific number.py file in the pyarabic language. All the number manipulations are done using this script from the package.
Let's take a look into a few use cases.
(a) how to turn arabic numbers into strings using pyarabic?
Here we use int2str function from the ArNumbers class of the number.py file of the pyarabic package. Check out the example below how it spells out 125.
(b) how to turn arabic text into numbers using pyarabic?
For this we use text2number function from the number.py script. In this example, it changes the spelled out number into western digits 523.
(c) How to count or find out number words in a text?
The necessary function for this is extract_number_phrases. Check the example below. It finds out 2 different number phrases from the given text string.
(d) How to normalize all types of numerals into same numeral format?
For this we will have to use a different script which is for translation, named as trans. You can use it like
from pyarabic.number import *
Check the following example how it works.
So these are the different kind of number manipulations one can do with pyarabic. Let's focus on other things.
It is known that, in arabic, it is often normal to write without the tashkeel or the harakats. A bunch of different functions are there to help us normalize our texts therefore by stripping it off those vowels and thereby normalizing them.
So let's take into a look into
Normalizing arabic texts by removing vowels:
(a) stripping diacritics:
Most of the times, the vowels are applied as diacritic marks on the arabic text. To strip diacritic therefore means removing harakat and small letters. You can use strip_diacritics function for the same just like below:
(b) reducing tashkeel :
In this case we only delete the tashkeels. The function to use for this is reduce_tashkeel. Check how we are using it below:
(c) strip_tatweel and strip_shadda are similarly used to strip the text off tatweel and shadda.
(d) other than these, there are normalizing functions such as normalize_hamda and so on. For learning about them and what specific cases they are used in, you can follow the official documentation.
Conclusion:
In this article, we have briefly explored what pyarabic is, how to manipulate and tokenize text, count or transform numbers into string and vice versa; and how to handle vowel normalization and vowel stripping which is a specific characteristic to the arabic language itself. This is good enough to get you started with some real nlp in arabic. But note that further critical attributes such as sentence parsing, parts of speech tagging are not provided by this library currently; which makes this library significantly weaker to the general comparison in other language's dominant libraries. But withstanding that, pyarabic stands for the first big nlp library for arabic natural language processing toolkits.
Please comment, share and let me know if any mistake, improvement, change or addition is needed to be made to the article.
References:
(2) arabic alphabet
Comments
Post a Comment