Understanding Readability Score:Implement readability in python

Introduction:

In the vast landscape of written communication, readability score stands as a crucial metric, often overlooked but profoundly impactful. In essence, readability score measures the ease with which a reader can comprehend a piece of text. This score is determined by various linguistic factors such as sentence structure, word choice, and overall complexity. While seemingly technical, readability score plays a vital role in shaping effective communication across diverse contexts, from literature and journalism to academia and business.

At its core, readability score serves as a bridge between the writer and the reader, facilitating a smoother flow of information and ideas. Imagine trying to traverse a rugged terrain versus a well-paved road; similarly, text with a high readability score offers a smoother journey for the reader's comprehension. By analyzing factors like sentence length, syllable count, and vocabulary complexity, readability formulas provide a quantitative measure of how accessible a piece of writing is to its intended audience.

One of the earliest and most widely used readability formulas is the Flesch Reading Ease formula, developed by Rudolf Flesch in 1948. This formula computes readability based on average sentence length and average syllables per word, assigning a score between 0 and 100. A higher score indicates easier readability, with scores above 60 generally considered easily understood by the average adult reader.

However, readability scores extend beyond mere simplification of language. They serve as a powerful tool for tailoring communication to specific audiences. For instance, a technical manual intended for engineers might require a different level of complexity compared to a children's storybook. By adjusting sentence structure, vocabulary, and overall complexity, writers can fine-tune their texts to suit the cognitive abilities and background knowledge of their target readers.

Moreover, readability scores play a crucial role in digital communication and content creation. In the era of online content consumption, where attention spans are fleeting and information overload is rampant, the readability of a piece can determine its success. Websites, blogs, and social media platforms often employ readability analysis tools to optimize their content for maximum engagement. By ensuring readability, content creators can captivate and retain their audience's attention amidst the endless stream of online distractions.

In academia, readability score serves as a barometer for the accessibility of scholarly research. While academic writing has traditionally been characterized by dense jargon and complex syntax, there is a growing recognition of the importance of making research findings more accessible to a broader audience. By striving for clarity and simplicity in their writing, researchers can enhance the dissemination of knowledge and foster greater engagement with their work.

Furthermore, readability score intersects with considerations of inclusivity and accessibility. Texts with high readability scores are more inclusive, ensuring that individuals with varying levels of literacy or cognitive abilities can engage with the content effectively. In educational settings, educators often use readability analysis to select textbooks and instructional materials that are appropriate for their students' reading levels, thus promoting equitable learning opportunities for all.

Despite its undeniable benefits, readability score should not be seen as a one-size-fits-all solution. Context, audience, and purpose remain paramount in crafting effective communication. While simplifying language may enhance readability, it should not compromise the richness of expression or depth of content. Striking the right balance between simplicity and sophistication is key to achieving optimal readability without sacrificing substance.

Implementing readability in python:

textstat:

One popular Python library for calculating readability scores is textstat. Below is an example of how to calculate readability scores using this library:

python

import textstat

def calculate_readability(text):
    # Flesch Reading Ease Score
    flesch_score = textstat.flesch_reading_ease(text)
    
    # Flesch-Kincaid Grade Level
    flesch_grade = textstat.flesch_kincaid_grade(text)
    
    # Automated Readability Index
    ari_score = textstat.automated_readability_index(text)
    
    # Dale-Chall Readability Score
    dale_chall_score = textstat.dale_chall_readability_score(text)
    
    # Coleman-Liau Index
    coleman_liau_score = textstat.coleman_liau_index(text)
    
    # Gunning Fog Index
    gunning_fog_score = textstat.gunning_fog(text)
    
    # SMOG Index
    smog_score = textstat.smog_index(text)
    
    return {
        "Flesch Reading Ease Score": flesch_score,
        "Flesch-Kincaid Grade Level": flesch_grade,
        "Automated Readability Index": ari_score,
        "Dale-Chall Readability Score": dale_chall_score,
        "Coleman-Liau Index": coleman_liau_score,
        "Gunning Fog Index": gunning_fog_score,
        "SMOG Index": smog_score
    }

# Example text
example_text = """
Readability is a measure of how easy a piece of text is to read and understand.
There are various formulas and indices that can be used to calculate readability scores.
These scores help writers ensure that their text is accessible and comprehensible to their intended audience.
"""

# Calculate readability scores
readability_scores = calculate_readability(example_text)

# Print readability scores
for score_name, score_value in readability_scores.items():
    print(score_name,score_value)

In this code:

• We import the textstat library.
• We define a function calculate_readability that takes a text as input and calculates various readability scores using functions provided by the textstat library.
• We define an example text.
• We call the calculate_readability function with the example text and store the results in a dictionary.
• We print out the calculated readability scores.

You'll need to install the textstat library if you haven't already using pip install textstat.

readability:

Another Python library commonly used for calculating readability scores is readability. Below is an example of how to calculate readability scores using this library:

 
commandline python example: 
>>> import readability
>>> import syntok.segmenter as segmenter
>>> text = "this is an example sentence. note that tokens will be separated by spaces and sentences by newlines"
>>> tokenized = '\n\n'.join('\n'.join(' '.join(token.value for token in sentence) for sentence in para) for para in segmenter.analyze(text))
>>> print(tokenized)
this is an example sentence . note that tokens will be separated by spaces and sentences by newlines
>>> results = readability.getmeasures(tokenized, lang = 'en')
>>> print(results)
OrderedDict([('readability grades', OrderedDict([('Kincaid', 9.781176470588235), ('ARI', 9.788823529411765), ('Coleman-Liau', 10.820402000000001), ('FleschReadingEase', 55.21529411764709), ('GunningFogIndex', 13.858823529411765), ('LIX', 46.41176470588235), ('SMOGIndex', 12.486832980505138), ('RIX', 5.0), ('DaleChallIndex', 10.052641176470589)])), ('sentence info', OrderedDict([('characters_per_word', 4.823529411764706), ('syll_per_word', 1.588235294117647), ('words_per_sentence', 17.0), ('sentences_per_paragraph', 1.0), ('type_token_ratio', 0.9411764705882353), ('characters', 82), ('syllables', 27), ('words', 17), ('wordtypes', 16), ('sentences', 1), ('paragraphs', 1), ('long_words', 5), ('complex_words', 3), ('complex_words_dc', 6)])), ('word usage', OrderedDict([('tobeverb', 2), ('auxverb', 1), ('conjunction', 1), ('pronoun', 2), ('preposition', 2), ('nominalization', 1)])), ('sentence beginnings', OrderedDict([('pronoun', 1), ('interrogative', 0), ('article', 0), ('subordination', 0), ('conjunction', 0), ('preposition', 0)]))])
>>> print(results['readability grades'])
OrderedDict([('Kincaid', 9.781176470588235), ('ARI', 9.788823529411765), ('Coleman-Liau', 10.820402000000001), ('FleschReadingEase', 55.21529411764709), ('GunningFogIndex', 13.858823529411765), ('LIX', 46.41176470588235), ('SMOGIndex', 12.486832980505138), ('RIX', 5.0), ('DaleChallIndex', 10.052641176470589)])
 
functional coding version 

from readability import Readability

def calculate_readability(text):
    # Create a Readability object
    tokenized = '\n\n'.join('\n'.join(' '.join(token.value for token in sentence) \
                for sentence in para) \
                for para in segmenter.analyze(text))
    
    results = readability.getmeasures(tokenized,lang = 'en')
    return results['readability grades']
 
 
 # Example text
example_text = """
Readability is a measure of how easy a piece of text is to read and understand.
There are various formulas and indices that can be used to calculate readability scores.
These scores help writers ensure that their text is accessible and comprehensible to their intended audience.
"""

# Calculate readability scores
readability_scores = calculate_readability(example_text)

# Print readability scores
for score_name, score_value in readability_scores.items():
    print(score_name,score_value)

In this code:

• We import the Readability class from the readability library.
• We define a function calculate_readability that takes a text as input and calculates various readability scores using methods provided by the Readability class.
• We define an example text.
• We call the calculate_readability function with the example text and store the results in a dictionary.
• We print out the calculated readability scores.

You'll need to install the readability library if you haven't already using pip install readability.

textacy:

Another Python library for calculating readability scores is textacy. This library provides various text analysis functionalities, including readability metrics. textacy is part of spacy universe and textacy uses spacy models under the hood. So to use texacy we need to download spacy and enable it first. Interested readers can read about spacy from my book on spacy.  below is an example of how to calculate readability scores using textacy:

first enable spacy:
$ pip install textacy
$ python -m spacy download en_core_web_sm
>>> import textacy
>>> text = (
...     "Many years later, as he faced the firing squad, Colonel Aureliano Buendía "
...     "was to remember that distant afternoon when his father took him to discover ice. "
...     "At that time Macondo was a village of twenty adobe houses, built on the bank "
...     "of a river of clear water that ran along a bed of polished stones, which were "
...     "white and enormous, like prehistoric eggs. The world was so recent "
...     "that many things lacked names, and in order to indicate them it was necessary to point."
... )
>>> doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")
>>> print(doc._.preview)
Doc(93 tokens: "Many years later, as he faced the firing squad,...") 

>>> from textacy import extract
>>> list(extract.entities(doc, include_types={"PERSON", "LOCATION"}))
[Aureliano Buendía, Macondo]
>>> list(extract.subject_verb_object_triples(doc))
[SVOTriple(subject=[he], verb=[faced], object=[firing, squad]),
 SVOTriple(subject=[father], verb=[took], object=[him]),
 SVOTriple(subject=[things], verb=[lacked], object=[names])]
>>> from textacy import text_stats as ts
>>> ts.n_words(doc), ts.n_unique_words(doc)
(84, 66)
>>> ts.diversity.ttr(doc)
0.7857142857142857
>>> ts.flesch_kincaid_grade_level(doc)
10.922857142857143

In this code:

• We download spacy first.
  
• We import the textacy library.
• We make a spacy doc object out of the sample text using the en_core_web_sm spacy language object.
  
• We call the flesh_kincaid_grade_level function on the doc object and get the readability score.

You'll need to install the textacy library if you haven't already using pip install textacy.

 

 Comparison:

To compare the performances of the three libraries (textstat, readability, and textacy), we can evaluate them based on several factors:

1. Ease of Use: How easy is it to use the library and its functions to calculate readability scores?
2. Speed: How fast does the library calculate readability scores?
3. Accuracy: How accurate are the readability scores provided by the library?
4. Documentation and Community Support: Is the library well-documented, and does it have a strong community to provide support and guidance?

Let's evaluate each of the libraries based on these criteria:

1. textstat

• Ease of Use: textstat is straightforward to use with simple function calls for various readability scores.
• Speed: It is relatively fast for small to medium-sized texts.
• Accuracy: While it provides commonly used readability scores, some users have reported discrepancies compared to other libraries.
• Documentation and Community Support: The documentation is decent, but the community support might not be as extensive as more widely used libraries.

2. readability

• Ease of Use: readability is also easy to use with a simple API for calculating readability scores.
• Speed: It tends to be quite fast for small to medium-sized texts.
• Accuracy: It provides accurate readability scores, and its methods are widely used and trusted.
• Documentation and Community Support: The documentation is comprehensive, and it has good community support with active contributors.

3. textacy

• Ease of Use: textacy offers a broader range of text analysis functionalities, including readability metrics, which might require a bit more familiarity with the library.
• Speed: It might be slightly slower due to the additional functionalities it offers compared to libraries specialized solely for readability analysis.
• Accuracy: The readability scores provided by textacy are generally accurate, as it relies on well-established formulas.
• Documentation and Community Support: textacy has comprehensive documentation and a supportive community, especially for text analysis tasks beyond readability.

Overall Comparison

• For simplicity and speed, both textstat and readability are excellent choices.
• If you need additional text analysis functionalities beyond just readability, textacy provides a comprehensive solution.
• In terms of accuracy and community support, readability and textacy are both strong contenders.

Ultimately, the choice of library depends on your specific requirements, familiarity with the library's API, and preference for additional features beyond readability analysis.

Conclusion:

In conclusion, the evaluation of readability analysis libraries—textstat, readability, and textacy—reveals a nuanced landscape where each option offers distinct advantages depending on the user's needs and priorities.

While textstat and readability excel in simplicity and speed of calculating readability scores, with straightforward APIs and efficient processing for small to medium-sized texts, textacy stands out for its comprehensive text analysis functionalities beyond readability metrics. This versatility makes textacy an appealing choice for users seeking a broader toolkit for text analysis tasks.

Accuracy, documentation, and community support emerge as critical factors influencing the choice of library. Readability, with its widely trusted methods and robust documentation, boasts strong accuracy and community support, making it a dependable option for users prioritizing accuracy and reliability.

Ultimately, the selection of a readability analysis library hinges on factors such as the complexity of analysis requirements, ease of use, speed, accuracy, and the availability of supportive documentation and community resources. By carefully weighing these considerations, users can make informed decisions to meet their specific needs for assessing and enhancing the readability of textual content.

References:

[1] [textstat]: kaggle code

[2] github: readability library source code

[3] textacy: quickstart tutorial