Which readability testing tool should we use?

Summary

Three text samples were tested in a range of tools using the Flesch-Kincaid grade level algorithm. The results varied. Against a manual calculation of the grade level, the tool built into MS Word gave the most reliable results.

The problem

Anecdotal evidence suggests readability testing tools do not give consistent results. However they are useful because they are faster than manual calculations. Is there a reliable tool we can use?

I took three text samples from the web and checked each with some of the freely available readability testing tools. All were based on the Flesch-Kincaid grade level algorithm. I used the same algorithm to test the samples manually.

Findings

My manual check gave grade scores of 11.97, 12.9 and 10.72 across the three samples. The tool that scored closest to these was MS Word. It had an average difference of only 0.18 grade levels.

Results from the other tools varied a little more, but most averaged less than 1 grade level. The tool from Added Bytes had an average different of over 2 grade levels.

Summary of results
Tool Sample 1 Difference
to manual test 		Sample 2 Difference to
manual test 		Sample 3 Difference to
manual test 		Average
difference
Edit Central 11.7 -0.27 12.3 -0.60 10.9 +0.18 0.35
Juicy Studio 11.44 -0.53 12.00 -0.90 11.11 +0.39 0.61
Added Bytes 14 +2.03 15 +2.01 13 +2.28 2.11
Online Utility 12.08 +0.11 13.65 +0.75 12.46 +1.74 0.87
Readability.info 11.1 -0.87 11.7 -1.20 10.4 -0.32 0.80
MS Word 2003 11.9 -0.07 12.8 -0.10 11.0 +0.28 0.18

Flesch-Kincaid algorithm

I chose this readability test because:

  • It shows results in grade levels. These are easier to understand than other types of test scores
  • Many testing tools use the Flesch-Kincaid algorithm, including MS Word which many web writers will have on their computers.
  • Usability and accessibility guidelines recommend writing to a given grade level
    • Grade 8 is recommended by Nielsen Norman Group research
    • Lower secondary level is recommended by the new Web Content Accessibility Guidelines 2.0

The algorithm is:

Grade level = (0.39 x ASL) + (11.8 x ASW) - 15.59, where:

  • ASL = average sentence length in words
  • ASW = average number of syllables per word

Readability testing tools used

Text samples and detailed results

I chose samples of varying lengths from corporate, government and non-profit web sites. I avoided text that contained many numbers, dates, acronyms or abbreviations because I was not sure how the automated tools would count the syllables in these words (e.g. will DVD be counted as 1 syllable or 3?).

Sample 1

We offer a range of accredited training and education programs to benefit people who are blind or have low vision and those who support and/or teach people who are blind or have low vision. You are able to search by the type of training we provide and you can further refine your search by location.

Length: under 100 words
Source: Vision Australia (non-profit)

Results for sample 1
Tool Flesch-Kincaid Grade level Syllables Words Sentences
Edit Central 11.7 77 (1.4 per word) 55 2 (27.5 words average)
Juicy Studio 11.44 1.38 per word 55 2 (27.50 words average)
Added Bytes 14 1.67 per word - 26.5 words average
Online Utility 12.08 1.44 per word 55 2 (27.50 words average)
Readability.info 11.1 1.34 per word 56 2 (28 words average)
MS Word 2003 11.9 - 56 2 (28 words average)
Manual check 11.97 79 (1.41 per word) 56 2 (28 words average)

Sample 2

Water restrictions are managed by Victoria’s urban water corporations and are applied across Victoria based on the Uniform Drought Water Restriction Guidelines.

This section provides a summary of the key restrictions that may be in place for urban water users under the various stages of restriction, Stage 1 (mild) to Stage 4 (severe).

At present a variation of Stage 3 water restrictions, known as Stage 3a, is in place in areas supplied from the Melbourne water supply system.

While all restriction policies are based on the uniform guidelines, there may be minor variations in the details to account for local conditions. Please check with your local water corporation for information on water restrictions that may be in place in your area.

Restrictions are only applicable to customers on a reticulated supply. These restrictions do not apply to customers using rainwater or bore water.

Water corporations may grant exemptions from restrictions for those with special needs, and have principles in place to ensure the consistent treatment of exemption applications. Exemptions from water restrictions may be applied for through your local water corporation.

Length: 100-200 words
Source: Our Water Our Future (government)

Results for sample 2
Tool Flesch-Kincaid Grade level Syllables Words Sentences
Edit Central 12.3 432 (1.49 per word) 289 11 (26.27 words average)
Juicy Studio 12.00 1.75 per word 181 9 (20.11 words average)
Added Bytes 15 1.9 per word - 20 words average
Online Utility 13.64 1.81 per word 181 9 (20.11 words average)
Readability.info 11.7 1.65 per word 182 9 (20.2 words average)
MS Word 2003 12.8 - 181 9 (20.1 words average)
Manual check 12.9 317 (1.75 per word) 181 9 (20.11 words average)

Sample 3

When you apply for a new policy your duty of disclosure applies, but you do not need to disclose something to us unless we specifically ask you about it. However, you must be honest in answering any questions we ask you. You have a legal duty to tell us anything you know, and which a reasonable person in your circumstances would include in answering the questions. We will use the answers in deciding whether to insure you and anyone else to be insured under the policy, and on what terms.

It is important that you understand you are disclosing to us and answering our questions for yourself and anyone else you want to be covered by the policy.

If you do not answer our questions honestly or do not properly disclose to us, we may reduce or refuse to pay a claim or may cancel the policy. If you act fraudulently in answering our questions or not disclosing to us, we may refuse to pay a claim or treat the policy as never having existed.

If you think we have let you down in any way, or our service is not what you expect (even if through one of our agents or representatives), please tell us so we can help.

You can tell us by phone, in writing or in person. If you tell us in writing it will help if you send us the full details of your complaint together with any supporting documents and an explanation of what you want us to do. If you would like to come in to talk to us face to face, please call and we will arrange an appointment for a meeting.

Length: 200-300 words
Source: Secure Motor Plus Insurance (corporate)

Results for sample 3
Tool Flesch-Kincaid Grade level Syllables Words Sentences
Edit Central 10.7 388 (1.39 per word) 280 11 (25.45 words average)
Juicy Studio 11.11 1.42 per word 280 11 (25.45 words average)
Added Bytes 13 1.56 per word - 25 words average
Online Utility 12.46 1.54 per word 280 11 (25.45 words average)
Readability.info 10.4 1.36 per word 280 11 (25.5 words average)
MS Word 2003 11.0 - 280 11 (25.4 words average)
Manual check 10.72 397 (1.42 per word) 280 11 (25.45 words average)

Related (external) links

Published: 27 July 2008