To summarize, Grammarly is only going to be of use to more advanced English learners, due to the fact that (1) its engine aims at correcting native speakers' mistakes, not those of non-native speakers, and (2) all of its explanations are in English. So if your mistakes are somewhere in the ballpark of a native speaker's (and many advanced learners' mistakes are), and you can understand their explanations, then you're in good shape to make use of Grammarly. However, given the free corrections you can get on sites like Lang-8, it's probably not necessary for most language learners to invest in a premium account on Grammarly.
Before I get into the review, let me first explain the difference between their free and premium accounts, which has been a source of confusion in the past. At this time, all you can get for free is a report quantifying the kinds of corrections you need; you cannot see the specific text that needs to be corrected. While this might be useful for native speakers (who can go through the document and fairly easily find the issues flagged in the report), it's of much less use to non-native speakers (who aren't going to be able to find their mistakes as easily). For that reason, this review focuses only on the premium account, which shows specifically where changes are suggested and thus is of most use to language learners. The standard prices for Grammarly's premium subscriptions are $19.95 for a monthly subscription, $39.95 for a quarterly subscription ($13.32/month), or $94.95 for an annual subscription ($7.91/month).
To conduct the review, I looked at how Grammarly handled three documents:
- A three-page selection from an article that I had published in a legal journal, which I and the journal's editors edited heavily before publication. The idea here was to explore the outer bounds of what Grammarly is capable of.
- A page-and-a-half English-language summary of a dissertation that an Italian-speaking English learner posted on Lang-8 for corrections. The text could be fairly easily understood, even if there were numerous grammatical problems. Here I wanted to see how Grammarly would work for an advanced but clearly non-native speaker.
- A short entry from another Lang-8 user whose English had many more problems. It was fairly difficult to figure out exactly what the author was trying to say in each sentence, and is what I would say is typical of a learner who still has a long way to go. Here I wanted to see how Grammarly would work for a beginner.
Even though the issues flagged were all false positives, Grammarly offers up concise but clear explanations of each of the points of grammar for which it thinks there might be an issue. These explanations are clear and informative, and provide good information to explain whether you should act on a flagged issue or not. However, whether or not you should act is often something that would require an advanced English level to decide. This is both because you need first to understand the suggestions and then to be able to determine whether to make a change. While any non-native speaker can probably eventually figure out Grammarly's suggestions with the help of a dictionary, determining whether to make a change based on a given suggestion is the much harder step for non-native speakers.
Turning to the Italian speaker's English text, Grammarly was a huge help in reducing the errors, but predictably missed a few key errors that are common among non-native speakers but uncommon among native speakers.
When I first ran the text through Grammarly, it flagged 41 errors. Of those 41, only three were false positives, two of which were false positives found in text quoted directly from Shakespeare. I fixed all the other mistakes it pointed out on the first run-through and then ran it through the system again. This time, it flagged four new errors. On the next run-through, there was one new error and after that none. All of the suggested changes were clear and relatively easy to make, and making them clearly improved the quality of the document greatly.
The situation was more problematic for the short beginner's text. Grammarly flagged 15 suggestions for change, most of which consisted of simply reminding that a space is needed after terminal periods and question marks, but none of which were false hits. However, even after making those changes, the text was still difficult to understand and needed a lot more work to make it anywhere near a native's speech.
In neither non-native text was Grammarly enough to get the text to full-fledged native language (something they don't claim to do, in any case), although it was much more useful with the advanced learner's text. For both, once I let Grammarly do all it could do, I went through the document and made all of the rest of the necessary edits. This revealed certain kinds of mistakes commonly made by non-native speakers that Grammarly does not seem to cover well:
- The use of the articles "a" and "the". Although it did catch one mistake where "a" was used before a plural noun, it missed all the rest of the mistakes: lack of a necessary "a" or "the"; mixing up "a", "the", or the omission of both; etc. As the use of articles varies greatly from language to language, and many languages don't even have articles at all, these mistakes are very common among non-native speakers but uncommon among native speakers.
- The placement of adverbs. Grammarly doesn't seem to check for awkwardly placed adverbs (like "I went to also the store" as opposed to "I also went to the store"), and this is another kind of common mistake made by non-native speakers.
- Punctuation. Grammarly does cover a number of punctuation issues, but its seems to miss some of the errors made by non-native speakers. For instance, the author of the dissertation summary put a space after each opening quotation mark, and Grammarly did not catch that error.
Indeed, Max Lytvyn, one of the founders of Grammarly, anticipated these kinds of issues from a language learners' perspective when he wrote to me with the review account:
[L]et me give you some background on Grammarly usage we are seeing. The biggest user segment right now is students who are native English speakers. The second largest segment is professionals who may or may not be native speakers (this segment is likely to outgrow the first one soon). English language learners are not very active users of Grammarly. There are several factors contributing to lower popularity of Grammarly among language learners: 1) explanations are in English, and thus may be difficult to understand for some English learners; 2) Grammarly, being a computer algorithm, cannot be 100% accurate in its detection of mistakes; for native speakers such inaccuracies (false positives) are easy to see and do not take away much from the learning value of the product, but for language learners with lower levels of English proficiency it can be an issue. We do have examples of successful applications of Grammarly in a number of international educational institutions (e.g., ICU Japan, IQRA University in Pakistan, and dozens of private schools in Korea are licensing Grammarly for all of their students), but these institutions have rather high levels of English proficiency, and understand that Grammarly is an additional resource and not the ultimate authority. …Finally, let me quickly run through a catch-all list of my other comments that don't seem to fit in elsewhere:
In general, individual users and educational institutions use our products to help writers 1) be more productive during the proofreading stage; 2) gain better understanding of different points of grammar and avoid repeatedly making the same mistakes; 3) identify areas of weakness of a particular writer to prepare for a meeting with a tutor/advisor or to plan a study session. Also, some tutors and educational institutions use Grammarly as a tool to help grade student papers and provide more expanded feedback in a time-efficient way. This is a legitimate usage scenario as long as users understand limitations of the product and use it as an additional resource, without expecting it to deliver the final grade for a paper.
- When I copied and pasted the selection from a Word document into their checker, new paragraphs were ignored, so, during the check, I got an error saying that I needed a space where those paragraphs were ignored. (I excluded these errors from the error counts above.) This didn't happen when I pasted directly from Lang-8.
- Their error explanation at times partially obscured the text it was referring to, as shown in the image below. You can move it around manually to see what's under it, but it'd be better if you didn't have to bother.
- I found their synonym suggestions somewhat arbitrary. I wasn't sure why certain words were picked to be replaced. Were the words to be replaced repetitive? It's not clear. The suggested replacements also seemed arbitrary. For instance, it suggested replacing the "late" in the "late 1990s" with "past", which just wouldn't make any sense. These kinds of suggestions could be especially troubling for language learners, so I would probably recommend that all but the most advanced language learners using Grammarly ignore the synonym suggestions completely, especially as I've seen language learners get in trouble before through haphazard use of thesauruses.
- In one of the articles I ran through the system, it missed the misspelled word "continuosly". It also didn't have a thing to say about a number of quotes in foreign languages, so I'm guessing that the spellcheck only looks for common mistakes rather than truly doing a thorough spellcheck. That said, it is, after all, a grammar-checking system as opposed to a spell-checking system.
The English learners who could make the most use of Grammarly are those who produce a lot of text in English, such as those whose jobs require them to liaise frequently with English speakers. In such cases, Grammarly could be a better choice than the free sites because of Grammarly's almost immediate turn-around time and because it can handle larger texts and larger volumes of texts more easily, even though Grammarly's corrections aren't necessarily going to result in native speech.
If Grammarly came out with a product for Japanese, I would be very interested in it. I write a good deal in Japanese, and getting my own errors systematically flagged down for me would be very helpful in improving my writing, just as Grammarly is used by native-English speakers. So while I think Grammarly is of limited use to language learners because of sites like Lang-8, I still hope that we might see them apply their technology to other languages so that language learners other than English speakers might be able to take advantage of what Grammarly can offer.
Update: On May 18, Max wrote to me with the following comments about the review:
Thanks for a good review. I think it is an accurate assessment of the product. Grammarly is designed for a specific population and specific purposes, and those who are starting to learn English are not the best target. Grammarly will benefit those who use language for work or education at a higher level of proficiency, regardless of them being native speakers or not. Still, our Korean clients are very successful in using Grammarly to help their students prepare for English essay tests, so there are scenarios where Grammarly may help English learners as well.
- The first text you checked had no mistakes, which explains that 100% of suggestions were false positives. It is a statistical phenomenon that many people do not immediately understand. For example, if you test 10,000 completely healthy people for a rare disease, and the accuracy of the test is 99%, the test will flag 100 people as sick, and will be wrong in 100% of the cases, while still being 99% accurate. It is a simplification, but in general should explain why testing texts with no mistakes produces non-representative results. We have to explain this phenomenon rather often, as people frequently test Grammarly by copying professionally edited texts from the Web.
- We are working on expanding the support for mistakes typical to non-native speakers. Support for articles, for example, is one of the most active areas of improvement (there is a limit there – in many cases it is impossible to determine if definite or indefinite article is required without actually understanding the meaning of the text). We also will look into adverb placement mistakes.
- Versions of Grammarly with cards in other languages are possible. We are considering a version with cards in Korean.
- Versions of Grammarly for languages other than English are not yet planned, but are a possibility.