Giving students close access to a program that evaluates essays many times would be the worst possible thing to do. People are generally very good at adapting to systems, and the algorithm would break down pretty quickly as people find out which factors it favors and which ones it penalises (while human graders are arguably less strict in their criteria and can hence recognize these attempts at writing to the letter).
If students en masse merely ran their shitty essays through spell and grammar check, the quality of even college-level essays would be improved by a huge margin.
For 99% of the high school students I've worked with, their essays would be better if they had to pass Microsoft's spelling and grammar checks before they were submitted. The exceptions were all exceptional enough that they knew that they no longer needed to listen to those automated filters.
This inevitably leads to an arms race, with the students and software getting better and better. At some point, sufficiently advanced gaming of essay scores is indistinguishable from students who naturally write well by following the rules which lead to good writing. As long as creativity and originality aren't sacrificed, is it really a problem?
You may be overestimating the sophistication of these algorithms.
At least for this training set, my algorithm rewarded the length of the essay most of all (something like 65% of the total prediction). The only other significant factors were misspellings and prevalence of certain parts of speech.
That model matched the accuracy of human graders and several commercial essay grading packages.
Students reverse-engineering comparable algorithms won't necessarily have to write well to score well.
currently, if a student writes nonsense, there's a fairly significant chance that they will be caught and penalised. a human can detect nonsense in three minutes.
in contrast, i suspect algorithmic approaches can be gamed more easily because they don't adapt in the same way. they're not solving the hard ai problem; they're grading essays (currently) written for a human reviewer.
for example, what happens if a child learns an existing text by heart and then substitutes appropriate nouns and verbs to suit the context? say they learn "We hold these truths to be self-evident, that all men are created equal" and then, for an essay on their favourite pet, they hand in "We hold these kittens to be furry, that all kittens are created hungry". That's good grammar; it's got suitable references to the subject; it's clearly nonsense.
No it's incorrect. Compare "these truths... that all men are created equal, (...)" with "these kittens... that all kittens are created hungry". The that in the second sentence is wrong.
In all probability, anyone setting out to actually defeat these algorithms could easily do it with a couple of hundred repetitions of the same sentence.
Or, if it's slightly cleverer than that, certainly you could defeat it by producing a single, perfect-length, stylistically fantastic essay... which would be regurgitated word-for-word regardless of the subject matter.
I think, mind you, that software does have a place in analyzing student essays. If I could scan in an essay and have it spit out a word count, highlight any spelling errors or potentially problematic turns of phrase, and [most importantly] analyze for plagiarism, that would be valuable.
It does not necessarily lead to an arms race. Once the students learn to write to the gradind software, the schools can claim the students are getting better, and point to an 'objective, unbiased' measuring stick. Everyone wins except for society, i.e. sll of us.
I'm assuming there will be competition among the companies who develop such software where, periodically, schools will re-evaluate their provider and select the best one. Company A will demonstrate their competitive advantage by showing how Company B's software incorrectly grades as "Excellent" an essay that looks like English but reads as gibberish.
Company B will correct their software, but meanwhile Company C has come along and introduced a sophisticated analyzer which is able to grade the presence and quality of the logical argument being made in the essay. Company C will demonstrate how Company A and B's software grades as "Excellent" an essay which is correct English and isn't gibberish but makes no logical sense.
After an indeterminate number of iterations with the software getting better and better at finding issues, the only way to game it will be to write a great essay.
You are forgetting the negative effect of the ridiculous ranking systems that determine teachers performance evaluations. Why would a school select software which will make thier grades go down? If your grades go down it decimates the schools ranking because these are based on year on year improvements. You would select the software which makes it easiest to teach to the test so the kids 'do well'
