Maple 17 includes a new package for linguistic analysis and grading of essays. The ability for a computer to successfully grade essays is inherently mathematical. Given a set of essays that have already been graded by hand, the computer looks for patterns in the essays and tries to weight them according to the given scores. Things like key words, sentence structure, length, and variation of words tend to have a significant correlation to good or bad scores. Maple's scoring model can pick from up to 20 algorithms, each measuring dozens of properties to formulate a model that can be used to predict scores for new essays.

The EssayTools package contains functions for:

• Grading essays
• Plagiarism detection and similarity comparisons
• Analysis of English word forms and sentence reduction
• Checking spelling

The grading commands are best used in an advisory capacity. They are great for giving insight into student responses for homework and practice. They are also effective as a double-check in high-stakes testing where many markers are utilized. For example, any essay where the human score disagrees with the computer score by more than one or two points could be flagged and re-graded by an independent human.

To get good predictions it is best to seed your model with hundreds of scored essays - the more the better. For the purposes of this overview, we will use examples that have insufficient data in order to illustrate the form of the commands as well as some of the pitfalls and limitations of technology like this.

Consider a wide open short-answer question like "Why is the sky blue?". Gather several responses and put them in an array. Here, we call the array Answers. Provide a second array with a grade for each of the answers. In this example, we call the second array Scores. In practice, these responses and grades could be read in from a .csv file using one of the ImportVector or ExcelTools:-Import commands.

>	with(EssayTools);

>	Question := "Why is the sky blue?":

>	Answers := Array(): Scores := Array():

>

Answers(1) := "The blue color of the sky is due to Rayleigh scattering. As light moves through the atmosphere, most of the longer wavelengths pass straight through. Little of the red, orange and yellow light is affected by the air. However, much of the shorter wavelength light is absorbed by the gas molecules. The absorbed blue light is then radiated in different directions. It gets scattered all around the sky. Whichever direction you look, some of this scattered blue light reaches you. Since you see the blue light from everywhere overhead, the sky looks blue. ":

>	Scores(1) := 10: #www.sciencemadesimple.com/sky_blue.html

>

Answers(2) := "Sunlight reaches Earth's atmosphere and is scattered in all directions by all the gases and particles in the air. Blue light is scattered in all directions by the tiny molecules of air in Earth's atmosphere. Blue is scattered more than other colors because it travels as shorter, smaller waves. This is why we see a blue sky most of the time.":

>	Scores(2) := 10: #spaceplace.nasa.gov/blue-sky

>	Answers(3) := "A clear cloudless day-time sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light.":

>	Scores(3) := 8: #math.ucr.edu/home/baez/physics/General/BlueSky/blue_sky.html

>

Answers(4) := "The blue color of the sky is due to Rayleigh scattering. As light moves through the atmosphere, most of the longer wavelengths pass straight through. Little of the red, orange and yellow light is affected by the air.  However, much of the shorter wavelength light is absorbed by the gas molecules. The absorbed blue light is then radiated in different directions. It gets scattered all around the sky. Whichever direction you look, some of this scattered blue light reaches you. Since you see the blue light from everywhere overhead, the sky looks blue.":

>	Scores(4) := 10: #www.sciencemadesimple.com/sky_blue.html

>

Answers(5) := "Sunlight is a mix of all the colours of the rainbow. Light is a spectrum of light of different wavelengths, from the shorter blue to the longer red  wavelengths.  When sunlight enters the atmosphere, the molecules in the air are the right size to scatter the blue wavelengths of the sunlight, which sort of paints the sky blue.":

>	Scores(5) := 7: #answers.yahoo.com/question/index?qid=20071031201452AAwbqie

>	Answers(6) := "Blue light is scattered by the air molecules in the atmosphere (referred to as Rayleigh scattering). The blue wavelength is scattered more, because the scatteing effect increases with the inverse of the fourth power of the incident wavelength. ":

>	Scores(6) := 7: #answers.yahoo.com/question/index?qid=20071031201452AAwbqie

>	Answers(7) := "The sky is blue because of a reflection of the sea.":

>	Scores(7) := 0:

>	Answers(8) := "The sky is blue because the sun is yellow and the grass is green. Yellow and green make blue.":

>	Scores(8) := 0:

>

Answers(9) := "These days, the atmosphere is mostly nitrogen and oxygen. Sunlight is made up of all the colors of the rainbow (as well as many wavelengths we can't see); as it jostles through air molecules, blue light is most efficiently reflected, so our eyes end up experiencing a beautiful azure shade":

>	Scores(9) := 10: #news.discovery.com/earth/no-seriously-why-is-the-sky-blue.html

Use the BuildScoreModel command from the EssayTools package to generate a model.  

>	model := BuildScoreModel(Answers, Scores, questionText = Question);

This model could be saved, for example, by calling LibraryTools:-Save(model,"blue_sky.mla");. Later, when you have more responses, you can just point libname := libname, "blue_sky.mla"; and the variable, model, will implicitly be available for use.

Now, let's examine a previously unseen response:

>	NewResponse := "Blue and violet wavelengths, on the other hand, are absorbed by the gas molecules and scattered across the sky. Your eye sees these reflected wavelengths as blue.": #wonderopolis.org/wonder/why-is-the-sky-blue/

>	Score(NewResponse,model);

Other EssayTools commands can be used to get an idea of what goes on behind the scenes. First, let's look at the words that occur with high frequency in the model answers.

>	WordUse(Answers, showcount, mincount = 3);

Let's use some of these high-frequency words to form nonsense answers.

>	Score("The is of blue light sky in scattered air.", model);

>	Score("Blue light sky scattered air molecules.", model);

This model gives a disproportionate weight to the use of the word "molecules", which always appears in good answers, but does not appear in low-scoring answers. This emphasizes the need to have a large sample of data to build the model with.

Reduction techniques can be used to simplify the essay by dropping unimportant words, coalescing words with similar meaning, and splitting the response into smaller idea-phrases.

>	ModifiedResponse := Reduce(NewResponse);

An important tool in this process is the use of word lemmas.

>	Lemma("is");

>	Lemma("are");

>	Lemma("flew");

>	Lemma("flies");

Knowing where to split a sentence requires that each sub-part usually has at least a noun and verb, and be split at a conjunction.

>	TwoIdeas := "the sky is blue because it scatters sunlight.";

>	map(PartOfSpeech, StringTools:-Words(TwoIdeas));

Part-of-speech information is not available for all words; as you can see, "scatters" is not known and returns FAIL. The Reduce command will attempt to use the Lemma command to get the root word.

>	Lemma("scatters");

>	PartOfSpeech(%);

>	Reduce(TwoIdeas);

Another technique for scoring essays is to find the most similar essay or essays in the model set. There are many ways to compute similarity scores.

>	SimilarityScore(NewResponse, Answers);

s

According to the default similarity metric, BinaryJaccardCoefficient, the second essay is most similar, receiving a score of .22.

>	SimilarityScore(NewResponse, Answers, methods = [CosineCoefficient]);

>	SimilarityScore(NewResponse, Answers, methods = [DiceCoefficient]);

The CosineCoefficient  metric instead picks answers one and four, while the DiceCoefficient metric picks essay eight. Each of these metrics, and their binary counterparts, measures similarity in a different way.

When a similarity score exceeds a certain minimum, it indicates that there is a good chance those essays are copies of each other in part or in whole.

>	DetectPlagiarism(Answers);

The DetectPlagiarism command compares all given essays with each other and flags those that exceed a minimum similarity score. In this case we see that essays 1 and 4 are certainly copies of each other, as indicated by a similarity score of 1. Essays 7 and 8 are flagged as possible copies since they are both short, and start with exactly the same 5 words.