IAT's Free-Text Assessment Engine

IAT's free-text assessment engine employs natural language processing (NLP) tools and techniques to provide robust computerised marking of short-answer free-text answers. The software includes a number of modules specifically developed to increase the robustness of marking when faced with errors in spelling, grammar, semantics and punctuation.

What Kind of Questions Can it Mark ?
The engine is ideally suited to marking questions which require short free-text answers. Such questions will typically be for one or two marks, requiring a one or two sentence answer. Example questions include :

How does the temperature affect solubility ?
Why are some flowers highly scented ?
What movement is affected by rupture of the supraspinatus tendon ?
Describe one thing that roots do for a tree.

How does it work ?
The engine employs NLP techniques to perform an intelligent match between free-text answers and predefined computerised model answers. This is analogous to the process carried out by human markers when marking free-text answers for content. And like human markers, the engine attempts to look for the understanding expressed in a free-text answer, without unduly penalising the student for errors in spelling, grammar, or semantics.

The engine employs a mark scheme which specifies acceptable and unacceptable answers for each question. The engine represents model answers as syntactic-semantic templates. Each template specifies one particular form of acceptable or unacceptable answer. For example, the diagram below illustrates a simple template for the model answer The Earth rotates around the Sun.

The template shown can be expected to match a student response if the response contains one of the stated verbs (rotate, revolve, orbit, travel, move) with one of the stated nouns (earth, world) as its subject, and around / round the Sun in its preposition. Verbs in the student response are lemmatised (reduced to their base form, i.e. 'went' lemmatises to 'go') so that, for example, the following student responses will all be matched by the template shown above.

The world rotates round the sun.
The earth is orbiting around the sun.
The earth travels in space around the sun.

Development of the templates in the computerised mark scheme is an offline process, achieved using IAT's FreeText Author software. The inputs to the process are the marking guidelines for the question, and if available a sample of student responses. The output from the process is a 'computerised mark scheme'.

Once the mark scheme for a question has been developed, it can be used by the engine to mark free-text student answers. Incoming free-text answers are processed by a sentence analyser, and the output intelligently matched against each mark scheme template. The result of the matching process determines the mark awarded to the response.

The output of the assessment engine can be use summatively or formatively, as required by the application. For formative uses, specific feedback can be attached to each particular model answer, so a student can receive feedback tailored to the response they submitted.