The following is the abstract of the paper presented at the Manchester IAEA conference in October 2003. You can also request the full paper.

Computerised Marking of Short-Answer Free-Text Responses.

Tom Mitchell1, Nicola Aldridge1, Peter Broomhead2

1. Intelligent Assessment Technologies Ltd. www.IntelligentAssessment.com

2. Dept of Systems Engineering, Brunel University.

Abstract

Open-ended items requiring a free-text response are highly valued in traditional paper-based assessment and learning, but have been absent from computer-based assessment due to limitations in computerised marking technology. Recent developments, however, have seen the introduction of natural language based assessment engines.

One such engine has been developed in the UK by Intelligent Assessment Technologies. The engine looks for specific content within free-text responses, the content being specified in the form of a number of mark scheme templates. Each template represents one form of a valid (or a specifically invalid) answer. The representation of the templates is such that they can be robustly mapped to multiple variations in the input text. The engine has been developed specifically to provide robust computerised marking of short-answer free-text items.

This paper describes the operation of the marking engine, and describes a model based on computerised marking and computer-assisted moderation. A case study for the technology is described, namely the computerisation of a medical progress test at Dundee University, where a test comprising 270 short-answer free-text items is now delivered, marked, and moderated using a computerised system.

Key words : Computer Assisted Assessment, Free-Text, Computerised Marking.