Intelligent Assessment Technologies’ proprietary text marking engine employs state-of-the-art Natural Language Processing (NLP) technology to provide robust computerised marking of short-answer free-text questions. The technology looks for specific content in the answer, without penalising students unduly for poor spelling, grammar or syntax – similar to the way in which human-markers work.
Computerised marking of short-answer free-text questions is a specialist area, providing a more realistic test of knowledge than multiple choice questions. However it takes significantly more time to create and moderate an automatically marked short answer question that a multiple choice question. They are therefore most suitable for assessments where there are large numbers of candidates and questions in the question bank can be re-used in subsequent years. You can read more about the technology below.
IAT have developed FreeText Author, an authoring tool which simplifies configuring the marking rules for the company’s free-text marking engine. FreeText Author is a sophisticated desktop application which enables you to add machine marked short-answer questions to your ExamOnline tests, examinations, and assessments. To see the main constituent parts of the user interface, click on the image on the above.
With FreeText Author, the mark scheme panel has been designed to resemble a traditional paper-based mark scheme layout, with lists of acceptable and unacceptable top-level mark scheme answers. You can add or delete mark scheme answers using the mark scheme answer toolbar.
In the model answer list you can enter a number of model answers for each top-level mark scheme answer. These model answers represent specific acceptable phrasings or wordings for the corresponding mark scheme answer, and are the basis from which the templates are generated. Template generation is automated – you just have to specify which are the keywords for each model answer.
The synonym editor allows you to add synonyms for the model answer keywords. You can enter your own synonyms, and you can select from lists of synonyms suggested by the authoring tool. The authoring tool provides three list of suggestions:
- Suggestions from the thesaurus
- Suggestions which were automatically derived from analysing other model answers
- Suggestions obtained by analysing the list of responses, looking for words which have been used in the same way as the current keyword.
At any time you can computer mark sample responses using the mark scheme you have created. The response list will be updated in real time as the responses are marked. Responses which have been matched will be shown in green with a tick icon, responses which have not been matched will be shown in red with a cross icon, as shown here.
FreeText Author also includes a marking wizard to help you refine the mark scheme, providing information on why responses are not being matched.
When you have finished refining your mark scheme, you can export it to ExamOnline, ready for use.