| Implementing a Question Answering Evaluation (2000) | |||||||||||||||
Abstract | |||||||||||||||
| The most recent TREC workshop contained a track for evaluating domain-independent question answering (QA) systems. In addition to fostering research on the QA task, the track was used to investigate whether the evaluation methodology used for document retrieval is appropriate for a different natural language processing task. This paper describes the implementation of the track, focusing on how the test questions were selected and the responses judged. By using multiple assessors to independently judge the responses, we verified that assessors do have legitimate differences of opinion as to the correctness of a response, even for the fact-based, short-answer questions used in the track. The Text REtrieval Conference (TREC) is a workshop series organized by the U.S. National Institute of Standards and Technology (NIST) designed to advance the state-ofthe -art in text retrieval (Voorhees, 2000). The most recent TREC, held in November 1999, included a "track" on question answering where t... | |||||||||||||||
Publication details | |||||||||||||||
| |||||||||||||||