| DRAFT Overview of the TREC 2004 Robust Retrieval Track (2008) | |||||||||||||||
Abstract | |||||||||||||||
| The robust retrieval track explores methods for improving the the consistency of retrieval technology by focusing on poorly performing topics. The task in the track is a traditional ad hoc retrieval task, but new evaluation measures emphasize a system’s least effective topics. The 2004 edition of the track used 250 topics and required systems to rank the topics by predicted difficulty. The ability to return at least passable results for any topic is an important feature of an operational retrieval system. While system effectiveness is generally reported as average effectiveness, an individual user does not see the average performance of the system, but only the effectiveness of the system on his or her requests. A user whose request is responded to by a set of documents that have no discernible connection with the information need is unlikely to be consoled by the fact that the system responds better to other people’s requests. The TREC robust retrieval track was started in TREC 2003 to investigate methods for improving the consistency of retrieval technology. The first year of the track had two main technical results: 1. The track provided ample evidence that optimizing average effectiveness using the standard Cranfield methodology and standard evaluation measures further improves the effectiveness of the already-effective topics, sometimes at the expense of the poor performers. | |||||||||||||||
Publication details | |||||||||||||||
| |||||||||||||||