Next: Queries and Judgments
Up: A Case Study in
Previous: TREC Web Tracks
To set up an experimental environment that would allow us to
objectively study the effectiveness of TREC ad-hoc algorithms for a
realistic web search task, we need the following tools:
In the following we discuss our approach to selecting and building
each of these components.
- Queries: A set of real user queries.
- Judgments: User judgments for the ``goodness'' of a
page/site retrieved by a search engine.
- TREC Implementation: An implementation of a
state-of-the-art TREC ad-hoc algorithm.
- Evaluation Measure: An objective evaluation measure that
accurately measures search effectiveness for the search task at hand.
- Collection: A large collection of fresh web pages.