News
- New
- Training and development data released on Hugging Face: 24 domains and 2,161,196 documents, divided 70/30 into training and development sets with relevance judgments for both. The starter kit — BM25 baseline, official scorer and submission format checker — was released at the same time.
- RETECO accepted as SemEval-2027 Task 1. See the SemEval-2027 task list.
- Sample data published: a small, human-inspected package illustrating every record and submission format. Inspect the samples.
Overview
Retrieval systems are commonly evaluated by topical relevance. RETECO instead addresses cases in which a system must reason about when information is valid, how events or facts change over time, and what has already been established in a conversation.
The task comprises two tracks and five sub-tracks, built on the TEMPO and RECOR benchmarks. Participants may enter a single sub-track or work across retrieval and grounded generation. Full definitions, including the required input and output for each sub-track, are given on the task page.
Data
The training and development data are available from Hugging Face in a single download. They comprise 24 domains and 2,161,196 documents, with each domain carrying its own complete retrieval corpus, so no further resources are required.
Each domain is divided 70/30 into training and development sets under a fixed, published seed, with relevance judgments released for both. The retrieval corpus itself is not divided: systems search the full corpus of a domain in either split.
| Item | Train | Dev |
|---|---|---|
| Track 1 queries | 1,211 | 519 |
| Track 1 decomposed steps | 2,762 | 1,214 |
| Track 2 conversations | 496 | 211 |
| Track 2 target turns | 2,113 | 858 |
The evaluation set is separate, newly annotated, and released without judgments during the evaluation window. Corpus statistics, record formats and licensing are documented on the data page; worked examples of every record type are given on the samples page.
Evaluation
nDCG@10, computed with pytrec_eval and macro-averaged, is
the official retrieval metric for Sub-tracks 1a, 1b, 2a and 2c. Generation quality for
Sub-tracks 2b and 2c is reported alongside the retrieval results rather than combined with
them. Temporal coverage, temporal precision, conversation-turn depth and per-domain
results are reported as diagnostics. The full protocol is set out in the
evaluation plan.
A reference BM25 baseline, the official scorer and a submission format checker are provided in the starter kit. Reference results for both splits are reported on the data page.
Participation
Announcements are distributed through the task mailing list. Registration, the evaluation platform and submission limits will be announced there and on this site once confirmed. The participation guide describes how to obtain the data, develop against the training split, score runs locally with the official metric, and submit.
Papers and proposal
RETECO builds on the TEMPO and RECOR benchmarks. The accepted task proposal, paper links and BibTeX records are given on the papers and citations page.