01 · SAMPLEInspect
Small real examples demonstrate queries, documents, conversations, qrels, and output structures.
RETECO · SemEval-2027 Task 1
Data, corpora, and baselines
RETECO builds on two operational benchmarks, documents every domain collection, and preserves a separate held-out SemEval test.
Release model
Release model
02 · TRAIN / DEVDevelop
Released 30 August 2026; the current version, v1.1 (25 September 2026), removes duplicate documents from the Track 1 corpora. Every domain corpus, task files for all five sub-tracks, and qrels for both splits, in one download. Get it on Hugging Face ↗
03 · TESTEvaluate
Queries are released for submission while gold judgments remain private.
LanguageAll official RETECO task data are in English.
Sample data
Curated sample package
The repository includes a compact, human-inspected package copied from the official public releases: five TEMPO examples and four RECOR conversations, every referenced positive passage, TREC-style qrels, and a pinned provenance manifest.
RETECO curated sample data
9 task records · 41 supporting passages · 58 positive judgments
Official release
Training and development data
One repository contains everything needed to build a system for any sub-track: the full retrieval corpus of each of the 24 domains, the task files, and TREC qrels for both splits. Nothing has to be assembled from elsewhere.
24domains
1.67Mcorpus documents
70 / 30train / dev split
3.2 GBsingle download
What each split contains
| Track | Item | train | dev |
|---|---|---|---|
| 1 · TEMPO | Queries | 1,211 | 519 |
| 1 · TEMPO | Decomposed steps | 2,762 | 1,214 |
| 2 · RECOR | Conversations | 496 | 211 |
| 2 · RECOR | Target turns | 2,113 | 858 |
How the split is drawnThe corpus is never split — retrieval runs against the full domain corpus in both splits. Track 1 splits on the query, with a query’s decomposed steps nested inside its own record. Track 2 splits on the whole conversation, so no dialogue history crosses the boundary. The seed and the exact per-domain ID lists ship in
split_manifest.json.Gold labels are public for both splitsThese splits come from already-public benchmarks, so withholding dev labels would protect nothing. Develop on train, use dev as your held-out check. The SemEval test set is separate, unseen, and never released.
Versions
Release versions
The download command always fetches the latest version. Earlier versions stay available under their tag: add --revision v1.0 to hf download.
| Version | Date | Change |
|---|---|---|
| v1.1 | 25 Sep 2026 | Track 1 corpora deduplicated. Documents whose text is identical (ignoring whitespace) are kept once, reducing Track 1 from 1,654,055 to 1,167,159 documents. The gold document is always the copy that is kept, so no gold label is lost. Queries, splits, topic IDs and all of Track 2 are unchanged. Each Track 1 domain adds duplicate_map.json, mapping every removed ID to the kept ID. |
| v1.0 | 30 Aug 2026 | Initial training and development release. |
Upgrading from v1.0Re-download the data and rebuild your Track 1 indexes. Run files made on v1.0 can still be scored: replace each document ID through
duplicate_map.json and drop repeated IDs within a ranking. Track 1 scores are slightly higher on v1.1 because repeated copies no longer fill top-10 slots, so compare systems on the same version.Track 01 source
TEMPO corpus
TEMPO contains temporal reasoning-intensive queries requiring evidence about periods, trends, events, or change. Its 13 independent domain corpora contain 1,654,055 documents in total as published; RETECO v1.1 keeps one copy of each duplicated text, leaving 1,167,159.
1,730complex queries
3,976decomposed steps
1.17Mdocuments in v1.1
13domain collections
Per-domain corpus statistics
| Group | Domain | Queries | Documents (TEMPO) | RETECO v1.1 | Avg. gold/query | Avg. steps |
|---|---|---|---|---|---|---|
| Blockchain | Bitcoin | 100 | 153,291 | 76,706 | 3.3 | 2.93 |
| Blockchain | Cardano | 51 | 87,201 | 47,509 | 2.5 | 2.84 |
| Blockchain | IOTA | 10 | 10,372 | 9,680 | 3.8 | 3.20 |
| Blockchain | Monero | 65 | 85,093 | 53,955 | 2.6 | 2.72 |
| Social Sciences | Economics | 83 | 93,756 | 74,692 | 3.6 | 3.08 |
| Social Sciences | Law | 35 | 43,288 | 36,110 | 3.0 | 3.23 |
| Social Sciences | Politics | 150 | 183,394 | 135,207 | 2.7 | 3.35 |
| Social Sciences | History | 801 | 356,493 | 200,252 | 4.5 | 3.42 |
| Applied | Quantitative Finance | 34 | 28,785 | 25,040 | 2.4 | 2.68 |
| Applied | Travel | 100 | 177,677 | 158,771 | 2.6 | 3.11 |
| Applied | Workplace | 36 | 64,659 | 43,584 | 2.8 | 2.42 |
| Applied | Genealogy | 115 | 156,228 | 136,101 | 2.8 | 3.78 |
| STEM | History of Science & Mathematics | 150 | 213,818 | 169,552 | 2.5 | 3.25 |
| — | Total | 1,730 | 1,654,055 | 1,167,159 | — | — |
Queries and TEMPO document counts follow the official TEMPO release and paper; the RETECO v1.1 column is the deduplicated corpus you download. Each domain is a separate dataset split and retrieval corpus.
Track 02 source
RECOR corpus
RECOR combines multi-turn context with reasoning-dependent passage relevance. Six domains originate from BRIGHT collections and five from StackExchange; together they contain 507,141 documents.
707conversations
2,971target turns
507Kdocuments
11domain collections
Per-domain corpus statistics
| Source | Domain | Conversations | Turns | Corpus documents | Avg. docs/turn |
|---|---|---|---|---|---|
| BRIGHT | Biology | 85 | 362 | 57,359 | 1.56 |
| BRIGHT | Earth Science | 98 | 454 | 121,249 | 1.58 |
| BRIGHT | Economics | 74 | 288 | 50,220 | 2.28 |
| BRIGHT | Psychology | 84 | 333 | 52,835 | 2.16 |
| BRIGHT | Robotics | 68 | 259 | 61,961 | 1.76 |
| BRIGHT | Sustainable Living | 78 | 319 | 60,792 | 1.88 |
| StackExchange | Drones | 37 | 142 | 16,381 | 2.36 |
| StackExchange | Hardware | 46 | 188 | 26,308 | 2.10 |
| StackExchange | Law | 50 | 230 | 20,027 | 2.55 |
| StackExchange | Medical Sciences | 44 | 183 | 23,297 | 2.23 |
| StackExchange | Politics | 43 | 213 | 16,712 | 2.49 |
| — | Total | 707 | 2,971 | 507,141 | 2.01 |
Corpus-document counts are verified from the official Hugging Face dataset metadata; conversation, turn, and relevance statistics follow the published RECOR paper.
Reference baseline
Reference baseline
nDCG@10 measured on the RETECO training and development splits, macro-averaged over domains. Retrieval and scoring use the same implementation as the source benchmarks: the Lucene analyzer with gensim LuceneBM25Model (k1 = 0.9, b = 0.4), scored with pytrec_eval. Both runs are reproducible from the starter kit. Track 1 numbers are measured on data v1.1; on v1.0 they were 0.0879 / 0.0967 (1a) and 0.0852 / 0.1063 (1b).
| Sub-track | Query given to the retriever | train | dev |
|---|---|---|---|
| 1a · Temporal retrieval | Whole query | 0.1075 | 0.1147 |
| 1b · Step-wise retrieval | Query + step instruction | 0.1024 | 0.1177 |
| 2a · Conversational retrieval | Current turn only | 0.1837 | 0.1827 |
| 2a · Conversational retrieval | Turn + conversation history | 0.4539 | 0.4379 |
What the numbers sayLexical matching alone is weak on Track 1: temporal grounding is not a keyword problem, and the headroom is large. On Track 2 the query representation dominates — appending the conversation history lifts the same retriever from 0.18 to 0.44, which is the size of the signal a conversational system has to exploit.
Not a SemEval leaderboardThese are public train/dev results, not hidden-test scores. Numbers reported in the task proposal were measured on the complete public benchmarks and on different query representations, so they are not directly comparable with this table. The official leaderboard begins during the SemEval evaluation phase.
Data formats
Core formats
The release uses JSON/JSONL records and TREC-style qrels. Every file below exists once per domain and once per split, named <file>_train / <file>_dev, except the corpus, which is shared by both splits.
| File | Fields | Used by |
|---|---|---|
documents.jsonl | id / doc_id, content | Retrieval corpus (both splits) |
examples_*.jsonl | id, query, gold_ids, gold_answers | Sub-track 1a |
steps_*.jsonl | id, query, steps[ step_id, step, step_instruction, gold_ids ] | Sub-track 1b |
guidance_*.jsonl | id, query_guidance, gold_passage_annotations | Temporal diagnostics |
benchmark_*.json | id, task, turns[ turn_id, query, answer, conversation_history, gold_doc_ids ], metadata | Sub-tracks 2a, 2b, 2c |
qrels_*.txt | topic id, 0, document id, relevance | Local scoring |
split_manifest.json | seed, per-domain counts, ID lists, provenance | Reproducing the split |
duplicate_map.json | removed ID → kept ID (Track 1, v1.1) | Converting v1.0 run files |
Topic identifiers
Use these exact forms as the first column of a submitted run.
| Sub-track | Topic id | Example |
|---|---|---|
| 1a | id | 124973_5 |
| 1b | step_id = <query_id>_step<n> | 124973_5_step1 |
| 2a, 2c | <conversation_id>_turn_<turn_id> | ex_3025_turn_1 |
Licensing
Licensing and data integrity
The release carries two licenses, because the text and the annotations have different origins.
- Corpus and question/answer text — CC BY-SA 4.0. All passage text originates from Stack Exchange, directly for Track 1 and the five StackExchange domains of Track 2, and via BRIGHT for the remaining six. Stack Exchange contributions are licensed CC BY-SA, which is share-alike: redistribution must preserve the same license and the attribution.
- RETECO annotations — CC BY 4.0. The train/dev split assignments, the TREC qrels, and
split_manifest.jsonare contributed by the organizers.
Attribution for the underlying content remains with the original Stack Exchange authors. Zenodo archival under the same terms will follow the evaluation phase. Code and existing benchmark repositories may carry their own licenses; participants must follow the license displayed with each resource.
- Do not redistribute hidden evaluation data or inferred gold labels.
- Preserve source attribution and dataset citations.
- Report suspected data issues privately during evaluation.