Participate — RETECO

Overview

Steps to participate

01

Get the data

One download gives you every domain corpus, the task files, and gold judgments for both splits.

02

Pick your sub-tracks

Enter one, several, or all five. There is no penalty for entering only one.

03

Build and score locally

Develop against train, keep dev as your honest check, and score with the same metric the leaderboard uses.

04

Submit

Upload predictions for the hidden test set during the evaluation window, 10–31 January 2027.

Download the data ↗Starter kit ↗

Registration is not open yetThe registration form, participant mailing list, and competition platform will be linked here as soon as they are confirmed. Nothing stops you building now — steps 01 to 03 need none of them.

Step 01

Obtaining the data

Everything is in one Hugging Face repository. There is nothing to assemble from other sources: each of the 24 domains carries its own full retrieval corpus alongside its task files.

Download · about 4.5 GBpip install huggingface_hub
hf download DataScience-UIBK/RETECO-SemEval2027 --repo-type dataset --local-dir reteco_data

Which files does my sub-track need?

Replace * with train or dev. The corpus has no split suffix — it is shared by both.

Sub-track	You read	You produce	You score against
1a Temporal retrieval	`examples_*.jsonl` `documents.jsonl`	Ranked documents per query	`qrels_*.txt`
1b Step-wise retrieval	`steps_*.jsonl` `documents.jsonl`	Ranked documents per step	`qrels_steps_*.txt`
2a Conversational retrieval	`benchmark_*.json` `documents.jsonl`	Ranked passages per turn	`qrels_*.txt`
2b Gold-passage generation	`benchmark_*.json`	An answer per turn	Five judge dimensions
2c Full conversational RAG	`benchmark_*.json` `documents.jsonl`	Ranked passages and an answer per turn	`qrels_*.txt` + judge

How to use train and dev

Both splits ship with gold judgments. Fit, tune, and select on train; touch dev only to measure. Nothing is enforced technically — but a system tuned on dev gives you a number that will not survive the hidden test.

Split	Track 1	Track 2	Use it for
`train`	1,211 queries · 2,762 steps	496 conversations · 2,113 turns	Development, tuning, model selection
`dev`	519 queries · 1,214 steps	211 conversations · 858 turns	Held-out check before you submit

The corpus is never splitRetrieval always runs against the complete corpus of the domain, in both splits. A query’s decomposed steps stay inside its own record, and every turn of a conversation stays in the same split, so no dialogue history leaks across the boundary.

Full record formats on the dataset card ↗Field reference →

Step 02

Building a system

The starter kit contains a working BM25 baseline, the official scorer, and a format checker. Use it as a reference implementation or as the skeleton for your own system.

Set upgit clone https://github.com/DataScienceUIBK/RETECO.git
cd RETECO/starter_kit
pip install -r requirements.txt

Run the baseline — develop on train, check on dev# develop here
python official_baseline.py --data ../../reteco_data --out ../../baselines --splits train

# then measure here, once
python official_baseline.py --data ../../reteco_data --out ../../baselines --splits dev

# a single domain, for a fast smoke test
python official_baseline.py --track1 iota --track2 drones --splits train dev

Each run writes a TREC run file per domain and sub-track, plus a summary.json with per-domain and macro-averaged scores. Results are cached per domain, so an interrupted run resumes.

Check your pipeline against the baseline

Reference BM25 scores for every retrieval sub-track, on both splits, are published on the data page. If your system reproduces them, your indexing, topic identifiers and scoring are wired up correctly — which is worth confirming before you tune anything. Per-domain numbers are in BASELINE_RESULTS.md.

Where the headroom isLexical matching is weak on Track 1 — temporal grounding is not a keyword problem. On Track 2 the query representation dominates: simply appending the conversation history more than doubles the same retriever. How you represent time and dialogue context is the task.

Step 03

Local evaluation

The official retrieval metric is nDCG@10, computed with pytrec_eval (ndcg_cut_10). You can reproduce the exact leaderboard number on your own machine before you ever submit.

1 · Write a run file

Standard six-column TREC format, tab or space separated, one line per retrieved document.

run.trec124973_5        Q0  bitcoin/45eff6bd_1297.txt  1  18.4213  my_system
124973_5        Q0  bitcoin/e3e39760_1295.txt  2  16.9007  my_system
ex_3025_turn_1  Q0  drones_ex_3025_doc_0       1  11.2284  my_system

The first column is the topic identifier, and it must match the sub-track exactly:

Sub-track	Topic id	Example
1a	`id` from `examples_*.jsonl`	`124973_5`
1b	`step_id` from the nested `steps` list	`124973_5_step1`
2a, 2c	`<conversation_id>_turn_<turn_id>`	`ex_3025_turn_1`

2 · Check the format

Catch problems before they cost a submission# structure only
python format_checker.py run.trec

# also validate topic ids and document ids against the release
python format_checker.py run.trec \
    --qrels reteco_data/track1_tempo/iota/qrels_dev.txt \
    --corpus reteco_data/track1_tempo/iota/documents.jsonl

It checks that every line has six columns, that ranks are unique positive integers within a topic, that scores do not increase as rank grows, and — with --qrels and --corpus — that your topic and document identifiers actually exist. It exits non-zero on failure, so it drops straight into a CI step.

3 · Score it

Official metric, on your own machinepython -c "
import pytrec_eval
qrels, run = {}, {}
for ln in open('reteco_data/track1_tempo/iota/qrels_dev.txt'):
    q, _, d, r = ln.split(); qrels.setdefault(q, {})[d] = int(r)
for ln in open('run.trec'):
    q, _, d, rank, sc, tag = ln.split(); run.setdefault(q, {})[d] = float(sc)
sc = pytrec_eval.RelevanceEvaluator(qrels, {'ndcg_cut.10'}).evaluate(run)
print('nDCG@10', sum(v['ndcg_cut_10'] for v in sc.values()) / len(sc))"

The bundled scorer.py adds RETECO’s diagnostics on top — temporal precision, coverage, and per-turn-depth breakdowns — which are reported alongside the leaderboard but do not determine ranking.

How partial entries are treatedA sub-track you did not enter is simply not ranked — never scored as zero. Within a sub-track you did enter, missing queries, steps, or turns score zero, duplicate document ids are dropped after their first occurrence, and unknown document ids are discarded.

Full evaluation plan →Scorer and format checker ↗

System scope

What systems may use

Participate in one, several, or all RETECO sub-tracks.
Use open-source or proprietary embedding models, retrievers, rerankers, generators, LLMs, and APIs.
Disclose all models, APIs, prompt-based components, and external services in the system description.
Develop on the RETECO training split; treat dev as a held-out check.
Use the organizer-provided document corpus for official retrieval.
Do not replace or augment that corpus with an external retrieval collection for an official run.
Do not attempt to infer, manually label, share, or reconstruct hidden gold judgments.

Final details—including team limits, daily submission caps, hardware reporting, and late-submission policy—will be published with the competition platform.

Reproducibility mattersKeep model versions, prompts, preprocessing, indexing configuration, random seeds, and API dates. These details will be necessary for a strong SemEval system paper.

Resources

Available resources

Resource	Status	Where
Training and development data	Available	Hugging Face ↗
BM25 baseline	Available	Starter kit ↗
Official scorer	Available	Starter kit ↗
Format checker	Available	Starter kit ↗
Registration form	Coming soon	This page
Participant mailing list	Coming soon	This page
Competition platform	Coming soon	This page
Sample submissions	Coming soon	Starter kit

Where the data comes fromThe RETECO release is built from the public TEMPO and RECOR benchmarks at pinned revisions. You do not need to download those separately — everything required is in the RETECO release. They are listed here only for provenance and citation.

SemEval reporting

System-description papers

Participating teams will be invited to describe their methods and analyze their results under the official SemEval paper process. A useful paper should make the system reproducible and explain where it succeeds or fails—not merely state its leaderboard position.

Recommended reporting checklist

Entered sub-tracks and submitted run identifiers.
Retriever, reranker, generator, and LLM/API versions.
Query rewriting, reasoning, or conversation-history strategy.
Temporal representation or decomposition strategy.
Training data, preprocessing, indexing, and hyperparameters.
Ablations and analysis by domain, turn depth, or temporal coverage.
Compute resources, runtime, and use of proprietary components.
Limitations, failure cases, and responsible-use considerations.

Dates remain tentativeSemEval currently lists February 2027 for paper submission, March for notification, and April for camera-ready. We will publish exact dates once the organizers confirm them.

FAQ

Common questions

Must a team enter both tracks?

No. Teams may enter any subset of the five sub-tracks, and entering only one is a legitimate submission. A sub-track you do not enter is not ranked, not scored zero.

May we use commercial LLM APIs?

Yes, according to the accepted task plan, provided every proprietary model, API, prompt-based component, and relevant version is disclosed. Final competition terms still apply.

May we retrieve from the open web or another corpus?

Not for an official retrieval run. Official scoring uses the organizer-provided corpus, which external retrieval corpora may not replace or augment.

Is generation used to rank retrieval systems?

No. The official retrieval leaderboard uses nDCG@10. Generation judgments for Sub-tracks 2b and 2c are reported separately.

Do I need to download TEMPO or RECOR separately?

No. The RETECO release on Hugging Face contains every corpus, task file, and qrels file you need. TEMPO and RECOR are the upstream sources it was built from; you only need them if you want to cite or inspect the originals.

Is the dev split the SemEval test set?

No. Both train and dev come with public gold labels and are for your own development. The SemEval test set is separate, unseen, and released without judgments during the evaluation window.

Can I train on the dev split too?

Nothing prevents it technically, and the final rules will not police it. But dev is your only honest estimate of how a system will behave on the hidden test — spend it carefully.

Which nDCG implementation is official?

pytrec_eval with ndcg_cut_10, macro-averaged over topics. The starter kit uses exactly this, so a local score and a leaderboard score are directly comparable.

Will source code be mandatory?

The final competition rules will specify release requirements. Regardless, system papers must disclose enough technical detail to support reproducibility.

Participant support

Correspondence

Until the task mailing list opens, task-specific questions may be sent to the lead organizers. Please do not send hidden-test predictions, credentials, or private dataset copies by email.

Email lead organizer RETECO GitHub ↗