RETECO · SemEval-2027 Task 1
Evaluation plan
A single official retrieval metric keeps the competition interpretable; targeted diagnostics expose temporal, conversational, and generative behavior.
Official leaderboard
Retrieval ranks by nDCG@10.
For both tracks, binary normalized Discounted Cumulative Gain at rank 10 is the official retrieval ranking metric. It rewards relevant documents appearing early while normalizing for the number of available relevant documents.
DCG@10 = Σ(i=1..10) relᵢ / log₂(i + 1)
nDCG@10 = DCG@10 / IDCG@10| Sub-track | Official ranking | Additional reporting |
|---|---|---|
| 1a Temporal retrieval | Query-level nDCG@10 | Temporal + standard diagnostics |
| 1b Step-wise retrieval | Step-level nDCG@10 | Temporal + standard diagnostics |
| 2a Conversational retrieval | Turn-level nDCG@10 | Turn depth + domain diagnostics |
| 2b Gold-passage generation | — | Five judge dimensions |
| 2c Full conversational RAG | Turn-level nDCG@10 | Five judge dimensions |
Track 01 scoring
TEMPO retrieval and temporal diagnostics
Aggregation
For Sub-track 1a, nDCG@10 is computed independently for each query and macro-averaged. For Sub-track 1b, step-specific nDCG@10 values are first averaged over the supplied steps for each query, then aggregated across queries.
Temporal diagnostics
- Temporal Precision@10: position-sensitive precision that rewards temporally relevant documents ranked earlier.
- Temporal Relevance@10: the fraction of the top ten results judged temporally relevant.
- Temporal Coverage@10: the fraction of required time periods covered by at least one top-ten document.
- nDCG|FC@10: nDCG@10 over the subset of queries achieving full temporal coverage.
MAP@10, Recall@10, MRR, and per-domain results are also reported diagnostically.
Track 02 scoring
RECOR retrieval diagnostics
For Sub-tracks 2a and 2c, nDCG@10 is computed per target turn and macro-averaged for official ranking. Diagnostic reporting includes MAP@10, Recall@10, MRR, per-domain results, and conversation-depth groups:
Turn-depth analysis helps distinguish single-turn retrieval quality from the ability to retain and resolve accumulated dialogue context.
Sub-tracks 2b & 2c
Five focused generation judgments
Each generated response is assessed on five independent dimensions with separate focused prompts and a fixed 1–5 Likert rubric.
| Dimension | Question answered |
|---|---|
| Correctness | Are the response’s factual claims accurate? |
| Completeness | Does it cover the information needed by the turn? |
| Relevance | Does it directly address the user’s current need? |
| Conversational coherence | Does it fit the preceding dialogue? |
| Faithfulness | Is it supported by supplied or retrieved passages? |
Each integer score is normalized as (score − 1) / 4 and macro-averaged for reporting. ROUGE-L, METEOR, and BERTScore are additional diagnostics.
Calibration & audit
Judge behavior is fixed before evaluation.
The five prompts, rubric anchors, GPT-4o version, decoding settings, and output schema are frozen before the competition. Golden-case meta-evaluation checks whether the judge distinguishes supported from unsupported, complete from incomplete, and coherent from incoherent responses.
Judge outputs are spot-validated against independent human ratings on the SemEval test set. The task report will disclose the sample, agreement results, prompts, judge version, retry procedure, and invalid or missing outputs.
Deterministic handling
How submitted runs are interpreted
Retrieval runs use the standard six-column TREC format:
query_id Q0 document_id rank score run_id- Missing queries, steps, or target turns receive zero.
- Duplicate document identifiers are removed after their first occurrence.
- Unknown document identifiers are discarded.
- Official retrieval must use the organizer-provided corpus.
- External corpora may not replace or augment the official corpus for official ranking.
Generation output is UTF-8 JSONL containing a target turn_id and answer. Exact validation utilities and sample submissions will be released before evaluation.