RETECO
SemEval 2027
Home/Task

RETECO · SemEval-2027 Task 1

Task definition

RETECO unifies temporal grounding and conversational reasoning under a common ranked-retrieval interface, then extends that interface to grounded generation.

Overview

Why reasoning-oriented retrieval?

Most retrieval benchmarks reward topical similarity. Real information needs are less forgiving: a document may discuss the right topic but the wrong period, or contain the right vocabulary without resolving what “that” means in the current conversation.

RETECO evaluates relevance conditioned on these missing dimensions. The tracks share the same core question—can a system rank the passages that truly support the information need?—while isolating two kinds of reasoning.

Official retrieval outputFor each query or target conversation turn, systems submit an ordered list of document identifiers. Higher-ranked relevant documents receive more credit through nDCG@10.

Track 01 · TEMPO

Temporal Grounded Retrieval

A system receives a temporally grounded query and a domain corpus. It must retrieve passages that are topically relevant and aligned with the time periods required by the query. A passage containing an unrelated date does not count as temporally aligned.

Sub-track 1a — Temporal Retrieval

Input

Query + corpus

A complex temporal information need and the corresponding domain document collection.

Output

Ranked documents

An ordered list of document identifiers for the complete query.

Sub-track 1b — Step-wise Temporal Retrieval

Input

Decomposed steps + corpus

The temporal query is represented through reasoning steps, each targeting a necessary part or period.

Output

Rankings by step

An ordered document list for each intermediate retrieval step.

What makes Track 1 difficult?The public TEMPO pilot covers ten temporal reasoning classes, including event analysis/localization and time-period contextualization. Strong systems still fail to cover all required periods for many queries.
Public pilot record · simplified{ "id": "122291_22", "query": "Why was there a jump in transactions per block in 2023?", "gold_ids": ["bitcoin/25785deb_4434.txt", "…"] }

Track 02 · RECOR

Reasoning-Intensive Conversational Retrieval

A system receives the current user turn together with the preceding conversation. It must resolve discourse dependencies—such as coreference and ellipsis—and identify passages whose relevance may require a multi-step conceptual link rather than surface overlap.

Sub-track 2a — Conversational Retrieval

Input

Conversation + corpus

The dialogue history, current target turn, and corresponding domain collection.

Output

Ranked passages

An ordered list of supporting document identifiers for the current turn.

Sub-track 2b — Generation with Gold Passages

Input

Conversation + gold evidence

Organizer-provided supporting passages isolate answer generation from retrieval quality.

Output

Grounded response

A direct, coherent answer faithful to the supplied passages and dialogue context.

Sub-track 2c — Full Conversational RAG

Input

Conversation + corpus

The complete end-to-end setting without gold passages supplied to the system.

Output

Top-5 + response

Up to five retrieved passage identifiers and a generated, evidence-grounded answer.

What makes Track 2 difficult?Later turns are rarely standalone. The RECOR pilot shows that adding conversation history and explicit reasoning can roughly double retrieval effectiveness, exposing how much context is lost by raw-query retrieval.
Public pilot turn · simplified{ "turn_id": 2, "query": "Wait, did any works actually get their copyright back?", "conversation_history": "Q: … A: …", "gold_doc_ids": ["law_ex_111149_doc_1", "…"] }

Evaluation data

A genuinely held-out test.

The public TEMPO and RECOR releases serve as the foundation for training and development. SemEval evaluation uses a separate, never-publicly-released test set drawn from unseen source material.

  • Approximately 350 Track 1 test queries, stratified across all 13 TEMPO domains.
  • Approximately 200 Track 2 target turns, stratified across all 11 RECOR domains.
  • Every item is manually re-verified; at least 25% receive double annotation and organizer adjudication.
  • Gold labels remain withheld throughout the evaluation phase.
Contamination protectionParticipants must not attempt to reconstruct, infer, or manually annotate hidden gold labels. Detailed competition rules will accompany the evaluation release.

Participation scope

Enter one sub-track or connect the full pipeline.

Teams may participate in any subset of the five sub-tracks. The provided corpus defines the official retrieval collection. Open-source models, proprietary models, and external model APIs are permitted when their use is clearly disclosed in the system description, subject to the final competition rules.

Participation guide →How scoring works

Ready to work with the data?

Explore datasets →