01Register
Register the team and join the announcement channel when forms open.
RETECO · SemEval-2027 Task 1
Participation guide
Information for selecting a track, validating a system, submitting predictions, and reporting results reproducibly.
Quick start
Five steps to participate.
02Choose
Select any combination of the five RETECO sub-tracks.
03Build
Develop on the public TEMPO and RECOR training resources.
04Validate
Run the official format checker and local scorer before submission.
05Submit
Upload hidden-test predictions during the evaluation window.
Links are intentionally not guessedOfficial registration, mailing-list, and competition URLs will be activated here as soon as they are confirmed.
System scope
What systems may use
- Participate in one, several, or all RETECO sub-tracks.
- Use open-source or proprietary embedding models, retrievers, rerankers, generators, LLMs, and APIs.
- Disclose all models, APIs, prompt-based components, and external services in the system description.
- Use the organizer-provided document corpus for official retrieval.
- Do not replace or augment that corpus with an external retrieval collection for an official run.
- Do not attempt to infer, manually label, share, or reconstruct hidden gold judgments.
Final details—including team limits, daily submission caps, hardware reporting, and late-submission policy—will be published with the competition platform.
Reproducibility mattersKeep model versions, prompts, preprocessing, indexing configuration, random seeds, and API dates. These details will be necessary for a strong SemEval system paper.
Build resources
Start from operational baselines.
| Resource | Status | Destination |
|---|---|---|
| TEMPO data | Available | Hugging Face ↗ |
| TEMPO baseline code | Available | GitHub ↗ |
| RECOR data | Available | Hugging Face ↗ |
| RECOR baseline code | Available | GitHub ↗ |
| SemEval starter package | Coming soon | RETECO repository |
| Format checker + scorer | Coming soon | RETECO repository |
SemEval reporting
System-description papers
Participating teams will be invited to describe their methods and analyze their results under the official SemEval paper process. A useful paper should make the system reproducible and explain where it succeeds or fails—not merely state its leaderboard position.
Recommended reporting checklist
- Entered sub-tracks and submitted run identifiers.
- Retriever, reranker, generator, and LLM/API versions.
- Query rewriting, reasoning, or conversation-history strategy.
- Temporal representation or decomposition strategy.
- Training data, preprocessing, indexing, and hyperparameters.
- Ablations and analysis by domain, turn depth, or temporal coverage.
- Compute resources, runtime, and use of proprietary components.
- Limitations, failure cases, and responsible-use considerations.
Dates remain tentativeSemEval currently lists February 2027 for paper submission, March for notification, and April for camera-ready. We will publish exact dates once the organizers confirm them.
FAQ
Common questions
Must a team enter both tracks?
No. Teams may participate in any subset of the five sub-tracks.
May we use commercial LLM APIs?
Yes, according to the accepted task plan, provided every proprietary model, API, prompt-based component, and relevant version is disclosed. Final competition terms still apply.
May we retrieve from the open web or another corpus?
Not for an official retrieval run. Official scoring uses the organizer-provided corpus, which external retrieval corpora may not replace or augment.
Is generation used to rank retrieval systems?
No. The official retrieval leaderboard uses nDCG@10. Generation judgments for Sub-tracks 2b and 2c are reported separately.
Are the public TEMPO and RECOR sets the SemEval test?
No. They are public training/development foundations. The SemEval test is separate, unseen, and held out.
Will source code be mandatory?
The final competition rules will specify release requirements. Regardless, system papers must disclose enough technical detail to support reproducibility.
Participant support
Contact the organizers.
Until the task mailing list opens, task-specific questions may be sent to the lead organizers. Please do not send hidden-test predictions, credentials, or private dataset copies by email.