Rules

🚧 This website is currently under construction. All content is subject to change and may not reflect the final details of the competition. Please refer to official announcements once the contest begins for the most accurate and up-to-date information.

1. When generating the training and validation datasets, ONLY the speech, nosie, and room impulse response (RIR) corpora listed in the Data tab shall be used to ensure a fair comparison and proper understanding of various SE approaches.

   • The first month of the challenge will be a grace period when participants can propose additional public datasets to be included in the list. We (organizers) will reply to the requests and may update the list. Updates will be recorded in the Notice tab.

   • It is NOT allowed to use pre-trained speech enhancement models trained on other than official Challenge data.
     • However, it IS allowed to use a pre-trained model trained on the official challenge data (e.g., URGENT 2024 / URGENT 2025 / URGENT 2026 official baseline )
   • Although the speech enhancement model should only be trained on the listed data, we allow the use of pre-trained foundation models such as HuBERT, WavLM, EnCodec, Llama, and so on. We also allow the use of a pretrained speech enhancement/restoration model to improve the quality of clean speech for simulation. The use of all pre-trained models must meet the following requirements:
     • They are publicly available before the challenge begins
     • They are explicitly mentioned in the submitted system description.
     • Note:
       • Their parameters can be fine-tuned on the listed data.
       • It is not allowed to fine-tune any model, be it pre-trained or not, on any extra data other than the listed data.
         
         
   • If you are unsure whether the pre-trained model you would like to use is allowed, please reach out to the organizers.

2. The test data should only be used for evaluation purposes. Techniques such as test-time adaptation, unsupervised domain adaptation, and self-training on the test data are NOT allowed.

3. There is no constraint on the latency or causality of the developed system in this challenge. Any type of model can be used as long as it conforms to the other rules as listed on this page.

4. Registration is required to submit results to the challenge (Check the Leaderboard tab for more information). Note that the team information (including affiliation, team name, and team members) should be provided when submitting the results. For detailed submission requirements, please check the Submission tab.
   • Only the team name will be shown in the leaderboard, while the affiliation and team members will be kept confidential.
     
     

   [update] Add four Non-intrusive metrics.

5. The following evaluation metrics will be calculated for evaluation.

   Category	Metric	Need Reference Signals?	Supported Sampling Frequencies	Value Range
   Non-intrusive SE metrics	DNSMOS ↑	❌	16 kHz	[1, 5]
   	NISQA ↑	❌	48 kHz	[1, 5]
   	UTMOS ↑	❌	16 kHz	[1, 5]
   	ScoreQ ↑	❌	16 kHz	[0.0, 1.0]
   Intrusive SE metrics	POLQAThis metric will only be used for evaluation of the final blind test set. ↑	✔	8~48 kHz	[1, 5]
   	PESQ ↑	✔	{8, 16} kHz	[-0.5, 4.5]
   	ESTOI ↑	✔	10 kHz	[0, 1]
   	SDR ↑	✔	Any	(-∞, +∞)
   	MCD ↓	✔	Any	[0, +∞)
   	LSD ↓	✔	Any	[0, +∞)
   Downstream-task-independent metrics	SpeechBERTScoreTo evaluate multilingual speech, we adopt the MHuBERT-147 backend for calculating the SpeechBERTScore, which differs from its defalut backend (WavLM-Large). ↑	✔	16 kHz	[-1, 1]
   	LPS ↑	✔	16 kHz	(-∞, 1]
   Downstream-task-dependent metrics	Speaker Similarity ↑	✔	16 kHz	[-1, 1]
   	EMOTION2VEC Similarity ↑	❌	16 kHz	[-1, 1]
   	Language identification accuracy ↑	❌	16 kHz	[0, 1]
   	Character accuracy (1 - CER) ↑	❌	16 kHz	(-∞, 1]
   Subjective SE metrics	MOSThis metric will only be used for evaluation of the final blind test set. ↑	❌	Any	[1, 5]

   
   

   Note: For real recorded test samples that do not have a strictly matched reference signal, part of the above metrics will be used.

6. The overall ranking will be determined via the following procedure:

   1. Calculate the average score of each metric for each submission.
   2. Calculate the per-metric ranking based on the average score.
      • We adopt the dense ranking (“1223” ranking)https://en.wikipedia.org/wiki/Ranking#Dense_ranking_("1223"_ranking) strategy for handling ties.
   3. Calculate the per-category ranking by averaging the rankings within each category.
   4. Calculate the overall ranking by averaging the per-category rankings.
      
      

    # Step 1: Calculate the average score of each metric
    scores = {}
    for submission in all_submissions:
      scores[submission] = {}
      for category in metric_categories:
        for metric in category:
          scores[submission][metric] = mean([metric(each_sample) for each_sample in submission])
   
    # Step 2: Calculate the per-metric ranking based on the average score
    rank_per_metric = {}
    rank_per_category = {}
    for category in metric_categories:
      for metric in category:
        rank_per_metric[metric] = get_ranking([scores[submission][metric] for submission in all_submissions])
   
      # Step 3: Calculate the `per-category ranking` by averaging the rankings within each category
      rank_per_category[category] = get_ranking([rank_per_metric[metric] for metric in category])
   
    # Step 4: Calculate the overall ranking by averaging the `per-category rankings`
    rank_overall = get_ranking([rank_per_category[category] for category in metric_categories])
   

   Note: Only the original test data, the best baseline system, and participant submissions are taken into account in the ranking procedure.