Rules

• This is to ensure a fair comparison and proper understanding of various SE approaches.
• The first month of the challenge will be a grace period when participants can propose additional public datasets to be included in the list. We (organizers) will reply to the requests and may update the list. Updates will be recorded in the Notices tab.
• Although the speech enhancement model should only be trained on the listed data, we allow the use of pre-trained foundation models such as HuBERT, WavLM, EnCodec, Llama, and so on as long as:
  • they are publicly available before the challenge begins
  • and they are explicitly mentioned in the submitted system description.
  • Note:
    • Their parameters can be fine-tuned on the listed data.
    • It is not allowed to fine-tune any model, be it pre-trained or not, on any extra data other than the listed data.
      
      

The test data should only be used for evaluation purposes. Techniques such as test-time adaptation, unsupervised domain adaptation, and self-training on the test data are not allowed for this challenge.

• Only the team name will be shown in the leaderboard, while the affiliation and team members will be kept confidential.
  
  

The following evaluation metrics will be calculated for evaluation.

1. Note that for the final ranking on the blind test data, further metrics (such as POLQAhttp://www.polqa.info and MOS) might be used, operating either at 8 kHz, at 16 kHz, or at 48 kHz SF.
2. For real recorded test samples that do not have a strictly matched reference signal, part of the above metrics will be used.

The overall ranking will be determined via the following procedure:

1. Calculate the average score of each metric for each submission.
2. Calculate the per-metric ranking based on the average score.
   • We adopt the standard competition ranking (“1224” ranking)https://en.wikipedia.org/wiki/Ranking#Standard_competition_ranking_("1224"_ranking) strategy for handling ties.
3. Calculate the per-category ranking by averaging the rankings within each category.
4. Calculate the overall ranking by averaging the per-category rankings.
   
   

 # Step 1: Calculate the average score of each metric
 scores = {}
 for submission in all_submissions:
   scores[submission] = {}
   for category in metric_categories:
     for metric in category:
       scores[submission][metric] = mean([metric(each_sample) for each_sample in submission])

 # Step 2: Calculate the per-metric ranking based on the average score
 rank_per_metric = {}
 rank_per_category = {}
 for category in metric_categories:
   for metric in category:
     rank_per_metric[metric] = get_ranking([scores[submission][metric] for submission in all_submissions])

   # Step 3: Calculate the `per-category ranking` by averaging the rankings within each category
   rank_per_category[category] = get_ranking([rank_per_metric[metric] for metric in category])

 # Step 4: Calculate the overall ranking by averaging the `per-category rankings`
 rank_overall = get_ranking([rank_per_category[category] for category in metric_categories])

Note: Only the original test data, the best baseline system, and participant submissions are taken into account in the ranking procedure.