Rules

• The first month of the challenge (until Dec. 15) will be a grace period when participants can propose additional public datasets to be included in the list. We (organizers) will reply to the requests and may update the list. Updates will be recorded in the Notice tab.
• Although the speech enhancement model should only be trained on the listed data, we allow the use of pre-trained foundation models such as HuBERT, WavLM, EnCodec, Llama, and so on as long as:
  • they are publicly available before the challenge begins
  • and they are explicitly mentioned in the submitted system description.
  • Note:
    • Their parameters can be fine-tuned on the listed data.
    • It is not allowed to fine-tune any model, be it pre-trained or not, on any extra data other than the listed data.
      
      
• It is allowed to use a model trained on the “NeurIPS 2024 URGENT challenge” data, as those data are included in this challenge too.

The test data should only be used for evaluation purposes. Techniques such as test-time adaptation, unsupervised domain adaptation, and self-training on the test data are NOT allowed for this challenge.

There is no constraint on the latency or causality of the developed system in this challenge. Any type of model can be used as long as they conform to the other rules as listed in this page.

• Only the team name will be shown in the leaderboard, while the affiliation and team members will be kept confidential.
  
  

The following evaluation metrics will be calculated for evaluation.

Note: For real recorded test samples that do not have a strictly matched reference signal, part of the above metrics will be used.

The overall ranking will be determined via the following procedure:

1. Calculate the average score of each metric for each submission.
2. Calculate the per-metric ranking based on the average score.
   • We adopt the dense ranking (“1223” ranking)https://en.wikipedia.org/wiki/Ranking#Dense_ranking_("1223"_ranking) strategy for handling ties.
3. Calculate the per-category ranking by averaging the rankings within each category.
4. Calculate the overall ranking by averaging the per-category rankings.
   
   

 # Step 1: Calculate the average score of each metric
 scores = {}
 for submission in all_submissions:
   scores[submission] = {}
   for category in metric_categories:
     for metric in category:
       scores[submission][metric] = mean([metric(each_sample) for each_sample in submission])

 # Step 2: Calculate the per-metric ranking based on the average score
 rank_per_metric = {}
 rank_per_category = {}
 for category in metric_categories:
   for metric in category:
     rank_per_metric[metric] = get_ranking([scores[submission][metric] for submission in all_submissions])

   # Step 3: Calculate the `per-category ranking` by averaging the rankings within each category
   rank_per_category[category] = get_ranking([rank_per_metric[metric] for metric in category])

 # Step 4: Calculate the overall ranking by averaging the `per-category rankings`
 rank_overall = get_ranking([rank_per_category[category] for category in metric_categories])

Note: Only the original test data, the best baseline system, and participant submissions are taken into account in the ranking procedure.