ESPnet Tutorial

.huge.bold[A quick tutorial on how to use<br/><br/>ESPnet]

---

# Table of Contents

1. Basic directory structure

* [Recipe](#espnet_recipe)

* [Framework](#espnet_framework)

2. [Data format](#data_format): `dump/raw/<subset-name>/`

3. [Task definition](#task_definition): `espnet2/tasks/enh.py`

4. [Model definition](#model_definition): `espnet2/enh/`

5. [Model configuration](#model_configuration): `conf/tuning/xxx.yaml`

6. [Script flow of training and inference](#script_flow)

---

# Basic directory structure - Recipe

</div>
]

.right-column-61[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** Model configurations (YAML files)
***** Data preparation scripts (used by run.sh)
***** .red[Directory generated by local/data.sh] <a href="#data_format" style="text-decoration:none">👉 <code class="remark-inline-code">Data format</code></a>
***** .red[Data directory used for training/evaluation] <a href="#data_format" style="text-decoration:none">👆</a>
***** SLURM config (cmd_backend: 'local' / 'slurm' /...)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

.full-column[
```bash
cd <espnet-root-dir>/egs2/urgent24/enh1
./run.sh --stage 1 --stop-stage 1
mkdir -p dump/raw
cp -r data/* dump/raw/
```
]

---

# Basic directory structure - Recipe (Cont'd)

.left-column-48[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24" style="text-decoration:none">📁 urgent24</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 enh1</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/conf" style="text-decoration:none; color:red;">📁 conf</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/local" style="text-decoration:none">📁 local</a>
│           ├─ <span style="color:#004276;">📁 data</span>
│           ├─ <span style="color:#004276;">📁 dump</span>
│           ├─ .red[📁 exp]
│           │   └─ .red[📁 enh_stats_16k]
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/cmd.sh" style="text-decoration:none">📄 cmd.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/path.sh" style="text-decoration:none">📄 path.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">📄 enh.sh</a>
│           └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none; color:red;">📄 run.sh</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>

</div>
]

.right-column-51[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** .red[Model configurations] <a href="#model_configuration" style="text-decoration:none">👉 <code class="remark-inline-code">Model configuration</code></a>
***** Data preparation scripts (used by run.sh)
***** Directory generated by local/data.sh
***** Data directory used for training/evaluation
***** .red[ Exp directory for model training/inference]
****** .red[Length stats used for minibatch construction]
***** SLURM config file ('local', 'slurm', etc.)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

.full-column[
```bash
cd <espnet-root-dir>/egs2/urgent24/enh1
./run.sh --stage 5 --stop-stage 5 --nj 8
```
]

---

# Basic directory structure - Recipe (Cont'd)

.left-column-48[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24" style="text-decoration:none">📁 urgent24</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 enh1</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/conf" style="text-decoration:none; color:red;">📁 conf</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/local" style="text-decoration:none">📁 local</a>
│           ├─ <span style="color:#004276;">📁 data</span>
│           ├─ <span style="color:#004276;">📁 dump</span>
│           ├─ .red[📁 exp]
│           │   └─ .red[📁 enh_train_enh_xxx_raw]
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/cmd.sh" style="text-decoration:none">📄 cmd.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/path.sh" style="text-decoration:none">📄 path.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">📄 enh.sh</a>
│           └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none; color:red;">📄 run.sh</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>

</div>
]

.right-column-51[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** .red[Model configurations] <a href="#model_configuration" style="text-decoration:none">👉 <code class="remark-inline-code">Model configuration</code></a>
***** Data preparation scripts (used by run.sh)
***** Directory generated by local/data.sh
***** Data directory used for training/evaluation
***** .red[Exp directory for model training/inference]
<span>          ****** <span style="color:red">Trained model directory</span></span>
***** SLURM config file ('local', 'slurm', etc.)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

.full-column[
```bash
cd <espnet-root-dir>/egs2/urgent24/enh1
./run.sh --stage 6 --stop-stage 6 --enh_config conf/tuning/xxx.yaml --num_nodes 1 --ngpu 1
```
]

---

# Basic directory structure - Recipe (Cont'd)

.left-column-48[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24" style="text-decoration:none">📁 urgent24</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 enh1</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/conf" style="text-decoration:none; color:red;">📁 conf</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/local" style="text-decoration:none">📁 local</a>
│           ├─ <span style="color:#004276;">📁 data</span>
│           ├─ <span style="color:#004276;">📁 dump</span>
│           ├─ .red[📁 exp]
│           │   └─ .red[📁 enh_train_enh_xxx_raw]
│           │       ├─ <span style="color:#004276;">📁 images</span>
│           │       ├─ <span style="color:#004276;">📄 1epoch.pth</span>
│           │       ├─ <span style="color:#004276;">📄 valid.loss.best.pth</span>
│           │       └─ <span style="color:#004276;">📄 train.log</span>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/cmd.sh" style="text-decoration:none">📄 cmd.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/path.sh" style="text-decoration:none">📄 path.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">📄 enh.sh</a>
│           └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none; color:red;">📄 run.sh</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>

</div>
]

.right-column-51[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** .red[Model configurations] <a href="#model_configuration" style="text-decoration:none">👉 <code class="remark-inline-code">Model configuration</code></a>
***** Data preparation scripts (used by run.sh)
***** Directory generated by local/data.sh
***** Data directory used for training/evaluation
***** .red[Exp directory for model training/inference]
<span>          ****** <span style="color:red">Trained model directory</span></span>
<span>          ****** <span style="color:#004276;">Training curves</span></span>
<span>          ****** <span style="color:#004276;">Latest checkpoint</span></span>
<span>          ****** <span style="color:#004276;">Best checkpoint (when finished)</span></span>
<span>          ****** <span style="color:#004276;">Training log</span></span>
***** SLURM config file ('local', 'slurm', etc.)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

---

# Basic directory structure - Recipe (Cont'd)

.left-column-48[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24" style="text-decoration:none">📁 urgent24</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 enh1</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/conf" style="text-decoration:none; color:red;">📁 conf</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/local" style="text-decoration:none">📁 local</a>
│           ├─ <span style="color:#004276;">📁 data</span>
│           ├─ <span style="color:#004276;">📁 dump</span>
│           ├─ .red[📁 exp]
│           │    └─ .red[📁 enh_train_enh_xxx_raw]
│           │        └─ .red[📁 enhanced_validation]
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/cmd.sh" style="text-decoration:none">📄 cmd.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/path.sh" style="text-decoration:none">📄 path.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">📄 enh.sh</a>
│           └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none; color:red;">📄 run.sh</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>

</div>
]

.right-column-51[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** .red[Model configurations] <a href="#model_configuration" style="text-decoration:none">👉 <code class="remark-inline-code">Model configuration</code></a>
***** Data preparation scripts (used by run.sh)
***** Directory generated by local/data.sh
***** Data directory used for training/evaluation
***** .red[Exp directory for model training/inference]
<span>          ****** <span style="color:red">Trained model directory</span></span>
<span>          ******* <span style="color:red">Enhanced audios (spk1.scp)</span></span>
***** SLURM config file ('local', 'slurm', etc.)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

.full-column[
```bash
cd <espnet-root-dir>/egs2/urgent24/enh1
./run.sh --stage 7 --stop-stage 7 --enh_config conf/tuning/xxx.yaml --inference_nj 8 --gpu_inference 8
```
]

---

# Basic directory structure - Recipe (Cont'd)

.left-column-48[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24" style="text-decoration:none">📁 urgent24</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 enh1</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/conf" style="text-decoration:none; color:red;">📁 conf</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/local" style="text-decoration:none">📁 local</a>
│           ├─ <span style="color:#004276;">📁 data</span>
│           ├─ <span style="color:#004276;">📁 dump</span>
│           ├─ .red[📁 exp]
│           │    └─ .red[📁 enh_train_enh_xxx_raw]
│           │        └─ .red[📁 enhanced_validation]
│           │            ├─ <span style="color:#004276;">📁 logdir</span>
│           │            └─ <span style="color:#004276;">📄 spk1.scp</span>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/cmd.sh" style="text-decoration:none">📄 cmd.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/path.sh" style="text-decoration:none">📄 path.sh</a>
│           ├─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">📄 enh.sh</a>
│           └─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none; color:red;">📄 run.sh</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>

</div>
]

.right-column-51[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
*** Recipe for a specific dataset
**** Speech enhancement recipe
***** .red[Model configurations] <a href="#model_configuration" style="text-decoration:none">👉 <code class="remark-inline-code">Model configuration</code></a>
***** Data preparation scripts (used by run.sh)
***** Directory generated by local/data.sh
***** Data directory used for training/evaluation
***** .red[Exp directory for model training/inference]
<span>          ****** <span style="color:red">Trained model directory</span></span>
<span>          ******* <span style="color:red">Enhanced audios (spk1.scp)</span></span>
<span>          ******** <span style="color:#004276;">Enhanced audio directory</span></span>
<span>          ******** <span style="color:#004276;">list of all enhanced audios</span></span>
***** SLURM config file ('local', 'slurm', etc.)
***** Envrionment config file (export XXX=YYY)
***** Common script used for speech enhancement
***** .red[Entry script for running the recipe]
** Python scripts used for training/inference
** Used for toolkit installation

</div>
]

---

# Basic directory structure - Framework

.left-column-38[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/egs2" style="text-decoration:none">📁 egs2</a>
├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2" style="text-decoration:none">📁 espnet2</a>
│   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/bin" style="text-decoration:none">📁 bin</a>
│   │   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/bin/enh_train.py" style="text-decoration:none">📄 enh_train.py</a>
│   │   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/bin/enh_inference.py" style="text-decoration:none; color:red;">📄 enh_inference.py</a>
│   │   └─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/bin/enh_scoring.py" style="text-decoration:none">📄 enh_scoring.py</a>
│   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/tasks" style="text-decoration:none">📁 tasks</a>
│   │   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/tasks/abs_task.py" style="text-decoration:none">📄 abs_task.py</a>
│   │   └─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/tasks/enh.py" style="text-decoration:none; color:red;">📄 enh.py</a>
│   └─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh" style="text-decoration:none">📁 enh</a>
│       ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/encoder" style="text-decoration:none">📁 encoder</a>
│       ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/separator" style="text-decoration:none">📁 separator</a>
│       ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/decoder" style="text-decoration:none">📁 decoder</a>
│       ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/layers" style="text-decoration:none">📁 layers</a>
│       ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/loss" style="text-decoration:none">📁 loss</a>
│       │   ├─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/loss/criterions" style="text-decoration:none">📁 criterions</a>
│       │   └─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/loss/wrappers" style="text-decoration:none">📁 wrappers</a>
│       └─ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/espnet_model.py" style="text-decoration:none; color:red;">📄 espnet_model.py</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/tools" style="text-decoration:none">📁 tools</a>
</div>
]

.right-column-61[
<div class="code-like-block remark-code">
* Root directory
** Directory containing all recipes
** Python scripts used for training/inference
*** Entry scripts (used by <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1/enh.sh" style="text-decoration:none">enh.sh</a>)
**** Entry script for training
**** .red[Entry script for inference (enhancing a subset)]
**** Entry script for scoring (evaluating enhanced audios)
*** Task definition
**** Abstract task definition (shared by all tasks)
**** .red[Speech enhancement task definition] <a href="#task_definition" style="text-decoration:none">👉 <code class="remark-inline-code">Task definition</code></a>
*** Speech enhancement model definition <a href="#model_definition" style="text-decoration:none">👉 <code class="remark-inline-code">Model definition</code></a>
**** Available encoders (<a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/encoder/stft_encoder.py" style="text-decoration:none;">STFT</a>, <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/encoder/conv_encoder.py" style="text-decoration:none;">Conv1d</a>, etc.)
**** Available separators (<a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/separator/bsrnn_separator.py" style="text-decoration:none;">BSRNN</a>, <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/separator/tfgridnetv3_separator.py" style="text-decoration:none;">TF-GridNet</a>, etc.)
**** Available decoders (<a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/decoder/stft_decoder.py" style="text-decoration:none;">iSTFT</a>, <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/decoder/conv_decoder.py" style="text-decoration:none;">ConvTranspose1d</a>, etc.)
**** Detailed layer definitions used in separators
*** Loss functions
**** Criterion functions (<a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/criterions/time_domain.py" style="text-decoration:none;">time</a> / <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/criterions/tf_domain.py" style="text-decoration:none;">time-frequency</a> domain)
**** Wrappers (<a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/wrappers/fixed_order.py" style="text-decoration:none;">fixed permutation</a>, <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/wrappers/pit_solver.py" style="text-decoration:none;">PIT</a>, etc.)
*** .red[Common framework of speech enhancement models]
** Used for toolkit installation
</div>
]

---

# Data format

In speech enhancement recipes, we need to prepare the following seven files:
.left-column-40[
<div class="code-like-block remark-code">
<a href="https://github.com/espnet/espnet" style="text-decoration:none">📁 [espnet-root-dir]</a>
└─ <a href="https://github.com/espnet/espnet/tree/master/egs2/urgent24/enh1" style="text-decoration:none">📁 egs2/urgent24/enh1</a>
    ├─ <span style="color:#004276;">📁 data</span>
    └─ <span style="color:#004276;">📁 dump/raw</span>
         ├─ <span style="color:#004276;">📁 train</span>
         │   ├─ <span style="color:red">📄 wav.scp</span>
         │   ├─ <span style="color:red">📄 spk1.scp</span>
         │   ├─ <span style="color:red">📄 text</span>
         │   ├─ <span style="color:red">📄 utt2spk</span>
         │   ├─ <span style="color:red">📄 spk2utt</span>
         │   ├─ <span style="color:red">📄 utt2fs</span>
         │   └─ <span style="color:red">📄 utt2category</span>
         ├─ <span style="color:#004276;">📁 validation</span>
         └─ <span style="color:#004276;">📁 test</span>

</div>
]

.right-column-59[
<div class="code-like-block remark-code">
* Root directory
** Recipe for the URGENT 2024 Challenge
*** Directory generated by local/data.sh
*** Data directory used for training/evaluation
**** Training set
***** .red[List of degraded speech (model input)]
***** .red[Corresponding clean speech (label)]
***** .red[Corresponding transcripts (for ASR evaluation)]
***** .red[Corresponding speaker IDs]
***** .red[Generated by utils/utt2spk_to_spk2utt.pl]
***** .red[Corresponding sampling rate in Hz]
***** .red[Corresponding category (for minibatch construction)]
**** Validation set
**** Test set
</div>
]

.full-column[
In each data file, the data format follows the <a href="https://kaldi-asr.org/doc/data_prep.html" style="text-decoration:none;">Kaldi</a> convention, i.e., a <span style="color:#004276;">table-style</span> format with the first column as the utterance ID (key) and the rest columns as the value.
]

---

# Data format (Cont'd)

|File name|Example content|Note|
|---|---|---|
|wav.scp|<span style="color:red">uid1</span> /path/to/noisy/uid1.flac|Essential|
|spk1.scp|<span style="color:red">uid1</span> /path/to/clean/uid1.flac|Needed in stage 8|
|text|<span style="color:red">uid1</span> It is also very valuable.|Needed in stage 8+|
|utt2spk|<span style="color:red">uid1</span> sid1|Essential|
|spk2utt|<span style="color:red">sid1</span> uid1 uid7 uid16 uid99|Essential|
|utt2fs|<span style="color:red">uid1</span> 32000|Essential for multi-fs data|
|utt2category|<span style="color:red">uid1</span> 1ch_32000Hz|Essential for multi-fs data|

> Note that `utt2fs` is required to run model inference in stage 7 of `run.sh` so that the enhanced audio can be stored in the correct sampling rate.
> <br/><br/>
> For the officially released validation/test subset (downloaded audios), you will need to generate the above data files manually.

---

# Data format (Cont'd)

To manually generate data files for an audio directory `audios`, you could run the following commands (assuming all file names are unique):
```bash
mkdir -p dump/raw/<subset-name>
find audios/ -iname '*.flac' | awk -F'[/.]' '{print($(NF-1)" "$0)}' | \
    sort -u > dump/raw/<subset-name>/wav.scp
find audios/ -iname '*.flac' | awk -F'[/.]' '{print($(NF-1)" "$(NF-1))}' | \
    sort -u > dump/raw/<subset-name>/utt2spk
find audios/ -iname '*.flac' | awk -F'[/.]' '{print($(NF-1)" "$(NF-1))}' | \
    sort -u > dump/raw/<subset-name>/spk2utt
python -c '
import soundfile as sf
with open("dump/raw/<subset-name>/utt2fs", "w") as f1:
    with open("dump/raw/<subset-name>/utt2category", "w") as f2:
        with open("dump/raw/<subset-name>/wav.scp", "r") as f3:
            for line in f3:
                uid, path = line.strip().split(maxsplit=1)
                info = sf.info(path)
                f1.write(f"{uid} {info.samplerate}\n")
                f2.write(f"{uid} {info.channels}ch_{info.samplerate}Hz\n")
'
```

---

# Task definition

The speech enhancement task <a href="https://github.com/espnet/espnet/tree/master/espnet2/tasks/enh.py" style="text-decoration:none;">`espnet2.tasks.enh.EnhancementTask`</a> is a high-level class used for model training as well as for loading pre-trained models.

It generally defines the supported

* .darkred[model architectures] (encoder_choices, separator_choices, decoder_choices)

* .darkred[loss functions] (loss_wrapper_choices, criterion_choices)

* The wrapper is used mainly for handling the permutation problem in speech separation tasks.

* For single-speaker tasks, please simply use `wrapper: fixed_order`.

* .darkred[preprocessing] (preprocessor_choices)

* used for preprocessing each input sample during both training and inference stages

* e.g., for data augmentation / dynamic mixing, and normalization

* and the .darkred[arguments] used for training.

---

# Task definition (Cont'd)

To use a pre-trained speech enhancement model for enhancing audios, you can run the following script:

<d-code block language="python">
import soundfile as sf
from espnet2.bin.enh_inference import SeparateSpeech
<br/>
model = SeparateSpeech(
    train_config="exp/xxx/config.yaml",
    model_file="exp/xx/valid.loss.best.pth",
    normalize_output_wav=True,
    device="cuda",
)
<br/>
audio, fs = sf.read("/path/to/noisy/utt1.flac")
enhanced = model(audio[None, :], fs=fs)[0]
</d-code>

---

# Model definition

All speech enhancement models generally share the same template defined in <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/espnet_model.py" style="text-decoration:none;">`espnet2.enh.espnet_model.ESPnetEnhancementModel`</a>, which consists of three modules: <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/encoder" style="text-decoration:none">encoder</a> → <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/separator" style="text-decoration:none">separator</a> → <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/decoder" style="text-decoration:none">decoder</a>.

The main enhancement function is achieved by the separator module, which can be of various architectures, e.g., <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/separator/bsrnn_separator.py" style="text-decoration:none;">BSRNN</a>, <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/separator/tfgridnetv3_separator.py" style="text-decoration:none;">TF-GridNet</a>, and so on.

To add your own model, you can follow the instructions in <a href="https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/enh1/README.md#instructions-on-creating-a-new-model" style="text-decoration:none;">egs2/TEMPLATE/enh1/README.md</a>.

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**1.** Basic hyperparameters (1/2)

```yaml
max_epoch: 100              # Max training epochs
batch_type: folded          # See espnet2/samplers/build_batch_sampler.py
batch_size: 4               # Batch size (number of chunks per minibatch when using the chunk iterator)
iterator_type: chunk        # Using chunk-based iterator for data loading
chunk_length: 200           # Each training sample will be segmented into overlapped 4-sec (= 200 / 50) chunks
chunk_default_fs: 50        # Used for automatically scaling the chunk length based on the input sampling rate
num_iters_per_epoch: 8000   # Number of samples per epoch (each sample can generate >1 chunks when using the chunk iterator)
num_workers: 4              # Number of parallel workers for data loading
grad_clip: 5.0              # Gradient clipping threshold
optim: adam                 # Optimizer type (See espnet2/tasks/abs_task.py)
optim_conf:                 # Optimizer configuration
    lr: 1.0e-03             # See the docstring of torch.optim.Adam
    eps: 1.0e-08
    weight_decay: 1.0e-05
patience: 40                # Patience for early stopping. If the validation performance does not improve for 40 consecutive epochs, the training will be ended.
```

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**1.** Basic hyperparameters (2/2)

```yaml
val_scheduler_criterion:    # Validation scheduler configuration. Used to collect stats for certain epoch-based optimizers that require the validation performance per epoch.
- valid                     # Name of the phase for stats recording. Can be 'train' or 'valid'.
- loss                      # Name of the stats. Can be any key name in the `stats` dict returned by ESPnetEnhancementModel.forward in espnet2/enh/espnet_model.py.
best_model_criterion:       # Configuration of recording stats for determining the best-performing checkpoint.
-   - valid                 # Name 1 of the phase for stats recording. Can be 'train' or 'valid'.
    - loss                  # Name of the stat. Same as above.
    - min                   # The best model 1 is determined by the minimum value of the stat. Can be 'min' or 'max'.
-   - valid                 # Name 2 of the phase for stats recording. Can be 'train' or 'valid'.
    - acc                   # Name of the stat. Same as above.
    - max                   # The best model 2 is determined by the maximum value of the stat. Can be 'min' or 'max'.
keep_nbest_models: 1        # Number of best models to keep. The best model is determined by `best_model_criterion`.
scheduler: steplr           # Scheduler type. See scheduler_classes in espnet2/tasks/abs_task.py.
scheduler_conf:             # Scheduler configuration
    step_size: 2            # See the docstring of torch.optim.lr_scheduler.StepLR
    gamma: 0.99
allow_multi_rates: true     # Whether to allow loading audios of different sampling rates (If true, special treatment is required in the preprocessor to make sure data of different sampling rates are grouped in different categories (thus minibatches))
```

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**2.** Preprocessor configuration

```yaml
preprocessor: enh           # Preprocessor type. See preprocessor_choices in espnet2/tasks/enh.py. The 'enh' preprocessor allows automatic processing of different sampling rates based on the utt2fs and utt2category files.
force_single_channel: true  # Whether to force the input audio to be single-channel in the preprocessor.
channel_reordering: true    # Whether to reorder the multiple channels of the input audio in the preprocessor.
categories:                 # List of all possible categories used in dump/raw/*/utt2category files. This is used by the preprocessor to group samples according to their category.
- 1ch_8000Hz                # For this challenge, the category format can be simply '1ch_{fs}Hz'.
- 1ch_16000Hz
- 1ch_22050Hz
- 1ch_24000Hz
- 1ch_32000Hz
- 1ch_44100Hz
- 1ch_48000Hz
num_spk: 1                  # Number of speakers in the input audio. Should be 1 for single-speaker tasks.
```

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**3.** Model configuration (1/2)

```yaml
model_conf:                 # Configuration for espnet2/enh/espnet_model.ESPnetEnhancementModel
    normalize_variance_per_ch: true     # Whether to normalize the variance of (each channel of) the input speech
    always_forward_in_48k: false        # Whether to always upsample the input speech to 48 kHz for model processing. The model output will be downsampled back to its original sampling rate.
                                        # When processing data of different sampling rates, this must be true for speech enhancement models that only support one sampling rate.
                                        # For sampling-frequency-independent (SFI) models such as BSRNN and TF-GridNet-v3, this can be false for more efficient processing.
    categories:             # Must have the same value as in the preprocessor config (see last slide).
    - 1ch_8000Hz
    - 1ch_16000Hz
    - 1ch_22050Hz
    - 1ch_24000Hz
    - 1ch_32000Hz
    - 1ch_44100Hz
    - 1ch_48000Hz
```

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**3.** Model configuration (2/2)

```yaml
encoder: stft               # Encoder type. See encoder_choices in espnet2/tasks/enh.py.
encoder_conf:               # Encoder configuration.
    n_fft: 960              # FFT size for each window.
    hop_length: 480         # Hop size for the sliding window.
    use_builtin_complex: true  # Whether to use PyTorch's builtin complex tensor type.
    default_fs: 48000       # Used for automatically adjusting the STFT window/hop sizes based on the input sampling rate.
                            # This should be used together with frequency-domain SFI models such as BSRNN and TF-GridNet-v3.
decoder: stft               # Decoder type. See decoder_choices in espnet2/tasks/enh.py.
decoder_conf:               # Decoder configuration.
    n_fft: 960              # Same as above. Usually should have the same config as in encoder_conf.
    hop_length: 480
    default_fs: 48000
separator: bsrnn            # Separator type. See separator_choices in espnet2/tasks/enh.py.
separator_conf:             # Separator configuration.
    num_spk: 1              # Number of speakers to separate. Should be 1 for single-speaker tasks.
    ...                     # See espnet2/enh/separator/bsrnn_separator.py.
```

---

# Model configuration

The configuration (YAML) file is defined in <a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/conf/tuning" style="text-decoration:none;">`conf/tuning/`</a>.

A typical config file consists of four parts:

**4.** Training criterion configuration

```yaml
criterions:                 # Training criterion configuration.
  - name: mr_l1_tfd         # The first criterion. See criterion_choices in espnet2/tasks/enh.py.
    conf:                   # Criterion configuration.
      window_sz: [256, 512, 768, 1024]  # See MultiResL1SpecLoss in espnet2/enh/loss/criterions/time_domain.py.
      hop_sz: null
      eps: 1.0e-8
      time_domain_weight: 0.5
      normalize_variance: true
    wrapper: fixed_order    # Wrapper type for handling the permutation problem. See loss_wrapper_choices in espnet2/tasks/enh.py.
    wrapper_conf:           # Wrapper configuration.
      weight: 1.0           # Loss weight for the first criterion. Can be used for balancing multiple criteria.
  - name: si_snr            # The second criterion.
    conf:                   # Criterion configuration.
      ...                   # See SISNRLoss in espnet2/enh/loss/criterions/time_domain.py.
```

---

# Script flow of training and inference

**Training:**

<a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none;">run.sh</a> → <a href="https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/enh1/enh.sh" style="text-decoration:none;">enh.sh</a> (stage 6) → <a href="https://github.com/espnet/espnet/blob/master/espnet2/bin/enh_train.py" style="text-decoration:none;">espnet2/bin/enh_train.py</a><br/>
→ <a href="https://github.com/espnet/espnet/blob/master/espnet2/tasks/enh.py" style="text-decoration:none;">espnet2/tasks/enh.py</a> `ESPnetEnhancementTask` &<br/>
<a href="https://github.com/espnet/espnet/blob/master/espnet2/tasks/abs_task.py" style="text-decoration:none;">espnet2/tasks/abs_task.py</a> `AbsTask.main`<br/>
→ <a href="https://github.com/espnet/espnet/blob/master/espnet2/train/trainer.py" style="text-decoration:none;">espnet2/train/trainer.py</a> `Trainer.run`<br/>
*  → <a href="https://github.com/espnet/espnet/blob/master/espnet2/train/dataset.py" style="text-decoration:none;">espnet2/train/dataset.py</a> `ESPnetDataset`<br/>
*  → <a href="https://github.com/espnet/espnet/blob/master/espnet2/train/preprocessor.py" style="text-decoration:none;">espnet2/train/preprocessor.py</a> `EnhPreprocessor` (If used)<br/>
*  → <a href="https://github.com/espnet/espnet/blob/master/espnet2/iterators/chunk_iter_factory.py" style="text-decoration:none;">espnet2/iterators/chunk_iter_factory.py</a> `ChunkIterFactory.build_iter` (If used)<br/>

→ <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/espnet_model.py" style="text-decoration:none;">espnet2/enh/espnet_model.py</a> `ESPnetEnhancementModel.forward`<br/>
* → <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/encoder/stft_encoder.py" style="text-decoration:none;">espnet2/enh/encoder/stft_encoder.py</a> `STFTEncoder.forward` (If used)<br/>
* → <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/separator/bsrnn_separator.py" style="text-decoration:none;">espnet2/enh/separator/bsrnn_separator.py</a> `BSRNNSeparator.forward` (If used)<br/>
* → <a href="https://github.com/espnet/espnet/tree/master/espnet2/enh/decoder/stft_decoder.py" style="text-decoration:none;">espnet2/enh/decoder/stft_decoder.py</a> `STFTDecoder.forward` (If used)<br/>
* → <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/wrappers/fixed_order.py" style="text-decoration:none;">espnet2/enh/loss/wrappers/fixed_order.py</a> `FixedOrderSolver.forward` (If used)<br/>
* → <a href="https://github.com/espnet/espnet/blob/master/espnet2/enh/loss/criterions/time_domain.py" style="text-decoration:none;">espnet2/enh/loss/criterions/time_domain.py</a> `SISNRLoss.forward` (If used)<br/>

---

# Script flow of training and inference (Cont'd)

**Inference:**

<a href="https://github.com/espnet/espnet/blob/master/egs2/urgent24/enh1/run.sh" style="text-decoration:none;">run.sh</a> → <a href="https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/enh1/enh.sh" style="text-decoration:none;">enh.sh</a> (stage 7) → <a href="https://github.com/espnet/espnet/blob/master/espnet2/bin/enh_inference.py" style="text-decoration:none;">espnet2/bin/enh_inference.py</a> `SeparateSpeech`<br/>
* → <a href="https://github.com/espnet/espnet/blob/master/espnet2/tasks/enh.py" style="text-decoration:none;">espnet2/tasks/enh.py</a> `ESPnetEnhancementTask` &<br/>
<a href="https://github.com/espnet/espnet/blob/master/espnet2/tasks/abs_task.py" style="text-decoration:none;">espnet2/tasks/abs_task.py</a> `AbsTask.build_model_from_file`<br/>
* → <a href="https://github.com/espnet/espnet/blob/master/espnet2/tasks/enh.py" style="text-decoration:none;">espnet2/tasks/enh.py</a> `ESPnetEnhancementTask.build_streaming_iterator`<br/>
    *  → <a href="https://github.com/espnet/espnet/blob/master/espnet2/train/iterable_dataset.py" style="text-decoration:none;">espnet2/train/iterable_dataset.py</a> `IterableESPnetDataset`<br/>
    *  → <a href="https://github.com/espnet/espnet/blob/master/espnet2/train/preprocessor.py" style="text-decoration:none;">espnet2/train/preprocessor.py</a> `EnhPreprocessor` (If used)<br/>