porhan commited on
Commit
919a76d
·
verified ·
1 Parent(s): 713909e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -2
README.md CHANGED
@@ -19,10 +19,12 @@ This work was granted access to the HPC resources of IDRIS under the allocations
19
 
20
  Models currently available are:
21
 
22
- - Wav2vec2 base model () pretrained (no fine-tuning) on Librispeech (English speech), FMA (music), subset of audioset, or all of them together. It also includes a model pretrained on VoxPopuli french dataset.
23
  - Wav2vec2 tiny model, where we used only 3 transformer layers. Models' performances are surprisingly high.
24
 
25
  Scientific papers using the models provided in this repository:
26
  Orhan, P., Boubenec, Y., & King, J.-R. (2024). Algebraic structures emerge from the self-supervised learning of natural sounds. https://doi.org/10.1101/2024.03.13.584776
27
 
28
- Models are pretrained using HuggingFace's trainer.
 
 
 
19
 
20
  Models currently available are:
21
 
22
+ - Wav2vec2 base model (https://huggingface.co/facebook/wav2vec2-base), but pretrained (no fine-tuning) on Librispeech (English speech), FMA (music), subset of audioset, or all of them together. It also includes a model pretrained on VoxPopuli french dataset.
23
  - Wav2vec2 tiny model, where we used only 3 transformer layers. Models' performances are surprisingly high.
24
 
25
  Scientific papers using the models provided in this repository:
26
  Orhan, P., Boubenec, Y., & King, J.-R. (2024). Algebraic structures emerge from the self-supervised learning of natural sounds. https://doi.org/10.1101/2024.03.13.584776
27
 
28
+ Models are pretrained using HuggingFace's trainer.
29
+ These models pretraining are often shorter (100,000 steps compared to 400 000) than original pretraining because of resource scarcity.
30
+ In my experience, most emergences I studied had happened before 100 000 steps.