large-v3 release

[jongwook]

We're pleased to announce the latest iteration of Whisper, called large-v3. Whisper-v3 has the same architecture as the previous large models except the following minor differences:

• The input uses 128 Mel frequency bins instead of 80
• A new language token for Cantonese

The large-v3 model is trained on 1 million hours of weakly labeled audio and 4 million hours of pseudolabeled audio collected using large-v2. The model was trained for 2.0 epochs over this mixture dataset.

The large-v3 model shows improved performance over a wide variety of languages, and the plot below includes all languages where Whisper large-v3 performs lower than 60% error rate on Common Voice 15 and Fleurs, showing 10% to 20% reduction of errors compared to large-v2:

• Languages evaluated using character error rates (CERs) instead of word error rates (WERs) are shown in Italic.
  • We used character error rates for Korean as well, in addition to the five languages that we used CERs in the paper (Chinese, Japanese, Thai, Lao, and Myanmar). While Korean does use spaces to separate words, there are many cases where it is acceptable to omit spaces between words, and we noticed that the labels in both Common Voice 15 and Fleurs have many instances of inconsistent or incorrect spacings.
• In the Fleurs dataset, we used the transcription column which contains labels that are pre-processed and normalized from the raw_transcription column, with these two exceptions:
  • Korean labels have many phrases inside parentheses that usually repeats the preceding word in the Latin script; we removed these using regular expressions.
  • Serbian labels contain transcriptions in both Latin and Cyrillic scripts, and Whisper's predictions for Serbian also often fluctuate between the two scripts. For evaluation, we converted both the labels and the model predictions into Cyrillic and computed the word error rates.
• In addition, we converted all Chinese texts into Simplified Chinese for evaluation, to measure the speech recognition performance without being penalized for outputting in Traditional Chinese when the labels are in Simplified Chinese and vice versa. In practice, the model can be steered to produce outputs in either form, as shown in this example. (The same trick should apply for Serbian)

---

The large-v3 model is available in openai-whisper==20231106 and after. To use large-v3, please update the Whisper package using the following command:

pip install -U openai-whisper

and load the model using the name "large-v3". The name "large" now aliases to the latest model in the series, "large-v3".

import whisper

model = whisper.load_model("large-v3")

[Root-FTW]

omg THANK YOU gwook

[uanandaraja]

let's goo

[spacenavy90]

Incredible work

[vidalfer]

Let's gooooo

[oenu]

Extremely cool, interesting to see Dutch as a high accuracy language. Wonder if it's the commonality with English mixed with the fact it hasn't been modified so many times by Roman, Viking, Frankish and Norman conquerors 🧐

Analysing high error rate words and categorising them by their ancient root could be a fun undergrad paper for someone in computer science with an interest in linguistics (or an avenue on which to focus future work)

[MarktHart]

v3 performs consistently worse than v2 on both public and private dutch datasets that I've tested it with. So I suspect there was testset leakage on CV dutch

[dgoryeo]

Congartulations @jongwook and team. Your great work is much appreciated!

[arabcoders]

Awesome @jongwook looking forward to trying it,

Edit: i was trying to convert the model into ct2 but it's fails as the large-v3 repository doesn't seems to exists or set to private in huggingface, is there plans to open the model on HF like large-v2?

[arabcoders]

@jongwook Thanks. It does seems the VAD probably making it generate those untranslated lines. I'll take closer look at later time and updated you about it. Thank you so much for your work.

@haiderasad I managed to convert it to ct2 however, i am unable to run it due to changes into mel and i really don't have huge experience in python and machine learning. so it will probably take a while.

[BBC-Esq]

I think that someone is working on it at this link...I'm waiting patiently as well:

OpenNMT/CTranslate2#1530

[AIhasArrived]

Wow thats's quite the command, mine is just:
whisper "video.mkv" --language en --model base.en --device cuda --task transcribe,
what is that last part of your command ". .aac"?

[arabcoders]

That's the audio file much like your "video.mkv"

[AIhasArrived]

What about the other parameters? Are they described somewhere maybe?

[bofenghuang]

@jongwook Congrats to the team on this exciting new release! With respect to the pseudolabeled data, were they treated the same as the weakly labeled data during training, or did they carry a lower weight? Are there any plans for a new paper for this release? Thanks!

[EmanuelCampos]

That's cool, what about streaming?

[zer0int]

See here

[martinkallstrom]

That link as about streaming TTS, not Whisper.

[billycasper]

한국인 짱

[liufei1656]

That's so cool. Great Job!!!

[abdulnim]

Thats a great news. will it be available on hugging face?

[jongwook]

https://huggingface.co/openai/whisper-large-v3 😄

[abdulnim]

감사합니다

[funboarder13920]

Is it possible to release a pytorch_model.bin and a fasttokenizer config tokenizer.json, in order to have a repo similar to the large-v2 one ?
I am mainly interested in these two files but people might also be interested in the other formats that are available on the hub in large-v2 but not in large-v3 yet.
Thanks for your work.

[funboarder13920]

In gardjitta/versae repo the tokenizer.json does not seem accurate, "yue" is missing in the added tokens. I don't know if it is an issue or not. I don't know weither it is the only error.
Therefore I would like to start working from files from the official release.

[bobqianic]

I'm interested in the approach your team took with Whisper-v3, particularly in the use of n_mel=128. Could you share the thought process behind choosing this over n_mel=80? Also, I'm quite intrigued about the decision to incorporate 1 million hours of weakly labeled audio and 4 million hours of pseudolabeled audio in your training data. Could you shed some light on the advantages of using pseudolabeled audio, as opposed to starting the training entirely from weakly labeled audio? Thank you! @jongwook

[Artyom17]

Are there plans to make new versions of other models to match large-v3 in terms of number of supported languages (100 in v3 vs 99 in others)?

[NPap0]

Thank you!

[aflyingnoob]

you are not closeai

[Headscracher9]

Hi guys, I'm a little new here.
is it possible for someone to give an example code for a simple audio file output from text = ""Hi, how are you?" ?
I use a pytorch for CPU without conda.

[Purfview]

This repo is opposite, it outputs text from audio, not audio from text.

[rounakdatta]

Check out OpenAI's new TTS APIs.

[aehlke]

https://github.com/yl4579/StyleTTS2

[HarikalarKutusu]

A question: How did you evaluate against Common Voice and/or Fleurs? Did you just take the test set or was it the whole validated?

[iamalin]

yes, need more detail information.

[HarikalarKutusu]

To scientifically replicate the results, more transparency on the practices is needed. E.g. a detailed explanation about the practice, open-sourced test code, if the dataset is pre-processed (e.g. what splits are used, are the sentences normalized or not, converted to lowercase or used as is, if any offending recordings are removed, etc,), and if the last one is true, the new dataset should also be published separately.

If we are guessing wild, nobody will get the same or similar results, so we will not have a baseline for our fine-tuning work.

[danielmerja]

It performs so bad with Albanian! Wish we could help on that and preserve more languages that are not commonly used these days.

[HarikalarKutusu]

Hey @sanchit-gandhi, I've started Whisper with your beautiful post and used it to create fine-tuned models using many Common Voice languages, especially Turkish and other Turkic languages. On the other hand, the accuracy depends on many things:

• Amount of data in the pre-trained model
• Model size === parameter count (obviously)
• Data size and dataset quality during fine-tuning (obviously)
• The script of the language (also marked in the paper)

So:

• If a language is not in the pre-trained multilingual model, you'll be out of luck
• If the pre-trained model does not have enough data from that language, you'll be out of luck
• If the "medium" model does not perform enough, you'll be out of luck (a local 24GB VRAM GPU cannot handle large models), also inference will be hard to achieve...

AI is advancing fast, but they do spit out new AI models, instead of creating new releases to existing ones. Most of us need better-performing tiny/base/small/medium models, which can only be achieved with new data & versions.

[AIhasArrived]

That's pretty cool @HarikalarKutusu , would you considermaking a video explainin thjeprocess step by step please? or some text document for less experimented? (I know git clone, make venv, install requirement, but I am fraid this fine tuning thing to be too exomplicated for me and need some little help)

[HarikalarKutusu]

@AIhasArrived: Please start with @sanchit-gandhi's post, try to replicate it for your language (which one?), and get help from the HuggingFace forums. We are all there.
Also, there are a lot of video tutorials already made, here.

[LYPinASR]

I used v3 to fine-tune Fleurs' af 'and' kk 'with completely different results. After fine-tuning "af", v3 was able to make a big improvement, but during fine-tuning "kk", the validation set WER even exceeded 200%, which was completely missing when I fine-tuned v2. Why do different languages have different situations for the same code?

[HarikalarKutusu]

@LYPinASR, what you explain seems like over-fitting... Low amount of data, thus not much to learn.

[ThioJoe]

In English, V3 is unusable for me. I basically never got hallucinations in V2 except one time at the end it added "[Translated by ...]" or something. But V3 hallucinates multiple times per 10 minutes.

[zml25904]

The sample notebook LibriSpeech.ipynb cannot run with large-v3 model
It always shows "RuntimeError: Given groups=1, weight of size [1280, 128, 3], expected input[16, 80, 3000] to have 128 channels, but got 80 channels instead" when decoding.

[Swami-Abhinav]

preprocessor_config.json must be present where you are loading the model from, if not download it from here and paste it wherever your model is loaded from

[iamalin]

Great job!!!, I plan to fine-tune based on different regional accents in China. Before that, I would like to repeat the current benchmark test. Could you please provide the test code?

[LYPinASR]

When I evaluate whisper-large-v3 in Fleurs for "afkk, mr, ne, sw", I find that the results I get are not consistent with the official release, and they are very different. Why?

[LYPinASR]

af kk mr ne sw
32.4 32.4 34.1 40.2 34.1
31.67 32.85 42.43 51.57 34.31
As shown above, the first row is the official release WER, the second row is the WER of my own evaluation. The results of af,kk,sw are roughly the same as the official release, but mr and ne are much worse.
Here's the code for my evaluation:

import torch
from evaluate import load
from transformers import WhisperProcessor, WhisperForConditionalGeneration, WhisperTokenizer
from datasets import load_dataset, load_from_disk, Audio

model_id = 'whisper-large-v3'
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda")
languages = ["af", "kk", "mr", "ne", "sw"]
for language in languages:
data = load_from_disk('fleurs/{}'.format(language))
data = data.cast_column("audio", Audio(sampling_rate=16000))
test_data = data["test"]

processor = WhisperProcessor.from_pretrained(model_id, language=language, task="transcribe")
tokenizer = WhisperTokenizer.from_pretrained(model_id, language=language, task="transcribe")

model.config.forced_decoder_ids = tokenizer.get_decoder_prompt_ids(language=language, task="transcribe")
model.generation_config.forced_decoder_ids = model.config.forced_decoder_ids

def map_to_pred(batch):
    audio = batch["audio"]
    input_features = processor(audio["array"], 
                               sampling_rate=audio["sampling_rate"], 
                               return_tensors="pt").input_features
    batch["reference"] = processor.tokenizer._normalize(batch['transcription'])

    with torch.no_grad():
        predicted_ids = model.generate(input_features.to("cuda"))[0]
    transcription = processor.decode(predicted_ids)
    batch["prediction"] = processor.tokenizer._normalize(transcription)
    return batch

result = test_data.map(map_to_pred)

wer = load("wer")
cer = load("cer")
print("wer = ", 100 * wer.compute(references=result["reference"], predictions=result["prediction"]), "cer = ", 100 * cer.compute(references=result["reference"], predictions=result["prediction"]))

[vernicifluum]

large-v3 is bad.

Whisper-v3 Hallucinations on Real World Data
https://deepgram.com/learn/whisper-v3-results

[huangruizhe]

That's a good article. Thanks for sharing your benchmark results!
In the benchmark, you showed Whisper-v3 has off-the-chart bad WERs -- that's interesting. It would be great there can be the WER breakdowns into insertion, deletion and substitution errors.
In my own experience, such large WER may be due to a lot of "ghost hallucinations" -- which can lead to a lot of insertion errors (e.g., repeating something 100 times). And the insertion errors can easily increase the overall WER in an unbounded way.

[avan06]

In my own transcription experience with large-v3 (for Japanese), the most common occurrence of hallucinations tends to be in silent or music-only sections.

After multiple transcriptions, I observed certain characteristics in the attribute values of duration, segment_last, avg_logprob, compression_ratio, and no_speech_prob from the transcription results. The following features may indicate hallucinated results:

• duration: calculated by subtracting start from end, it might indicate hallucinated results when inversely proportional to text length.
• segment_last: the last result for each segment during VAD transcription has a certain probability of being a hallucinated result.
• avg_logprob: average log probability, ranging from logprob_threshold (default: -1) to 0, is better when a larger value. A value lower than -0.9 might suggest a poor result.
• compression_ratio: gzip compression ratio, ranging from 0 to compression_ratio_threshold (default: 2.4), a higher positive value is preferable. If it is lower than 0.9, it might indicate suboptimal results.
• no_speech_prob: no_speech(<|nospeech|> token) probability, ranging from 0 to no_speech_threshold (default: 0.6), a smaller positive value is preferable. If it exceeds 0.1, it might suggest suboptimal results.

And I have set several conditions that I believe can filter out hallucinated results:

1. avg_logprob < -0.9
2. (durationLen < 1.5 || segment_last), textLen > 5, avg_logprob < -0.4, no_speech_prob > 0.5
3. (durationLen < 1.5 || segment_last), textLen > 5, avg_logprob < -0.4, no_speech_prob > 0.07, compression_ratio < 0.9
4. (durationLen < 1.5 || segment_last), compression_ratio < 0.9, no_speech_prob > 0.1

---

Due to the limited number of tests, these filtering conditions may not be applicable to everyone. However, they can be provided for reference. Additionally, I have implemented the aforementioned filtering functionality in the whisper-webui-translate spaces on Hugging Face.

Note 1: This spaces is built based on the aadnk/whisper-webui version.
Note 2: The filtering conditions will only be activated when the Whisper Segments Filter options in the Whisper Segments Filter are checked.
Note 3: The 'whisper' in whisper-webui-translate is a model transformed through CTranslate2, theoretically producing the same results as the original version.

[arabcoders]

Thank you

[dgoryeo]

Hi @avan06 , thanks for the insight. In your experience, are those threshold values the same for the large-v2 model? Do you see any benefits in using large-v3 (after applying the hallucination filters)?

[arabcoders]

@avan06, I've found that using #1903 changes on webui ctranslate2 backed project significantly reduces hallucinations, i am mainly using it to translate Japanese's audio as well. Before this patch i would get similar results like the one you described.

If you have experience does this work similar to your patch or yours better?

[avan06]

Hi @dgoryeo,

1. In my experience, these threshold values are most suitable for the large-v3 model. For example, in a 45-minute video, these filtering conditions would detect only 1-3 hallucinations in large-v2, while in large-v3, it might detect 5-10 or even more hallucinations.
2. After removing hallucinations, I don't think large-v3 performs worse than large-v2, but I can't definitively say which one is better.

[avan06]

Hi @arabcoders, I'm not entirely clear on the purpose of the decode_with_fallback function in Whisper. From a brief look, it seems to iterate through decoding results with temperatures ranging from low to high, given the temperature hyperparameter. It continues this loop until it determines that needs_fallback is false. This functionality appears to be different from the approach I used for filtering.

[Swami-Abhinav]

Hi, I was calculating the word error rates and I got around ~37% in hindi for large-v3 model but here it shows 16.9% in the dataset that openai used in announcements, so is that the normalized word error rate or actual word error rate that you guys have mentioned in the announcements?

[HarikalarKutusu]

Please see this: #1762 (comment)

[zhuangweiji]

The large-v3 model is trained on 1 million hours of weakly labeled audio and 4 million hours of pseudolabeled audio collected using large-v2. The model was trained for 2.0 epochs over this mixture dataset.

What's the difference between "weakly labeled audio" and "pseudolabeled audio"?

[HarikalarKutusu]

I think it worked like this:

Suppose you get your hand on the whole set of YouTube videos and extract the audio from them. Some of these have subtitles, but you are not sure if these subtitles are of quality.

Weakly labeled audio: Those videos' audio with subtitles. You don't make quality control, you assume they are correct.

Then you create a model using them and use this model to transcribe those videos which do not have any subtitles. These are called pseudo labelled audio.

Then you generate another model using the whole set (or fine tune the first one, I don't know).

This is semi-supervised learning and this is why the models perform worse out of the box - than those models created fully using supervised learning which use quality datasets. If you could quality-label 5 Mh of audio, the models would thrive, but think of the man-hours required and the related budget. Instead you get a less than perfect model, which can be fine-tuned with quality datasets for a language.

[hclsys]

The presence of unrelated text during transcriptions is not due to hallucination, or at least much different from the word used in LLM.

It's just because OpenAI uses a data crawler to extract (audio,subtitle) from websites like YouTube and other video sharing sites that provide subtitles. Contributors supply these subtitles along with notes crediting their contribution, for example, 'contributed by ... community'. However, these notes aren't accurately paired with the corresponding audio content. The notes aren't removed from the transcript. Hence, the end-to-end ASR, whether it's based on transformers trained by CTC or attention, learns to associate the silence at the end of the audio data with these contributor notes. This is why, during the inference stage, silence tends to be transcribed as these contribution notes.

[HarikalarKutusu]

Exactly, this is the result of "weak supervision". Dataset quality is everything.

[SIlver--]

@HarikalarKutusu as said in the whitepaper though, having non golden data helps improves accuracy because audio capture in the real world is never perfect. I imagine this issue is better fixed in the code rather than from the data.

[hkrutzer]

You want to get perfect transcription from imperfect audio. So while the audio shouldn't be perfect, the transcription should be.

[HarikalarKutusu]

@SIlver-- this is the whole idea behind SOTA methodology for every actor out there - because they are racing. They dump in anything they can find, use huge infrastructure, train for very long times, maximize the carbon print.

It helps, but up to a point. The same results could be achieved with much less properly curated data - and you will not have "hallucinations" as defined above. Whisper even does not support language models (I have nice n-grams which are not useful for it), so that inference can thrive.

And, as @hkrutzer mentioned, the training audio should not be clean (or be augmented), but the transcription must match it and be correct, grammatically and in spelling. This is what we have been trying in Common Voice. Non ideal and diverse audio, but nice transcriptions (hopefully). Just compare Common Voice & Fleurs test results. CV results are much more realistic for real world scenarios.

This is also valid for LLMs, they all use Common Crawl data, but the arxiv is full of papers on its quality.

We should not take quality human effort out of the loop, otherwise what comes in comes out.

[DiwakarBasnet]

Is large-v3 pre trained on common voice and fleurs or they are just evaluated on those dataset?

[HarikalarKutusu]

Training with them AND testing against them would be against science.

I think nobody should use CV, Fleurs, Voxpopuli, Libri* etc in the training of new architectures/pre-trained models. These are better quality, controlled datasets - very good for testing - and also good for finetuning :)

[clort81]

I cannot parse your second paragraph. To what does "these" refer?

[HarikalarKutusu]

@clort81, "these" refer to fine-curated datasets like "Mozilla CommonVoice, Fleurs, Voxpopuli, Libri*".
Please read my other replies which you can find a bit above to understand why...

[clort81]

@clort81, "these" refer to fine-curated datasets like "Mozilla CommonVoice, Fleurs, Voxpopuli, Libri*". Please read my other replies which you can find a bit above to understand why...

So if these are better quality, why should nobody use them?

[HarikalarKutusu]

As I said in the comments above, the "big boys" like OpenAI use weakly/semi-supervised learning, but use huge (I mean HUGE) amounts of data for their multilingual models to get there. This allows us to fine-tune these with good-quality datasets for our special purposes, e.g. dropping WER from 20% to 8% for a specific language.
If they already included the SMALL but quality datasets in the training, we could not fine-tune them, it would be against the underlying science (train twice, train and test with the same data etc).
They already train on 5 million hours, adding 100 hours from a good quality dataset would not move their results much. But we can fine-tune and get much better results if they don't include them.
So, I meant, "nobody" === "big_boys", not us...

[margo2130]

We're pleased to announce the latest iteration of Whisper, called large-v3. Whisper-v3 has the same architecture as the previous large models except the following minor differences:

* The input uses 128 Mel frequency bins instead of 80

* A new language token for Cantonese

The large-v3 model is trained on 1 million hours of weakly labeled audio and 4 million hours of pseudolabeled audio collected using large-v2. The model was trained for 2.0 epochs over this mixture dataset.

The large-v3 model shows improved performance over a wide variety of languages, and the plot below includes all languages where Whisper large-v3 performs lower than 60% error rate on Common Voice 15 and Fleurs, showing 10% to 20% reduction of errors compared to large-v2:

* Languages evaluated using character error rates (CERs) instead of word error rates (WERs) are shown in _Italic_.
  
  * We used character error rates for Korean as well, in addition to the five languages that we used CERs in the paper (Chinese, Japanese, Thai, Lao, and Myanmar). While Korean does use spaces to separate words, there are many cases where it is acceptable to omit spaces between words, and we noticed that the labels in both Common Voice 15 and Fleurs have many instances of inconsistent or incorrect spacings.

* In the Fleurs dataset, we used the `transcription` column which contains labels that are pre-processed and normalized from the `raw_transcription` column, with these two exceptions:
  
  * Korean labels have many phrases inside parentheses that usually repeats the preceding word in the Latin script; we removed these using regular expressions.
  * Serbian labels contain transcriptions in both Latin and Cyrillic scripts, and Whisper's predictions for Serbian also often fluctuate between the two scripts. For evaluation, we converted both the labels and the model predictions into Cyrillic and computed the word error rates.

* In addition, we converted all Chinese texts into Simplified Chinese for evaluation, to measure the speech recognition performance without being penalized for outputting in Traditional Chinese when the labels are in Simplified Chinese and vice versa. In practice, the model can be steered to produce outputs in either form, as shown in [this example](https://github.com/openai/whisper/discussions/277). (The same trick should apply for Serbian)

The large-v3 model is available in openai-whisper==20231106 and after. To use large-v3, please update the Whisper package using the following command:

pip install -U openai-whisper

and load the model using the name "large-v3". The name "large" now aliases to the latest model in the series, "large-v3".

import whisper

model = whisper.load_model("large-v3")

hi, where the model downloaded saved in my computer? i downloaded large model (2.8gb) but i don't know where it is.

[HarikalarKutusu]

That should be under ~/.cache/huggingface/hub with the following structure:

<CACHE_DIR>
├─ <MODELS>      <==
├─ <DATASETS>
├─ <SPACES>

[saiakarsh-augnito]

@jongwook I have a query regarding the performance of large-v3, specifically regarding FLEURS Telugu test dataset. The numbers I got vary quite a bit from the reported numbers by the paper.

this is the code I used for getting the WER:

import os
import whisper
from jiwer import wer
from tqdm import tqdm
from datasets import load_dataset

model = whisper.load_model('large-v3', device="cuda")
dataset = load_dataset("google/fleurs", "te_in", split='test')

refs = []
hyps = []
for item in tqdm(dataset):
    audio_path = os.path.join(os.path.dirname(item['path']), item['audio']['path'])
    result = model.transcribe(audio_path)
    refs.append(item['transcription'])
    hyps.append(result['text'])

print(f'WER: {wer(refs, hyps) * 100:.2f}%')

this is the output I got: WER: 77.75%

but the paper reports a WER of 39.3

Can you please tell me if I made a mistake and why there is a divergence in the performance?