buildfastwithaibuildfastwithai
AI WorkshopsAll blogsPartner with usFDE Program
Unrot mascotGet yourselfupskilled in AIfor free.Get the appNext live workshopFree AI WorkshopLive session, recording includedReserve a seat

Newsletter

Stay ahead

AI tools and tips. No spam.

Share
Back to blogs
Tools
Open Source
Voice AI

WhisperASR : Multilingual Speech Recognition

By Satvik Paramkusham
December 22, 2024
5 min read
Share:
WhisperASR : Multilingual Speech Recognition
Share:

What if Your Innovation Could Shape the Next Era of AI?

Join Gen AI Launch Pad 2024 and bring your ideas to life. Lead the way in building the future of artificial intelligence.

Introduction

In today's world, automatic speech recognition (ASR) has revolutionized accessibility, real-time transcription, and language processing. OpenAI's Whisper, a state-of-the-art ASR model, pushes the boundaries of accuracy and language support, making it a go-to solution for developers and researchers. This blog post provides a comprehensive guide to setting up and leveraging Whisper for multilingual transcription, incorporating essential pre- and post-processing techniques to enhance results.

Here’s what you’ll learn:

  • Setting up the Whisper ASR environment.
  • Using prompts to improve transcription accuracy.
  • Techniques for audio preprocessing and postprocessing.
  • Practical applications in real-world scenarios.

WhisperASR : Multilingual Speech Recognition illustration

Setting Up the Environment

To use Whisper effectively, you need to set up the required dependencies and initialize the tools. Here’s how to do it step by step.

Installing Dependencies

First, ensure you have the necessary libraries for audio processing and Whisper integration. The pydub library is a great tool for handling audio files efficiently.

!pip install pydub
Explanation:

pydub simplifies audio processing tasks like trimming, splitting, and format conversion.

Expected Output:

Installation completes successfully:

Collecting pydub
...
Successfully installed pydub-x.x.x
Use Case:

Install pydub to preprocess audio files, such as trimming silence or converting formats, making them Whisper-ready.

Authenticating with OpenAI API

To use Whisper, you need access to OpenAI’s API. Here’s how you securely authenticate.

from openai import OpenAI
import os

from google.colab import userdata
os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
Explanation:
  1. Environment Variables: The API key is stored securely in environment variables to prevent exposure in code.
  2. OpenAI Client: Initializes a client object to interact with the API.
Expected Output:

No direct output. The client is ready for API calls.

Use Case:

Securely interact with OpenAI models like Whisper for transcription tasks.

Downloading and Preparing Audio Data

For transcription tasks, you need an audio file. Here’s how to download a sample audio dataset.

import urllib

bbq_plans_remote_filepath = "https://cdn.openai.com/API/examples/data/bbq_plans.wav"
bbq_plans_filepath = "bbq_plans.wav"

urllib.request.urlretrieve(bbq_plans_remote_filepath, bbq_plans_filepath)
Explanation:
  1. urllib Library: Downloads the audio file from a URL.
  2. File Path: Saves the file locally for further processing.
Expected Output:
bbq_plans.wav downloaded successfully.
Use Case:

Use this method to prepare audio files for Whisper or any ASR system.

Whisper Transcription Function

Here’s the core function to transcribe audio using Whisper.

def transcribe(audio_filepath, prompt: str) -> str:
    transcript = client.audio.transcriptions.create(
        file=open(audio_filepath, "rb"),
        model="whisper-1",
        prompt=prompt,
    )
    return transcript.text
Explanation:
  1. audio_filepath: Path to the audio file to be transcribed.
  2. Prompt: Contextual hints to improve transcription accuracy.
  3. Output: Returns the transcription as a string.
Example Usage:
transcription = transcribe("bbq_plans.wav", "A conversation about BBQ plans.")
print(transcription)
Expected Output:
"Hi, I was thinking about having a barbecue this weekend."
Use Case:

Use this function for automated transcription tasks in domains like accessibility, journalism, or call centers.

Role of Contextual Prompts

Prompts can significantly enhance transcription quality by providing domain-specific context.

Experiment:

Transcribe the same audio with and without prompts to observe the difference.

Without Prompt:
transcription = transcribe("bbq_plans.wav", "")
print(transcription)

Expected Output:

"Hi, I was thinking about having a barbecue."
With Prompt:
transcription = transcribe("bbq_plans.wav", "A conversation about BBQ plans.")
print(transcription)

Expected Output:

"Hi, I was thinking about having a barbecue this weekend."

Explanation:

The prompt helps Whisper understand the domain-specific vocabulary and structure, improving accuracy.

Audio Preprocessing: Trimming Silence

To enhance transcription accuracy, preprocess the audio to remove silence or noise.

Silence Trimming Function:

from pydub import AudioSegment, silence

def trim_silence(audio_path):
    sound = AudioSegment.from_file(audio_path, format="wav")
    non_silent = silence.detect_nonsilent(sound, min_silence_len=1000, silence_thresh=-40)
    start, end = non_silent[0]
    trimmed_audio = sound[start:end]
    trimmed_audio.export("trimmed_audio.wav", format="wav")
    return "trimmed_audio.wav"
Explanation:
  1. AudioSegment: Loads the audio file.
  2. Silence Detection: Identifies segments with audio activity.
  3. Export: Saves the trimmed audio.
Expected Output:

The output file trimmed_audio.wav contains only the active portion of the audio.

Use Case:

Improves transcription speed and accuracy by focusing on relevant audio segments.

WhisperASR : Multilingual Speech Recognition illustration

Postprocessing: Adding Punctuation

Raw ASR outputs often lack punctuation. Here’s how to enhance readability.

def punctuation_assistant(raw_transcript):
    punctuated = client.text.completions.create(
        model="text-davinci-003",
        prompt=f"Add punctuation: {raw_transcript}",
        temperature=0
    )
    return punctuated.choices[0].text.strip()
Example Usage:
raw = "hi I was thinking about having a barbecue this weekend"
punctuated = punctuation_assistant(raw)
print(punctuated)
Expected Output:
"Hi, I was thinking about having a barbecue this weekend."
Use Case:

Enhances transcripts for readability and usability in official documents or subtitles.

Visualization and Results

  • Audio Waveform: Display before and after trimming silence.
  • Transcription Comparison: Side-by-side results with and without prompts.

Conclusion

We’ve explored:

  1. Setting up and using OpenAI’s Whisper for multilingual transcription.
  2. The significance of preprocessing and postprocessing techniques.
  3. How prompts enhance transcription quality.

Resources

  1. OpenAI Whisper Documentation
  2. PyDub Documentation
  3. Build Fast With AI Whisper Google Colab Documentation

---------------------------------

Stay Updated:- Follow Build Fast with AI pages for all the latest AI updates and resources.

Experts predict 2025 will be the defining year for Gen AI implementation.Want to be ahead of the curve?

Join Build Fast with AI’s Gen AI Launch Pad 2025 - your accelerated path to mastering AI tools and building revolutionary applications.

Satvik Paramkusham
Written by

Satvik Paramkusham

Founder at Build Fast with AI. AI Engineer, educator, and researcher passionate about agentic AI workflows, LLM application architecture, and training the next generation of AI builders.

Share:

Partner with Build Fast with AI

Guest posts, sponsored articles, tool reviews and in-blog CTAs. See what we offer and get in touch.

Partner with us

Topics

  • Claude AI
  • Best AI Models
  • AI Coding Tools
  • AI Agent Frameworks
  • Agentic AI
  • Open-Source LLMs
  • RAG & Vector Databases
  • AI Industry News

Popular guides

  • 100 Best AI Photo Editing Prompts
  • 75 Best ChatGPT Images 2.5 Prompts
  • 100 Best AI Video Prompts
  • Best AI Models 2026, Tested
  • Claude AI Complete Guide
  • What Is MCP?

Build Fast with AI

  • Build Fast with AI
  • AI Workshops
  • Corporate Training
  • FDE Program
  • Partner with Us
  • Contact
© 2026 Intellify Edventures Private Limited. All rights reserved.
    You Might Also Like
    50 Best Diwali AI Photo Prompts 2026 for Gemini, ChatGPT & Nano Banana
    Tools
    50 Best Diwali AI Photo Prompts 2026 for Gemini, ChatGPT & Nano Banana

    Create stunning Diwali photos with 50 copy-paste AI prompts for Gemini, ChatGPT and Nano Banana. Try realistic portraits, couple photos, saree looks, diyas, rangoli and festive posters.

    Best AI Models in October 2026: Tested & Ranked
    Interactive
    Best AI Models in October 2026: Tested & Ranked

    The best AI models in October 2026, tested and ranked: Claude Opus 5.5, GPT-6 Astra, Gemini 3.8 and DeepSeek V4.1. Benchmarks, pricing, and which to use for what.