Local audio transcription with speaker labels in Claude Code

TLDR; Local audio transcription with speaker labels takes two models and a short script, and can run entirely on your Mac with no audio upload and no API key. A speech-to-text model, Parakeet or Whisper, gives the words and their timings. A diarization model called pyannote finds who spoke when. The script joins the two by time. On my M5 Pro, Parakeet ran at about 100 times real time. Claude Code then reads the text and names the speakers.

When to use local audio transcription

• A recording you already have. A client call, a supplier negotiation or a team retro becomes a labelled transcript, then a brief.
• Audio you shouldn’t upload. The recording stays on your machine. The transcript text reaches Anthropic once Claude reads it (what Claude keeps); if the words must stay local too, run the script without Claude.
• The Core ML engines need an Apple Silicon Mac, with macOS 14 or later, Apple’s free Xcode Command Line Tools and 8 GB of memory (16 GB for multi-hour files). The Python path also runs on Linux, but without a CUDA GPU (an NVIDIA graphics card) it’s slow on a recording over an hour.
• Skip it for a live transcript. Apple’s Notes app can record and transcribe in the note itself, in ten languages including Japanese, on an iPhone 12 or later or a Mac with an M1 chip or later.

How local transcription with speaker labels works

Speech recognition (speech-to-text) writes down the words and gives each one a timestamp: the time it starts and ends. Diarization finds who spoke when, not what they said.

Both models run as trained, with no prompt. On the Parakeet path, four runs of a 17-second Apollo 13 radio clip gave byte-identical transcripts. With the models loaded, each took half a second and used no tokens (the units a language model bills by). Joining words to speakers is plain arithmetic on timestamps; a language model would cost tokens on every line and could answer differently each run.

Claude never hears the audio. It works from your request and the transcript, and makes the calls that need context: how many speakers to expect and who each one is. My test: would I accept a different answer on a second run? A word’s timestamp, no. A guess at who “Speaker 2” is, yes, as long as it’s flagged.

// WHAT CODE DOES, WHAT CLAUDE DOES

1. Code

   • Decode the audio once
   • Run the speech and speaker models, no prompt
   • Join words to speakers by timestamp
   • Merge words into lines with timestamps
   • Repeat exactly: four Parakeet runs byte-identical, no tokens

2. Claude

   • Which engine
   • How many speakers
   • Whether to run a second pass
   • Who each speaker is
   • What was decided, who owes what

How to transcribe audio locally with speaker labels

1. Pick the speech-to-text engine: Parakeet or Whisper

Word error rate is the share of words a model gets wrong, so lower is better. On the Open ASR Leaderboard, a public speech-recognition benchmark, the model cards quote averages of 6.34% for Parakeet v3 and 7.83% for Whisper large-v3-turbo.

I timed both on a clean two-speaker clip. Parakeet was faster, and it alone counted the speakers correctly:

// SECONDS FOR A 53-SECOND TWO-SPEAKER CLIP

• Parakeet0.5 sFound both speakers
• Whisper8.4 sFound three speakers

Whisper supports many more languages, including Japanese and Chinese. It also got the proper nouns right on my Apollo clips, so I use it as the second pass when names matter (step 4).

Parakeet and pyannote community-1 are CC-BY-4.0, which allows commercial use with attribution; Whisper uses the MIT licence, which also allows commercial use.

Check: you can name the engine your recording’s language calls for.

2. Run speech recognition and speaker diarization

For the Python path, first accept the pyannote model’s terms on Hugging Face, the site that hosts it. Then create an access token in your Hugging Face account settings and set HF_TOKEN to it. Install the libraries with pip, Python’s installer, and convert the audio with the free ffmpeg tool.

pyannote sends anonymous usage data by default (the model name, file length and speaker count, never the audio); the first line switches that off:

export PYANNOTE_METRICS_ENABLED=0
pip install faster-whisper "pyannote.audio>=4" torch torchaudio
ffmpeg -i call.m4a -ar 16000 -ac 1 call.wav

import os
from faster_whisper import WhisperModel
from pyannote.audio import Pipeline

segments, _ = WhisperModel("large-v3-turbo").transcribe("call.wav", word_timestamps=True)
words = [(w.start, w.end, w.word) for s in segments for w in s.words]

pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-community-1",
                                    token=os.environ["HF_TOKEN"])
result = pipeline("call.wav", num_speakers=2)  # omit num_speakers if unsure
turns = [(t.start, t.end, spk)
         for t, _, spk in result.speaker_diarization.itertracks(yield_label=True)]

On an Apple Silicon Mac, swap faster-whisper for mlx-whisper: it runs the same Whisper model on the Mac’s graphics cores instead of its main processor. Name the model, or it loads a tiny one: mlx_whisper.transcribe("call.wav", path_or_hf_repo="mlx-community/whisper-large-v3-turbo", word_timestamps=True). It returns plain dictionaries: the segments are in result["segments"], each segment’s words in seg["words"], and each word’s start in w["start"].

FluidAudio does the same two jobs in Swift. It runs them through Apple’s Core ML framework. Its models are built for the Neural Engine, the machine-learning unit in the Mac’s chip. AsrManager gives Parakeet’s words; OfflineDiarizerManager gives pyannote’s turns.

The first run downloads about half a gigabyte of models, which takes a few minutes; after that it works offline. Core ML also compiles the models during the first few runs: my first Apollo 13 run took 18 seconds for 17 seconds of audio, and the next file took under a second.

Check: every entry in words has a start and end time, and turns covers the speech.

3. Join words to speakers by timestamp

Each word goes to the turn that covers its midpoint, or the nearest when none does. Consecutive words from one speaker merge into a line.

def speaker_at(t):
    if not turns:
        return "Unknown"
    for start, end, spk in turns:
        if start <= t < end:
            return spk
    return min(turns, key=lambda x: max(x[0] - t, t - x[1], 0))[2]  # nearest

lines = []
for start, end, word in words:
    spk = speaker_at((start + end) / 2)
    if lines and lines[-1][0] == spk:
        lines[-1][2], lines[-1][3] = end, lines[-1][3] + word
    else:
        lines.append([spk, start, end, word])

for spk, start, end, text in lines:
    print(f"[{start:.0f}s - {end:.0f}s] {spk}: {text.strip()}")

Check: run steps 2 and 3 twice and compare. Whisper output may differ: on hard audio its decoder falls back to sampling (picking among likely words by chance). In my tests the Parakeet path in FluidAudio gave byte-identical output on four runs.

4. Rerun with Whisper when names matter

I ran two clips from NASA’s historical sounds page through both engines: Apollo 11’s landing and Apollo 13’s “Houston, we’ve had a problem”. Both are public domain and full of radio static. Whisper got every word. Parakeet got the sentence structure but misheard the names and a few short phrases.

// WHERE PARAKEET MISHEARD

1. Listen (misheard)→ , correct: Houston, uh Tranguility (misheard)→ , correct: Tranquility Base here, the angle (misheard)→ , correct: Eagle has landed.

2. Mrs. (misheard)→ , correct: This is Houston, say again, please.

3. Uh Houston we’ve had a bowl (misheard)→ , correct: a problem.

On radio, neither engine got the speakers right. On Apollo 11 both heard Neil Armstrong (on the Moon) and Charlie Duke (in Houston) as one speaker. The transcript is a draft that someone who was on the call should skim.

Check: you recognise every proper noun. If not, rerun the file through Whisper and keep both transcripts.

5. Have Claude name the speakers

Give Claude both transcripts and this prompt:

whisper.txt and parakeet.txt are two transcripts of one recording.
Take the words from whisper.txt and the speaker turns from parakeet.txt.
Name each speaker from what they say and how others address them,
and mark every name you inferred. Don't invent timestamps.

On Apollo 13, Claude produced:

Swigert:          Okay, Houston, we've had a problem here.
Lousma (Houston): This is Houston, say again, please.
Lovell:           Houston, we've had a problem. We've had a main B bus undervolt.
Lousma (Houston): Roger, main B undervolt.
Lousma (Houston): Okay, stand by, 13, we're looking at it.

Claude got this right for two reasons. The Apollo 13 exchange is famous and widely published, so Claude may well recognise it. And “This is Houston” marks the capsule communicator, the one person in Houston who talks to the crew. On a client call, Claude has only the names people use for each other and any document you give it, such as the meeting invite, so every name is a guess to check.

Claude dropped the timestamps rather than invent them: timing is the code’s job.

Lining up two transcripts is a join too, and Claude can get it wrong. The Python path in steps 2 and 3 avoids that: code joins Whisper’s words to pyannote’s turns, so Claude gets one labelled transcript and only names the speakers.

Check: every name traces to something said in the recording or a document you gave Claude.

6. Hand the routine to Claude Code

Save steps 2 and 3 as one script, scripts/transcribe.py, that takes the audio file and an optional --num-speakers. Then write down the rules for Claude’s choices. Put them in CLAUDE.md, the file Claude Code reads at the start of every session, or in a skill, which it loads only when a task needs it:

## Transcripts
- Run scripts/transcribe.py <file>. Never edit its timestamps.
- A language outside Parakeet's 25: check Whisper supports it, use Whisper, and say so.
- Pass --num-speakers only when I say how many people spoke.
- Garbled names, or I say names matter: rerun with Whisper.

Once the transcript is in the session, your next prompt can be “summarise this and list who owes what”.

Check: ask for a transcript of “my call with a client, just the two of us”, and Claude passes --num-speakers 2 unprompted.

Pitfalls

In practice

In the small businesses I work with, most of the useful material is spoken, and turning a call into a brief meant an hour of listening back. So I packaged these steps as claude-transcribe, an MIT-licensed Claude Code skill. /transcribe meeting.m4a saves the labelled transcript beside the input as meeting.transcript.txt. Its SKILL.md holds the rules from step 6.

git clone https://github.com/kam/claude-transcribe.git
cd claude-transcribe
./install.sh

The installer builds the two Swift programs, one for Parakeet and one for Whisper, and links the skill into ~/.claude/skills/. Restart Claude Code and make your first /transcribe a short recording: that run also downloads the models.

I tested the alternatives in September 2026. Apple’s Speech framework ran at 60 times real time but made five word errors on a test clip where WhisperKit made none, and has no speaker labels. Parakeet-mlx, a version of Parakeet for Apple’s MLX library, was slower than FluidAudio and has no diarization. Senko, another diarizer, can’t label overlapping speech, which meetings need. SpeakerKit Pro comes in a separate package from Argmax, WhisperKit’s maker; the free pyannote pipeline already did what I needed.

Turning recorded calls into briefs and follow-ups is the kind of narrow, repeated job I automate in Applied AI engagements.

---

Related reading

• Claude Code skills, hooks, subagents and commands: when to use each, how a skill loads and runs
• Stop letting your AI agent do a script’s job, the same split measured on a reporting workflow
• A local 27B model shipped code through my dev workflow, more on running models locally
• Claude Code hook examples: turning mistakes into tested rules, my other open-source Claude Code tool

Sources

• nvidia/parakeet-tdt-0.6b-v3, model card: Open ASR Leaderboard average, 25 languages, CC-BY-4.0. Model card
• openai/whisper-large-v3-turbo, model card: Open ASR Leaderboard mean WER, MIT licence. Model card
• pyannote/speaker-diarization-community-1, the diarization model, CC-BY-4.0. Model card
• NASA historical sounds, the Apollo 11 and Apollo 13 clips. Public-domain audio
• claude-transcribe on GitHub, the skill’s source, benchmarks and the alternatives record. Repository

Common questions

Can Claude Code transcribe audio files?

Not by itself, but it can run the tools that do. A local speech-to-text model gives the words with timings, a speaker model (diarization) gives who spoke when, and a short script joins the two by timestamp. Claude reads the labelled transcript and can then name the speakers, summarise the call or pull out actions. claude-transcribe, a /transcribe skill, packages these steps.

Can you transcribe a meeting without uploading the recording?

Yes. The speech-to-text model and the speaker model both run on your own machine. On an Apple Silicon Mac, NVIDIA’s Parakeet and the pyannote speaker model run locally through Apple’s Core ML, in versions built for the machine-learning unit in the Mac’s chip. After a one-time model download, the audio never leaves your Mac and no API key is needed. Once Claude reads the transcript, the text goes to Anthropic like any other prompt; run the script without Claude if the words must stay local too.

How fast is local transcription on a Mac?

On my M5 Pro, the Parakeet path transcribed and labelled a 53-second, two-speaker clip in half a second with the models loaded, roughly 100 times real time. The first run after install or a cold start is slower while Core ML compiles and caches the models. Whisper ran at about six times real time on the same clip.

Which parts of transcription should an AI agent do, and which should be code?

Anything that must give the same answer twice belongs in code: decoding the audio, running the speech and speaker models (they run as trained and take no prompt), and joining words to speakers by timestamp. Claude gets the choices that need context: which engine, how many speakers, who each speaker is, and what the call decided. If you would not accept a different answer on a second run, keep the step away from the agent.

When should you use Whisper instead of Parakeet?

Use Whisper for Japanese, Chinese and any other language Whisper covers but Parakeet’s 25 don’t, and as a second pass when proper nouns matter. Parakeet is faster and has the lower average word error rate on the Open ASR Leaderboard, but on my Apollo 11 clip Whisper heard “the Eagle has landed” where Parakeet heard “the angle”.