How to Build a Voice-First AI System That Thinks With You

Typing is the bottleneck most AI workflows never fix. Here's how I built a local voice-first AI system with Whisper and a local LLM — and you can build this exact thing with any AI coding tool you already use: Cursor, Claude Code, Bolt, ChatGPT, anything. One hotkey, voice input wherever your cursor is, zero API costs. Full setup: the architecture, the 4 processing modes, the global hotkey implementation, and the workflow that replaced typing for most of my AI interactions.

---

Do you use AI voice input when working with your projects, or are you still typing everything?

I used to be firmly in the typing camp, until I realized I was sabotaging my own productivity.

Here's what happened: I picked up a specific way of interacting with AI that actually works. I don't just ask AI to "fix this problem"; I give full context: explain what I tried before, share relevant examples, describe what went wrong, and lay out the bigger picture.

This approach gets great results. But there’s a cruel irony: by the time I finish typing all that context, half my original insights have evaporated.

ChatGPT’s voice input seemed like the obvious solution. Until I discovered its special torture: speak for two minutes, watch it process, then watch it fail and lose everything. Back to typing.

I’d been optimizing every other part of my workflow, but voice input remained this glaring gap. So I decided to fix it properly, not with another standalone app, but with something that actually integrates into how I work.

I started building it in Cursor, then used Claude Code to dramatically improve the AI layer. But the process itself, the architecture, the prompts, the hotkey logic, works with whatever AI coding tool you’re already in. You don’t need to switch anything.

---

What you’ll go through with me:

• Why Voice Input Keeps Failing — And the 3 Goals I Set — skip this and you’ll rebuild the same broken thing

• The Local Setup: Whisper + Claude Code + Local LLM — architecture, zero API costs, and 4 modes that solve the “AI too helpful” problem

• Global Hotkeys: Voice Input in Every App — any text field on your system becomes voice-enabled

• How It Looks in Daily Use — the workflow across coding, writing, and communication

• What Didn’t Work — streaming transcription, chunk processing, and what I gave up optimizing

• Where to Start Based on Your Setup — 4 stages, from ChatGPT voice to full local integration

---

Hi, I’m Jenny 👋
I run the Practical AI Builder program — for people who already use AI and want to build real things with it. AI builder behind VibeCoding.Builders and other products with hundreds of paying customers. See all my launches →

If you’re new to Build to Launch, welcome! Here’s what you might enjoy:

• How to Build Your First Claude Code Project — start here if you haven’t built with Claude Code before

• 12 Claude Code Project Ideas (with Prompts) — more things you can build with Claude Code

• The Universal AI Prompting Framework — get better outputs from every AI session

---

Why Voice Input Keeps Failing — And the 3 Goals I Set

If you've used Apple’s built-in dictation, you know it can work really well sometimes. I actually have dictation shortcuts set up on both my phone and laptop. Just by selecting the shortcuts icon and it would transcribe my voice, then place the result in my clipboard.

But it doesn't work very well for people like me. As a non-native speaker, my pronunciation isn't great, and the built-in dictation just doesn't recognize technical terms or professional vocabulary correctly.

In this example, I wanted to say “no, I already have a virtual environment existing in my repo, now find that venv and activate it”. But the built-in dictation perfectly missed the “repo” and “activate it”.

This became a real problem because my pain point was pretty urgent. So I set three specific goals:

1. Real-time voice-to-text flow that actually works for my accent and vocabulary.

2. Local hosting using Ollama and Whisper so I don't have to deal with API limits or connectivity issues.
   Plus, with local processing, I could easily chain Whisper's raw output through other local models to transform the text into any format I wanted.

3. True workflow integration.
   Not just another voice-to-text app where I have to copy, paste, and juggle between different tabs. I was really bothered by that discontinuous feeling of switching contexts all the time.

Why some voice systems work better than others

As I researched solutions, I learned that voice-to-text is basically AI listening to audio patterns and predicting what words match those sounds. Modern systems like Whisper use neural networks trained on massive datasets of human speech, with different accents, languages, background noise, technical jargon.

This explained why built-in dictation struggled with my accent and technical vocabulary, while Whisper handles both pretty well. The training data makes a huge difference.

I knew there were multiple open-source projects that already solved the hard technical problems: real-time audio processing, model integration, and WebSocket handling. So it's silly for me to build everything from scratch again. I finally settled on one that provided solid scaffolding with all the fundamental infrastructure built.

The beauty was that I could focus on the AI enhancement layer instead of rebuilding the wheel.

---

💎 Keep reading with a paid subscription

Inside: the complete technical build — architecture, processing modes, and the exact implementation I use daily:

• The local Whisper + LLM architecture — audio in, enhanced text out, zero API costs and complete privacy, with the model choices that actually work

• 4 processing modes — raw transcription, clean output, polished prose, and AI assistant mode, with when to switch between them

• Global hotkey implementation — how to make voice work in every text field on your system, including the macOS permission setup that trips most people up

• The complete daily workflow — how I use this across coding, writing, and communication with different modes for each

Plus: what I tried that failed (streaming transcription, chunk processing) and why the simpler batch approach works better — so you don’t spend time rebuilding the same dead ends.

Upgrade