335 lines
13 KiB
Markdown
335 lines
13 KiB
Markdown
# Technical Definition — “Menu-Whisper” (macOS, Swift, Offline STT)
|
||
|
||
## 0) Owner Decisions (Locked)
|
||
- **Platform:** Apple Silicon only (M1/M2/M3), macOS 13+.
|
||
- **STT backends:** Start with **whisper.cpp (Metal)** for simplicity; add **Core ML** backend later.
|
||
- **Models:** Do **not** auto-download. On first run, user **chooses & downloads** a model.
|
||
- **VAD:** Post-MVP.
|
||
- **Insertion behavior:** Configurable; **direct insertion** is default (no preview).
|
||
- **Default hotkey:** **⌘⇧V** (user-configurable).
|
||
- **Punctuation:** Let the model handle punctuation automatically (no spoken commands).
|
||
- **Privacy/Connectivity:** 100% local at runtime; model downloads only when the user explicitly requests. **No telemetry**.
|
||
- **Distribution:** **.app/.dmg** (signed + notarized), outside the Mac App Store initially.
|
||
- **UI languages:** **ES/EN**.
|
||
- **Low-power mode:** Still allow downloads if the user starts them.
|
||
- **License:** **MIT**.
|
||
- **Per-dictation limit:** **10 minutes** by default (configurable).
|
||
|
||
---
|
||
|
||
## 1) Goal
|
||
A **menu bar** app for macOS that performs **offline speech-to-text** using Whisper-family models and **inserts the transcribed text** into whichever app currently has focus. Shows a minimal **HUD** while listening and processing. No internet required during normal operation.
|
||
|
||
---
|
||
|
||
## 2) MVP Scope
|
||
- Persistent **menu bar** item (NSStatusItem / `MenuBarExtra`).
|
||
- **Global hotkey** (push-to-talk and toggle modes).
|
||
- **HUD** (centered NSPanel + SwiftUI):
|
||
- “Listening” with audio-level animation (RMS/peak).
|
||
- “Processing” with a spinner/animation.
|
||
- **Offline STT** with **whisper.cpp** (GGUF models; Metal acceleration on Apple Silicon).
|
||
- **Model Manager**: curated list, manual download with progress + SHA256 check, user selection.
|
||
- **Text injection**:
|
||
- Preferred: **Clipboard + ⌘V** paste.
|
||
- Fallback: **simulated typing** via CGEvent.
|
||
- If **Secure Input** is active, **do not inject**; show notice and keep text on clipboard.
|
||
- **Preferences**: hotkey & mode, model & language, insertion method, HUD styling, sounds, dictation limit.
|
||
- **Permissions onboarding**: Microphone, Accessibility, Input Monitoring.
|
||
|
||
---
|
||
|
||
## 3) Functional Requirements
|
||
|
||
### 3.1 Capture
|
||
- Prompt for permissions on first use.
|
||
- Global hotkey (default ⌘⇧V).
|
||
- **Push-to-talk**: start on key down, stop on key up.
|
||
- **Toggle**: press to start, press again to stop.
|
||
- Per-dictation limit (default 10 min, range 10 s–30 min).
|
||
|
||
### 3.2 HUD / UX
|
||
- Non-activating, centered **NSPanel** (~320×160), no focus stealing.
|
||
- **Listening**: bar-style audio visualization driven by live RMS/peak.
|
||
- **Processing**: spinner + “Transcribing…” label.
|
||
- **Esc** to cancel.
|
||
- Optional start/stop sounds (user-toggleable).
|
||
|
||
### 3.3 STT
|
||
- Backend A (MVP): **whisper.cpp** with **GGUF** and **Metal**.
|
||
- Language: auto-detect or forced (persisted).
|
||
- Basic text normalization; punctuation from the model.
|
||
- UTF-8 output; standard replacements (quotes, dashes, etc.).
|
||
|
||
### 3.4 Injection
|
||
- Preferred method: **NSPasteboard** + **CGEvent** to send ⌘V.
|
||
- Fallback: **CGEventCreateKeyboardEvent** (character-by-character), respecting active keyboard layout.
|
||
- **Secure Input**: detect with `IsSecureEventInputEnabled()`; if enabled, **do not inject**. Show a non-intrusive notice and leave the text on the clipboard.
|
||
|
||
### 3.5 Preferences
|
||
- **General:** hotkey + mode (push/toggle), sounds, HUD options.
|
||
- **Models:** catalog, download, select active model, language, local storage path.
|
||
- **Insertion:** direct vs preview (preview **off** by default), paste vs type.
|
||
- **Advanced:** limits, performance knobs (threads/batch), **local** logs opt-in.
|
||
|
||
---
|
||
|
||
## 4) Non-Functional Requirements
|
||
- **Offline** execution after models are installed.
|
||
- **Latency target** (M1 + “small” model): < 4 s for 10 s of audio.
|
||
- **Memory target:** ~1.5–2.5 GB with “small”.
|
||
- **Privacy:** audio and text never leave the device.
|
||
- **Accessibility:** sufficient contrast; VoiceOver labels; focus never stolen by HUD.
|
||
|
||
---
|
||
|
||
## 5) Architecture (High-Level)
|
||
- **App (SwiftUI)** with AppKit bridges for NSStatusItem and NSPanel.
|
||
- **Shortcut Manager** (Carbon `RegisterEventHotKey` or HotKey/MASShortcut).
|
||
- **Audio**: AVAudioEngine (downsample to 16 kHz mono, 16-bit PCM).
|
||
- **STT Engine**:
|
||
- **whisper.cpp** (C/C++ via SPM/CMake) with Metal.
|
||
- **Core ML backend** (e.g., WhisperKit / custom) in a later phase.
|
||
- **Model Manager**: curated catalog, downloads (progress + SHA256), selection, caching.
|
||
- **Text Injection**: pasteboard + CGEvent; typing fallback; Secure Input detection.
|
||
- **Permissions Manager**: guided flows to System Settings panes.
|
||
- **Settings**: UserDefaults + JSON export/import.
|
||
- **Packaging**: .app + .dmg (signed & notarized).
|
||
|
||
---
|
||
|
||
## 6) Main Flow
|
||
1. User presses global hotkey.
|
||
2. Check permissions; guide if missing.
|
||
3. Show HUD → **Listening**; start capture.
|
||
4. Stop (key up/toggle/timeout).
|
||
5. HUD → **Processing**; run STT in background.
|
||
6. On result → (optional preview) → **insert** (paste) or **fallback** (type). If Secure Input, **do not inject**; keep in clipboard + show notice.
|
||
7. Close HUD → **Idle**.
|
||
|
||
---
|
||
|
||
## 7) Finite State Machine (FSM)
|
||
- **Idle** → (Hotkey) → **Listening**
|
||
- **Listening** → (Stop/Timeout) → **Processing**
|
||
- **Processing** → (Done) → **Injecting**
|
||
- **Injecting** → (Done) → **Idle**
|
||
- Any → (Error) → **ErrorModal** → **Idle**
|
||
|
||
---
|
||
|
||
## 8) Model Management (Manual Downloads)
|
||
**Goal:** Offer a clear list of **free** Whisper-family models (names, sizes, languages, recommended backend) with one-click downloads. No automatic downloads.
|
||
|
||
### 8.1 OpenAI Whisper (official weights)
|
||
- Families: **tiny**, **base**, **small**, **medium**, **large-v2**, **large-v3** (multilingual; some `.en` variants).
|
||
- Usable with **whisper.cpp** via **GGUF** (community conversions widely available).
|
||
|
||
### 8.2 Whisper for whisper.cpp (converted GGUF)
|
||
- Community-maintained conversions for whisper.cpp (GGUF), optimized for CPU/GPU Metal on macOS.
|
||
|
||
### 8.3 Faster-Whisper (CTranslate2)
|
||
- Optimized variants (tiny/base/small/medium/large-v2/large-v3). Useful if a CT2-based or Core-ML-assisted backend is added later.
|
||
|
||
### 8.4 Distil-Whisper (distilled)
|
||
- Distilled models (e.g., **distil-large-v2/v3/v3.5**, **distil-small.en**), significantly smaller/faster with near-large accuracy.
|
||
|
||
> **UI must show:** model file size, languages, license, **RAM estimate**, and a warning if a large model is selected on lower-memory machines.
|
||
|
||
**Optional JSON Schema for catalog entries (for the app’s first-run picker):**
|
||
|
||
```json
|
||
{
|
||
"name": "whisper-small",
|
||
"family": "OpenAI-Whisper",
|
||
"format": "gguf",
|
||
"size_mb": 466,
|
||
"languages": ["multilingual"],
|
||
"recommended_backend": "whisper.cpp",
|
||
"quality_tier": "small",
|
||
"license": "MIT",
|
||
"sha256": "…",
|
||
"download_url": "…",
|
||
"notes": "Good balance of speed/accuracy on M1/M2."
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 9) Security & Permissions
|
||
|
||
* **Info.plist:** `NSMicrophoneUsageDescription`.
|
||
* **Accessibility & Input Monitoring:** required for CGEvent; provide clear step-by-step guidance and deep-links.
|
||
* **Secure Input:** check `IsSecureEventInputEnabled()`; **never** attempt to bypass. Provide help text to identify apps that enable it (password fields, 2FA prompts, etc.).
|
||
|
||
---
|
||
|
||
## 10) Performance
|
||
|
||
* Lazy-load and reuse model (warm cache).
|
||
* Real-time downsampling to 16 kHz mono; chunked streaming into backend.
|
||
* Configurable threads; prefer **Metal** path on Apple Silicon.
|
||
* “Fast path” tweaks for short clips (<15 s).
|
||
|
||
---
|
||
|
||
## 11) Logging & Privacy
|
||
|
||
* **No remote telemetry.**
|
||
* Local logs **opt-in** (timings, errors only). Never store audio/text unless user explicitly enables a debug flag.
|
||
* “Wipe local data” button (models remain unless the user removes them).
|
||
|
||
---
|
||
|
||
## 12) Internationalization
|
||
|
||
* UI in **Spanish** and **English** (Localizable.strings).
|
||
* STT multilingual; language auto or forced per user preference.
|
||
|
||
---
|
||
|
||
## 13) Testing (Minimum)
|
||
|
||
* macOS 13/14/15 on M1/M2/M3.
|
||
* Injection works in Safari, Chrome, Notes, VS Code, Terminal, iTerm2, Mail.
|
||
* **Secure Input**: correctly detected; no injection; clipboard + notice.
|
||
* Meet latency target with **small** model on M1.
|
||
* Model download & selection flows (simulate network errors).
|
||
|
||
---
|
||
|
||
## 14) Phased Plan (AI-Deliverables)
|
||
|
||
### Phase 0 — Scaffolding (MVP-0)
|
||
|
||
**Goal:** Base project + menubar.
|
||
**Deliverables:**
|
||
|
||
* SwiftUI app with `MenuBarExtra`, microphone icon, “Idle” state.
|
||
* `ARCHITECTURE.md` describing modules (Audio/STT/Injection/Models/Permissions/Settings).
|
||
* Build scripts and signing/notarization templates.
|
||
**DoD:** Compiles; menu bar item visible; SPM structure ready.
|
||
|
||
---
|
||
|
||
### Phase 1 — Hotkey + HUD + Audio (MVP-1)
|
||
|
||
**Goal:** Listening UX without real STT.
|
||
**Deliverables:**
|
||
|
||
* Global hotkey (default ⌘⇧V) with **push** and **toggle**.
|
||
* NSPanel HUD (Listening/Processing) + **real** RMS bars from AVAudioEngine.
|
||
* Per-dictation limit (default 10 min).
|
||
**DoD:** Live meter responds to mic; correct state transitions.
|
||
|
||
---
|
||
|
||
### Phase 2 — STT via whisper.cpp (MVP-2)
|
||
|
||
**Goal:** Real offline transcription.
|
||
**Deliverables:**
|
||
|
||
* **whisper.cpp** module (C/C++), background inference with **Metal**.
|
||
* **Model Manager** (curated list, download with SHA256, selection).
|
||
* Language auto/forced; basic normalization.
|
||
**DoD:** 10-second clip → coherent ES/EN text offline; meets timing targets.
|
||
|
||
---
|
||
|
||
### Phase 3 — Robust Insertion (MVP-3)
|
||
|
||
**Goal:** Reliable insertion into focused app.
|
||
**Deliverables:**
|
||
|
||
* Paste (clipboard + ⌘V) and typing fallback.
|
||
* **Secure Input** detection; safe behavior (no injection, clipboard + notice).
|
||
**DoD:** Works across target apps; correct Secure Input handling.
|
||
|
||
---
|
||
|
||
### Phase 4 — Preferences + UX Polish (MVP-4)
|
||
|
||
**Goal:** Complete options & stability.
|
||
**Deliverables:**
|
||
|
||
* Full Preferences (hotkey, modes, model, language, insertion, HUD, sounds).
|
||
* Optional preview dialog (off by default).
|
||
* Config export/import (JSON).
|
||
**DoD:** All settings persist and are honored.
|
||
|
||
---
|
||
|
||
### Phase 5 — Distribution (MVP-5)
|
||
|
||
**Goal:** Installable package.
|
||
**Deliverables:**
|
||
|
||
* Error handling; permission prompts & help (incl. Secure Input troubleshooting).
|
||
* **.dmg** (signed + notarized) and install guide.
|
||
* **USER\_GUIDE.md** + **TROUBLESHOOTING.md**.
|
||
**DoD:** Clean install on test machines; distribution checklist passed.
|
||
|
||
---
|
||
|
||
### Phase 6 — Core ML Backend (Post-MVP)
|
||
|
||
**Goal:** Second backend.
|
||
**Deliverables:**
|
||
|
||
* **Core ML** integration (e.g., WhisperKit or custom conversion).
|
||
* Backend selector (whisper.cpp/Core ML) in Preferences; local benchmarks table.
|
||
**DoD:** Feature parity and stability; documented pros/cons.
|
||
|
||
---
|
||
|
||
## 15) Mini-Prompts for the Builder AI (per Phase)
|
||
|
||
* **P0:** “Create macOS 13+ SwiftUI menubar app (`MenuBarExtra`), microphone icon, SPM layout with modules in `ARCHITECTURE.md`.”
|
||
* **P1:** “Add global hotkey (push & toggle) with `RegisterEventHotKey`; NSPanel HUD with RMS bars from AVAudioEngine; 10-minute dictation limit.”
|
||
* **P2:** “Integrate **whisper.cpp** (Metal); add Model Manager (curated list, SHA256-verified downloads, selection); language auto/forced; transcribe WAV 16 kHz mono.”
|
||
* **P3:** “Implement insertion: pasteboard+⌘V and CGEvent typing fallback; detect `IsSecureEventInputEnabled()` and avoid injection.”
|
||
* **P4:** “Implement full Preferences, optional preview, JSON export/import; UX polish and messages.”
|
||
* **P5:** “Signing + notarization; produce .dmg; write USER\_GUIDE and TROUBLESHOOTING (with Secure Input section).”
|
||
* **P6:** “Add Core ML backend (WhisperKit/custom), backend selector, and local benchmarks.”
|
||
|
||
---
|
||
|
||
## 16) Suggested Repo Layout
|
||
|
||
```
|
||
MenuWhisper/
|
||
Sources/
|
||
App/ # SwiftUI + AppKit bridges
|
||
Core/
|
||
Audio/ # AVAudioEngine capture + meters
|
||
STT/
|
||
WhisperCPP/ # C/C++ wrapper + Metal path
|
||
CoreML/ # post-MVP
|
||
Models/ # catalog, downloads, hashes
|
||
Injection/ # clipboard, CGEvent typing, secure input checks
|
||
Permissions/
|
||
Settings/
|
||
Utils/
|
||
Resources/ # icons, sounds, localizations
|
||
Docs/ # ARCHITECTURE.md, USER_GUIDE.md, TROUBLESHOOTING.md
|
||
Scripts/ # build, sign, notarize
|
||
Tests/ # unit + integration
|
||
```
|
||
|
||
---
|
||
|
||
## 17) Risks & Mitigations
|
||
|
||
* **Hotkey collision (⌘⇧V)** with “Paste and Match Style” in some apps → make it discoverable & easily rebindable; warn on conflict.
|
||
* **Secure Input** blocks injection → inform the user, keep text on clipboard, provide help to identify the app enabling it.
|
||
* **RAM/latency** with large models → recommend **small/base** by default; show RAM/latency hints in the model picker.
|
||
* **Keyboard layouts** → prefer paste; if typing, map using the active layout.
|
||
|
||
---
|
||
|
||
## 18) Global MVP Definition of Done
|
||
|
||
* A 30–90 s dictation yields accurate ES/EN text **offline** and inserts correctly in common apps.
|
||
* Secure Input is correctly detected and handled.
|
||
* Model download/selection is robust and user-driven.
|
||
* Shippable **.dmg** (signed + notarized) and clear docs included.
|