PocketPal

▶ Quick start

See it running before reading a word about how it works.

# no clone needed
uvx git+https://github.com/ameroyer/pocketpal

# or, from a clone of the repo
uv run python -m pocketpal

…or just double-click PocketPal.command in Finder (first time: right-click → Open, since it isn't signed; it installs uv for you if needed). First run downloads the models; Clippy shows up on your desktop a minute or so later, already listening. Just talk to him; nothing to configure first.

§1 What this is

Read this first: the one-paragraph mental model everything else builds on.

PocketPal is a single small window that floats on top of your desktop: a mascot (Clippy, by default) that listens continuously, transcribes what you say on-device, and answers out loud. Speech in and speech out are both local models running on CPU: Kyutai's pocket-tts for the voice and Moonshine for the ears, with Silero VAD doing speech detection. The brain behind the mascot is always Claude, through the user's claude CLI (no API key needed) or, as a fallback, an OpenRouter model.

The interesting engineering is not the TTS/ASR models themselves (they're upstream libraries); it's the plumbing around them: an always-on listener that has to tell "background noise" from "the user is talking," barge-in detection that has to ignore the mascot's own voice leaking back through the microphone, and a brain dispatcher that can hand control to a live terminal session (typing into someone's real tmux pane, local or over SSH) as easily as it calls a hosted model.

§2 Architecture at a glance

Nine modules, each one concern. Nothing reaches past its neighbor's boundary.

graph TD
  UI["ui.html: SVG mascots + JS bridge"] <--> API["api.py: Api, the JS↔Python bridge"]
  API --> ENGINE["engine.py: job queue, TTS, always-on ears"]
  API --> SESSIONS["sessions.py: discover / drive live sessions"]
  API --> HIST["history.html: 📜 conversation window"]
  ENGINE --> AUDIO["audio.py: Vad + Player"]
  ENGINE --> BRAINS["brains.py: claude CLI / OpenRouter streaming"]
  ENGINE --> SESSIONS
  ENGINE --> TEXT["text.py: sentence split + speech filter"]
  ENGINE --> PERSONAS["personas.py: persona_for()"]
  ENGINE --> CONFIG["config.py: load / save / update_config"]
  BRAINS --> CONFIG
  PERSONAS --> CONFIG
  APP["app.py: run_app()"] --> API
  APP --> UI
  MAIN["__main__.py"] --> APP
  MAIN --> SELFTEST["selftest.py: headless end-to-end test"]
  SELFTEST --> ENGINE

Two rules keep this graph from turning into a hairball: engine.py owns all state that changes during a conversation (the job queue, the ears thread, per-mascot session ids), and config.py is the only place allowed to touch config.json: every other module reads it through load_config() and writes through update_config(), never around it.

§3 How a voice turn flows

From "you start talking" to "he's talking back", and how he yields the floor again.

graph LR
  MIC[Microphone] --> VAD["Silero VAD
32 ms frames"]
  VAD --> END["Endpoint on pause
('patience' knob)"]
  END --> ASR["Moonshine ASR
(or asr_cmd)"]
  ASR --> CHAT["Engine.chat()"]
  CHAT --> PICK{model prefix}
  PICK -->|cli:*| CLI["claude -p
read-only eyes"]
  PICK -->|session:*| DRIVE["type into local tmux pane"]
  PICK -->|rsession:*| SSH["ssh + tmux, same dance"]
  PICK -->|or:*| OR["OpenRouter chat completion"]
  CLI --> STREAM["sentence-by-sentence TTS"]
  DRIVE --> STREAM
  SSH --> STREAM
  OR --> STREAM
  STREAM --> SPK[Speakers]
  SPK -. echo .-> MIC

The dotted line back into the microphone is the crux of the whole ears design: while the mascot is speaking, its own voice is audible in the mic. Engine._ears_loop calibrates, in the first 0.7 seconds of real playback of every reply, how loud that echo is relative to what's coming out of the speakers (couple). That ratio is then frozen for the rest of the reply: only sustained speech clearly louder than the predicted echo counts as a barge-in (the interrupt knob controls how long). Get this calibration wrong and either the mascot can never be interrupted, or it interrupts itself.

Text comes back from the brain as a stream of deltas. Engine._voice_stream buffers those into sentences (text.split_stream_buffer) and speaks each one as soon as it's complete; it never waits for the whole reply. A CodeFenceStripper drops fenced code blocks before they reach the speech filter, and is_speakable() rejects anything that looks like a path, a table row, or a rule rather than a spoken sentence. When driving a live terminal session, text can arrive faster than it can be spoken; _next_spoken keeps strict FIFO order normally, but once the oldest queued sentence is stale and a fresher one has landed since, it drops the backlog and jumps to the newest. A batch that all arrived at once (one long reply) is always read in full; there's nothing fresher to catch up to.

§4 The three brains

Everything the mascot can do (as opposed to just say) depends entirely on which brain answers. This is also the security boundary; see §8.

The model string's prefix is the entire dispatch key: Engine._chat_job switches on it once, at the top. A live session then splits again on how it's reached:

§5 File-by-file tour

What lives where, and the one or two things worth knowing before you touch it.

§6 Config & persistence

One JSON file, one lock, one write path.

Everything persists to ~/.cache/pocket_tts_widget/config.json: voice, mascot, current brain, ear-tuning knobs, persona overrides, the OpenRouter key (plaintext, see §8), and the SSH host list. The same directory holds the recorded-voice files, the widget's pid file, and a capped pastes/ folder for screenshots pasted into the chat bar.

§7 Personas & mascots

The character is a costume, never a cage, and that line is literally in every prompt.

persona_for() builds the final system prompt as three layers, always in this order:

1. Character: the mascot's voice and personality (PERSONAS[mascot], or a user override from config["personas"] saved through the ⚙ panel's personality editor).
2. Capability clause: exactly one of NO_TOOLS_STYLE, READONLY_STYLE, or HANDS_STYLE.format(dir=…), chosen by which brain is answering (§4). This is what actually gates what the model will attempt, not just what it's told: a CLI call without hands genuinely has no write/shell tools wired in.
3. SPOKEN_STYLE, always appended last: write like speech, no markdown, no lists, keep it short, never read code or files aloud verbatim, and, importantly, drop the bit and do real work when the user actually needs it.

Each mascot's default voice

personas.py also holds DEFAULT_VOICES, a plain {"mascot": "voice"} dict, edited directly in source (not through the UI), that maps each mascot to one of the pocket-tts presets in config.PRESET_VOICES:

DEFAULT_VOICES = {
    "clippy": "charles",
    "pocket": "alba",
    "tama": "eve",
    "moshi": "george",
    "imp": "vera",
}

This is not the same thing as config["voice"]: that's the one persisted voice the engine is actually speaking with right now. DEFAULT_VOICES is only consulted at the moment you switch mascots: Api.set_mascot looks up the new mascot's entry and writes it into config["voice"] for you (and hands the value back so the ⚙ panel's voice dropdown updates to match). Pick a different voice from that dropdown afterward and it sticks (same as any other persona switch) until you switch mascots again, at which point the new mascot's default takes over. Add a mascot without an entry here and it simply keeps whatever voice was already selected.

§8 Security model

The honest framing, straight from SECURITY.md: who gets to drive Claude, and what Claude can then touch.

Mitigations that already exist in code, worth knowing before you remove or "simplify" any of them:

• SSH host validation: valid_ssh_host rejects any host token starting with -, since ssh has no -- end-of-options and would otherwise parse -oProxyCommand=… as a local option, a local-RCE path closed at both write and read of ssh_hosts.
• Fixed remote actions only: the SSH driver runs two hardcoded shell templates; the model never composes an SSH command, and the one user-controlled string (the transcribed message) crosses as a single control-char-scrubbed, shell-quoted argument.
• Path-traversal guard on fork cleanup: _forget_forked_session validates the fork id as a bare UUID and checks the resolved path stays inside the project directory before any unlink/rmtree.
• Bounded, collision-free paste directory: pasted screenshots get unique names and the directory is capped at 20 files.
• Atomic, locked config writes: see §6.
• Escaped script-embedding in the history window: Api.open_history escapes a literal </script> in chat text before baking it into the initial HTML, so a reply can't break out of its own script tag (§5).

The webview layer has been audited end to end: every dynamic value rendered into the page is HTML-escaped or set via textContent, the JS transport uses json.dumps with ensure_ascii=True, and every subprocess call is argv-based (tmux send-keys -l --, pbcopy via stdin), except two intentional shell hooks: asr_cmd (config-controlled, by design: treat write access to config as code execution) and the underlying ssh … "bash -s" calls (fixed templates, no user text ever becomes shell syntax). Read the full write-up in SECURITY.md before changing anything touching sessions, SSH, or asr_cmd.

§9 Running & testing

One command to run it, one to prove the voice pipeline still works.

# run it (no clone needed)
uvx git+https://github.com/ameroyer/pocketpal

# or, from a clone of the repo
uv run python -m pocketpal

# or just double-click PocketPal.command in Finder
# (first time: right-click -> Open; it installs uv if needed)

The self-test

uv run python -m pocketpal --selftest is headless (muted, earless) but exercises the real pipeline end to end, against a temporary copy of the config so it never touches real settings:

Debug switches

§10 Sharp edges

Non-obvious constraints that look like bugs if you don't know they're intentional.

§11 Common edits

Where the wiring for each of these actually lives.