Idea
TUSK (Task Unified Speech Kernel) is my attempt at a real-life Jarvis. The goal is to take your hands off the keyboard and the mouse, and to make human-computer interaction work like human-to-human conversation. You talk, it listens all the time, and it works on your computer for you: it dictates, opens and arranges windows, and drives your apps. Everything below is the engineering that makes this practical.
Context
A desktop assistant for GNOME that runs in the background and listens all the time. Python. Speech comes in through WebRTC VAD and Whisper (local or Groq); the agent calls tools, switches models at runtime, takes dictation and drives the desktop.
Problem
An assistant that listens all the time hears everything: background noise, speech addressed to other people, and speech-to-text hallucinations. Sending all of that, with the full conversation history, to a large model on every turn made replies slow and expensive. Conversation only feels natural if the reply is fast.
Approach
- A cheap gatekeeper model sees every transcript first and drops noise, hallucinations and speech not addressed to the assistant. Only the rest reaches the main agents.
- Work is split into talk, planner and executor agents, so the model that answers is not the model that plans tool calls.
- The agent no longer keeps the full conversation. Context is compressed into topic chunks; when the topic changes, the agent compacts the old one and opens a new context window.
Outcome
Replies got noticeably faster and LLM token use dropped, because each request carries only the current topic instead of the whole session. Model switching at runtime lets a cheap model do the filtering and a strong one do the work.