24/7 Chief of Staff: Building 'Jarvis' - Hermes Agent as the Brain on an Azure VM, DeepSeek Harness as the Pluggable Face, Agent Client Protocol Instead of MCP, a Realtime Voice Orb, and Tailscale to Keep It Yours

OpenAI's Dot (launched at its dev day) is a personal agent that lives inside your Codex context with a 'start a call' button - a different harness around the same voice model - and settings for plugins, passwords and an opt-in computer-use history that watches your work and builds on it. Instinct (app.instinct.com) hands you a phone number to message and a short list of connectors (Workspace, Outlook, Linear, Notion, GitHub, Slack, Granola, shopping); Grok Bot is the third. For many people this is enough and these consumer products are stable - they rarely break. The custom harness exists for four things they do not offer: your own connectors and MCPs, SSH and CLI access, and custom workflows and APIs - and for not waiting on a vendor to add a feature. The cost is that custom harnesses are not very stable, need constant bug fixes, and sometimes stop working.

The Hermes agent (Nous Research, open source) is where memory, knowledge graph and decision graphs live - every harness the cohort has built sits there. Harshit's own instance, Jarvis Brain, has run since February 2025 (OpenClaw first, now Hermes) and holds his emails, meeting notes, contacts and domains. The DeepSeek Harness (DSH) sits on top as interface and orchestration: the chat, the API, the model selector, the prompt, the tool registry, the session log and the agent loop are all plugins, so any part can be swapped to suit Hermes - Codex's harness is also open source but not built to plug into other tools. DSH has no memory of its own; it pulls memory from Hermes. The agenda: pick a VM on Azure, install Hermes, install and connect DSH, add a voice command to the interface, secure it with Tailscale. DSH's desktop app for Mac and Windows is not customisable; the source is. Awesome-DSH lists plugins, themes and UI kits (much of it from Chinese developers - 'one prompt makes it English').

Three homes. Your own computer: not recommended - eats space, cannot run around the clock, and the Hermes desktop app's gateway keeps disconnecting; a dedicated Mac Studio or DGX Spark is reliable but any fix needs you physically present. Cloud: enterprise-grade, stable, controllable from a CLI - Azure first (every OpenAI model is on Azure and its CLI is excellent, so the agent can restart or redeploy its own brain), AWS second, GCP third (good with Gemini). Hostinger: one-click deployment on Hostinger's servers, also with a CLI, and fine alongside an existing n8n install - the agent can then edit the n8n workflows. Moving later is just copying files and reconnecting Google and calendar accounts. Cost: a 1 vCPU / 1 GB VM is $8 a month and too small; 4 GB is workable; 2 vCPU / 8 GB (Standard B2ms, about $60) runs Hermes 24/7 reliably, more if the agent will drive browsers. Azure takes scheduled snapshots of the whole VM as backup, so a corrupted machine can be restored and respun exactly where it stopped. All of it is done by prompting Claude Code to install and drive the Azure, AWS and gcloud CLIs - 'it's been a while since I even opened the console'. One caution the session surfaced live: Claude printed an Anthropic API key into the terminal twice and told him to revoke it.

Hermes: a fresh clone from GitHub, installed with Nous Research's Linux installer script over SSH - Node 26, Python 3, dependencies, an agent browser, a terminal UI and a web UI; the gateway for Discord and Slack is a later step. DSH: rather than exposing an arbitrary command-execution UI to the internet, Claude opened an SSH tunnel so the harness appears at a localhost link (port 3080) in Chrome; DSH 2 is still in preview. Configuring a model: the UI only offered a DeepSeek slot, so the OpenAI key was stored under the DeepSeek reference and pointed at DeepSeek's endpoint - Claude read the config (batch-file based, multi-provider) without printing the key, moved it to the right provider, added the model catalogue, restarted DSH, and GPT-6 appeared with an effort setting (medium). Harshit's observation that the same key feels faster inside DSH than in ChatGPT, which he attributes to 'mixture of experts', is his explanation, not a verified one. A ChatGPT subscription can now also be used inside DSH instead of an API key.

Claude's first architecture put an MCP connection between DSH and Hermes. Harshit's objection: DSH sits on top of Hermes, so a middleman adds latency - 'the more vectors you add, the slower it gets' - and MCP is slow. Pushed to re-read the DSH packages, Claude found that both projects implement the Agent Client Protocol (ACP): the agent connects to the other agent or plugin directly, no MCP, no API intermediary. The settled diagram: microphone -> Tailscale private net -> layer one Hermes (bound to loopback, reached only by DSH over ACP) -> layer two the custom plugin -> layer three the web interface. Later the voice plugin's ACP target is made configurable - DSH's agent now, Hermes directly later - a one-line change. Two mismatched DSH versions (the VM's installed build vs the cloned source) were caught by reading the built artifact on the VM as ground truth. Sonnet 'fought' the architecture; Opus 5.5 understood it but is slow - the class voted for Opus.

The prompt: keep DSH's interface, add a button that opens a simple moving orb, and when clicked use the realtime voice API underneath, with the persona 'Jarvis'. Claude verified the key could mint a realtime session, first shipped a standalone voice server copied to the VM (the orb answered), then - because DSH's rules require a plugin package registered in three places and published to npm, 'hours not minutes' - built it properly as the Jarvis voice plugin that appears in DSH's bottom-right. The design decision Claude resolved: the realtime model is only ears and mouth; the harness agent does the thinking, so talking to the orb is talking to the harness, not a separate chatbot. At test time Jarvis reported no API key for the DeepSeek provider and could not confirm the Hermes connection even though the key existed - Harshit's diagnosis is that the model assumes no connections 'to be on the safer side' and needs explicit prompting that it lives inside the dashboard with full access; a couple of restarts are normal. Meanwhile a second prompt had DSH itself read its own Hermes and set up a backfill-plus-sync so every harness chat is saved into the agent. Next steps named: Gmail through the GWS CLI, then every other plugin; a Jeff/decisions-API layer is possible.

Tailscale (free for up to six users, unlimited devices) creates a private VPN - a tailnet - tied to your account; install it on the VM (Jarvis did this itself), approve the machine at the sign-in link, install the app on your phone, Mac and Mac mini, and the Jarvis link only works inside that network. Beyond voice: Harshit's own harness has a spatial workspace plugin - click 'spatial' and move windows by gesture, zoom, open the sub-agent harness, backend and audit views, 'fully Tony Stark mode'. Because the chat, the sidebar and the dashboard are plugins (shadcn/Tweakcn UI), the same harness can become a statistics dashboard customers see, an AI CRM, or a single-purpose voice interface with the chat removed; ask the harness and it changes its own interface. For messaging bridges: Telegram first (bot support), then Slack, then Teams. He will publish the voice plugin and spatial workspace on GitHub with all prompts and the VM table.