# Why I left Hermes Agent and built my own harness

> In July I wrote that Hermes Agent ran the logistics of my digital life. On 18 October I turned it off and replaced it with a small harness of fixed pipelines, one gated door to the models and an allowlist per job. Here is why, what the rewrite cost me, and when I would still pick a general agent.

- URL: https://ashkmb.com/blog/why-i-left-hermes-agent/
- Author: Ashkan Kamyab (https://ashkmb.com)
- Published: 2026-11-07
- Language: en

In July I wrote that [Hermes Agent](https://ashkmb.com/blog/hermes-automation/) handled the logistics around my blog, YouTube and taxes. On 18 October I turned it off and replaced it with a harness I wrote myself. It had run for about three months.

This is not a takedown. Hermes worked, and that post is still accurate about what it did. It stopped fitting for two reasons, and the second one only became visible once I had lived with the first.

## What I was actually running

Look at what the box did every week and it is a short list: a morning research digest, invoice extraction, a few YouTube pipelines, a fitness coach, a spend page. Each one is the same sequence of steps, run again and again, with a human approving the output.

That matters, because a general agent is built for the opposite case: work where nobody knows the steps in advance.

## Reason 1: the fence was a prompt

I ran Hermes natively on the mini-PC, as my own user, with passwordless sudo. I did that on purpose, so it could see `terraform`, `kubectl` and `gh` like I can. The cost was written in my own notes: a Telegram message was a root shell.

The real fence was two things, a Telegram allowlist and a dashboard password. The gateway turns Telegram on from the bot token alone, with no enable flag, so a missing allowlist entry means anyone who finds the bot gets a shell.

On top of that I added guardrails as skills: plan before apply, never auto-apply, branch before commit, no AI attribution. I needed them because the agent never read my own `CLAUDE.md`. I tested them, and it did refuse "deploy to prod now".

But a skill is a request to a model. Only its description sits in context all the time; the rest is loaded when the model decides it is relevant. A rule the model has to remember to apply is a hope, not a control. I wanted rules that still hold on the day the model gets it wrong.

![Hermes Agent architecture: you, one gateway with a general agent loop, native tools with sudo, and a hosted model called directly](https://ashkmb.com/images/blog/hermes-before.svg)

*How it looked: one general loop, full tool access, and two fences that were not part of the loop itself.*

## Reason 2: a general layer hides where it fails

The failure that made this concrete was OCR. Through Hermes, an image never reached the model. It invented an answer from the filename: four attempts, four fabrications, including a wrong digit and a made-up test card. The same vision model, called directly, transcribed the image perfectly.

That is a plumbing bug, and plumbing bugs get fixed. The lesson is about where it shows up. In a general agent, a broken layer between you and the model looks exactly like the model being wrong. In a pipeline, one step makes one model call with one checkable output, and you can see which part lied.

There was other friction of the same kind: provider secrets could not come from a plugin source because the loader runs before plugins register, and the dashboard session token rotated on every restart, so each deploy logged the desktop app out. None of it was fatal. All of it was somebody else's decisions sitting between me and my own jobs.

## What I built instead

The harness is deliberately dull. Code owns the flow; the model fills in steps.

- **Skills are fixed pipelines.** A skill is a small config plus a prompt, run by one runner, and each skill gets its own tool allowlist. No skill lets a model loop freely over tools.
- **One door to the models.** Everything goes through a single module. It applies alias rules, an approval gate for the expensive aliases and a monthly cap on the strongest one. Models sit behind LiteLLM with a virtual key per client, so a new app cannot spend what it was not given.
- **Validators in code.** The debate show checks every number against its source, including spelled-out ones. A Short may carry at most three numbers. If I force a claim the sources do not support, the script says "X reports" instead of stating it as fact.
- **Boring storage.** The queues are SQLite tables behind a handful of functions. A digest is staged as an outbox, so a kill mid-send repeats at most one part. Postgres and a real queue are a swap behind that one file, and I will do it when volume asks for it.

![Harness architecture: triggers, job queue, skill runner, one model gate, LiteLLM, models, with human approval feeding the gate](https://ashkmb.com/images/blog/harness-after.svg)

*The harness at a glance. Everything that can spend money passes through the gate, and a human tap is part of the path.*

A simplified sketch of the gate, because this is the part that Hermes could not give me:

```python
def call(alias, messages, purpose):
    if alias in GATED and not approved(purpose):
        raise ApprovalRequired(alias, purpose)      # waits for a Telegram tap
    if alias == "strong" and month_spend("strong") >= STRONG_CAP:
        raise BudgetExceeded(alias)
    return gateway.complete(alias, messages, tags=[purpose])
```

I put the rule in code instead of in a prompt because code does not get talked out of it. The same idea later moved into my coding-agent setup as a command guard that lives in a config file and is checked by a hook before a command runs.

The spend page shows what this buys. Over the last 30 days all model spend across the apps was 3.66 USD: AgenticTube half, the chat UI a quarter, my memory server a fifth, the harness itself 3%, the fitness coach nothing. Each number has a purpose tag behind it, so I know what it was for.

## Why I did not just adopt an SDK

I looked at AWS's open-source Strands Agents, which is well made and has the multi-agent primitives I would have to write otherwise. I decided to copy its patterns and not use it. Pointed at LiteLLM directly, it would bypass the single door: no approval gate, no cap. If I ever spike with it, it gets its own key limited to the cheap aliases and a small budget.

That is the whole build-versus-adopt argument for me. The thing I care about most sits exactly where a framework wants to put its own loop.

## What it cost, honestly

- **Time.** I rebuilt things Hermes shipped: the Telegram front door, approvals, a dashboard, a job queue, nightly backups. That is weeks of work I did not have to do before.
- **Open-ended asks.** "Go work out X on the box" was a normal request with Hermes. The harness cannot do that. Free-text chat that routes to the existing skills is on my list and not built.
- **It is all mine to maintain.** When something breaks there is no upstream, only me.
- **Some of what annoyed me was fixable.** The OCR bug and the token rotation were both things I could have wrapped or patched. If those had been the only issues I would probably still be running it. The real reason is structural: I wanted enforcement in code, and I had a narrow set of jobs.

## When I would still pick a general agent

When I do not yet know the jobs. Hermes was a good way to find out what I wanted to automate, because it let me try things without building anything. The harness only exists because three months of that told me what the fixed list was.

If your work is a handful of repeating pipelines and you care about who can spend, send or delete what, build the dull version. If you are still exploring, start general, write down what you keep doing, and treat the day that list stops changing as the signal to leave.
