November 20, 2026 · Intermediate · 60 min
Multi-user Hermes Agent on Kubernetes
What you'll have at the end
Give every person on your team their own Hermes Agent on one Kubernetes cluster. Each user gets a namespace, a Helm release, a volume, a secret, a network policy and an HTTPS hostname, so no agent can read another's memory or files, use another's API key or dashboard password, or connect to another agent directly inside the cluster. Every command here was run end to end on a fresh cluster.
Prerequisites
- Docker, for a local practice cluster with kind (or any cluster whose CNI enforces NetworkPolicy)
- kubectl 1.36 or newer, Helm 3.8 or newer (OCI charts), kind, curl, openssl and python3
- An OpenRouter API key, ideally a separate one with a small credit limit
Tested with
- kubernetes
- 1.37.0 (kind 0.33.0)
- hermes-agent
- v2026.9.24 (0.21.5)
- helm
- 4.3.1
- calico
- 3.33.1
- envoy-gateway
- 1.10.0 (Gateway API 1.6.1)
- cert-manager
- 1.21.3
Tested on November 20, 2026
By the end of this tutorial, every person on your team has their own Hermes Agent running on one Kubernetes cluster, at their own HTTPS address such as alice.hermes.example.com. Each agent lives in its own namespace with its own volume, secret and API key. Network policy stops any agent from connecting to another directly, and none of them holds credentials for the Kubernetes API. Adding a person is one command.
I ran every command below in order on a fresh local cluster before publishing. The versions are in the box above. If you have a cluster already, the only hard requirement is a CNI that enforces NetworkPolicy, which kind’s default one does not.
Why one Hermes per user
Hermes can serve several people from one process. Its messaging gateway has user allowlists, and its profiles give each agent its own config and memory. Both share one process and one filesystem, though. The agent has a real shell (the default terminal backend is local, so it runs commands inside its own container), and anything one person’s agent can read, another person’s agent can read too.
So the unit of isolation here is the pod: one namespace, one Deployment, one volume per person. Kubernetes then gives you hard boundaries for free: separate filesystems, separate secrets, network policy and resource quotas.
Is a general agent per person the right tool at all? I moved my own automation from Hermes to fixed pipelines, because my jobs were known in advance. A team that wants a general assistant, one that answers open questions, browses and runs tools on request, is exactly where a general agent fits. If you only need one agent for yourself, the single-user version on EKS is simpler.
What you’ll build
flowchart TB
alice(["https://alice.hermes.example.com"])
bob(["https://bob.hermes.example.com"])
gw["Gateway hermes<br/>Envoy Gateway, wildcard TLS from cert-manager"]
alice --> gw
bob --> gw
subgraph A["namespace hermes-alice"]
adep["Deployment hermes, 1 pod<br/>:8642 API (/v1), :9119 dashboard (/)"]
apvc[("PVC hermes-data<br/>/opt/data: memory")]
asec[/"Secret hermes-secrets<br/>keys, password"/]
apol["NetworkPolicy, quota<br/>in: gateway only<br/>out: DNS + public HTTPS"]
adep --- apvc
adep --- asec
adep --- apol
end
subgraph B["namespace hermes-bob"]
bdep["Deployment hermes, 1 pod<br/>same chart, own values"]
bown["own PVC, Secret,<br/>NetworkPolicy, quota"]
bdep --- bown
end
gw --> adep
gw --> bdep
Everything per user comes from one small Helm chart that you write in Step 3. The gateway and the certificate are shared and installed once.
Before you start, put your OpenRouter key in your shell. All later commands read it from there:
export OPENROUTER_API_KEY='sk-or-...'
Create a working directory for the files you are about to write:
mkdir -p hermes-k8s && cd hermes-k8s
Step 1: Create a practice cluster
Skip this step if you already have a cluster whose CNI enforces NetworkPolicy (Calico, Cilium and most managed offerings do).
kind’s default CNI accepts NetworkPolicy objects and then ignores them: the policies look applied while every connection still goes through, and the isolation checks in Step 9 fail. So the cluster starts without a CNI and gets Calico:
cat > kind-hermes.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true # kind's default CNI ignores NetworkPolicy; Calico enforces it
podSubnet: 192.168.0.0/16
nodes:
- role: control-plane
- role: worker
EOF
kind create cluster --name hermes --config kind-hermes.yaml --image kindest/node:v1.37.0
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.33.1/manifests/calico.yaml >/dev/null
kubectl -n kube-system rollout status ds/calico-node --timeout=300s
kubectl wait --for=condition=Ready nodes --all --timeout=180s
Verify: both nodes are Ready.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
hermes-control-plane Ready control-plane 2m v1.37.0
hermes-worker Ready <none> 2m v1.37.0
Step 2: Install Envoy Gateway and cert-manager
Users reach their agent over HTTPS through the Gateway API. Envoy Gateway implements it and installs the Gateway API CRDs with its chart. I picked the Gateway API over an Ingress controller on purpose: the community ingress-nginx project was archived in March 2026 (what that meant in practice). cert-manager issues the TLS certificate.
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.10.0 \
-n envoy-gateway-system --create-namespace --wait >/dev/null
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 \
-n cert-manager --create-namespace --set crds.enabled=true --wait >/dev/null
Verify: both controllers are running and the Gateway API CRDs exist.
kubectl get deploy -n envoy-gateway-system envoy-gateway -o jsonpath='{.status.readyReplicas}{"\n"}'
kubectl get deploy -n cert-manager cert-manager -o jsonpath='{.status.readyReplicas}{"\n"}'
kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'
1
1
v1.6.1
Step 3: Write the per-user Helm chart
The chart creates everything one user needs inside their namespace. Two facts about the official image shape it.
The image starts as root and then drops to an unprivileged user. Its init system (s6-overlay) fixes ownership of the data volume, then runs every Hermes process as user hermes (UID 10000). Setting runAsNonRoot breaks that start, and the image refuses an arbitrary --user. So the pod keeps the default entrypoint, but with privilege escalation off, every Linux capability dropped except the five the init needs, and the default seccomp profile.
The model is pinned in Hermes’ own config. The gateway reads the model from config.yaml on the volume. The image’s default is a large, expensive model, and an environment variable does not override it for the gateway. An init container runs hermes config set on every start, so the Helm value stays the source of truth.
The files:
mkdir -p charts/hermes-user/templates
cat > charts/hermes-user/Chart.yaml <<'EOF'
apiVersion: v2
name: hermes-user
description: One isolated Hermes Agent for one user
type: application
version: 0.1.0
appVersion: "v2026.9.24"
EOF
cat > charts/hermes-user/values.yaml <<'EOF'
# The user this release belongs to. Used in labels and the hostname.
user: ""
image:
repository: nousresearch/hermes-agent
tag: v2026.9.24
pullPolicy: IfNotPresent
# Model the agent uses, through OpenRouter.
model: google/gemini-2.5-flash
# Secret (created outside Helm) holding OPENROUTER_API_KEY, API_SERVER_KEY,
# HERMES_DASHBOARD_BASIC_AUTH_PASSWORD and HERMES_DASHBOARD_BASIC_AUTH_SECRET.
existingSecret: hermes-secrets
storage:
size: 5Gi
storageClassName: "" # empty = cluster default
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "2", memory: 2Gi }
quota:
cpu: "2"
memory: 4Gi
storage: 20Gi
gateway:
enabled: false
name: hermes
namespace: hermes-gateway
domain: hermes.example.com # the user gets <user>.<domain>
EOF
cat > charts/hermes-user/templates/_helpers.tpl <<'EOF'
{{- define "hermes.labels" -}}
app.kubernetes.io/name: hermes
app.kubernetes.io/instance: {{ .Release.Name }}
hermes.example.com/user: {{ required "set --set user=<name>" .Values.user | quote }}
{{- end }}
{{- define "hermes.selector" -}}
app.kubernetes.io/name: hermes
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
EOF
The ServiceAccount mounts no token, so the agent’s shell has no credentials for the Kubernetes API. The volume is kept when the release is uninstalled, because it holds the user’s memory:
cat > charts/hermes-user/templates/serviceaccount.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
automountServiceAccountToken: false
EOF
cat > charts/hermes-user/templates/pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data
labels: {{- include "hermes.labels" . | nindent 4 }}
annotations:
helm.sh/resource-policy: keep # uninstalling the release keeps the user's memory
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: {{ .Values.storage.size }}
{{- with .Values.storage.storageClassName }}
storageClassName: {{ . }}
{{- end }}
EOF
The Deployment runs the gateway with the OpenAI-compatible API server (port 8642) and the web dashboard (port 9119) in one pod. The dashboard refuses to start on a non-loopback address without an auth provider, so it gets the built-in username and password provider:
cat > charts/hermes-user/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
replicas: 1
strategy:
type: Recreate # the volume is ReadWriteOnce: never two pods at once
selector:
matchLabels: {{- include "hermes.selector" . | nindent 6 }}
template:
metadata:
labels: {{- include "hermes.labels" . | nindent 8 }}
spec:
serviceAccountName: hermes
automountServiceAccountToken: false
securityContext:
seccompProfile: { type: RuntimeDefault }
initContainers:
# Pin the model in the user's config.yaml on every start, so the Helm value is the source of truth.
- name: set-model
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["sh", "-c"]
args:
- |
hermes config set model.provider openrouter &&
hermes config set model.default "$MODEL"
env:
- { name: MODEL, value: {{ .Values.model | quote }} }
- { name: HERMES_DOCKER_EXEC_AS_ROOT, value: "1" }
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
add: [CHOWN, DAC_OVERRIDE, FOWNER]
volumeMounts:
- { name: data, mountPath: /opt/data }
containers:
- name: hermes
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
args: ["gateway", "run"]
env:
- { name: API_SERVER_ENABLED, value: "true" }
- { name: API_SERVER_HOST, value: "0.0.0.0" }
- { name: API_SERVER_MODEL_NAME, value: {{ printf "hermes-%s" .Values.user | quote }} }
- { name: HERMES_DASHBOARD, value: "1" }
- { name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME, value: {{ .Values.user | quote }} }
envFrom:
- secretRef:
name: {{ .Values.existingSecret }}
ports:
- { name: api, containerPort: 8642 }
- { name: dashboard, containerPort: 9119 }
readinessProbe:
httpGet: { path: /health, port: api }
periodSeconds: 10
livenessProbe:
httpGet: { path: /health, port: api }
initialDelaySeconds: 60
periodSeconds: 30
resources: {{- toYaml .Values.resources | nindent 12 }}
securityContext:
# The image's init starts as root, fixes ownership of /opt/data,
# then runs every Hermes process as the unprivileged user hermes (UID 10000).
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
add: [CHOWN, DAC_OVERRIDE, FOWNER, SETUID, SETGID]
volumeMounts:
- { name: data, mountPath: /opt/data }
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data
EOF
cat > charts/hermes-user/templates/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
selector: {{- include "hermes.selector" . | nindent 4 }}
ports:
- { name: api, port: 8642, targetPort: api }
- { name: dashboard, port: 9119, targetPort: dashboard }
EOF
A quota caps what one user can consume, so one runaway agent cannot starve the others:
cat > charts/hermes-user/templates/quota.yaml <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
hard:
limits.cpu: {{ .Values.quota.cpu | quote }}
limits.memory: {{ .Values.quota.memory }}
requests.storage: {{ .Values.quota.storage }}
persistentvolumeclaims: "2"
pods: "4"
---
apiVersion: v1
kind: LimitRange
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
limits:
- type: Container
default: { cpu: 500m, memory: 512Mi }
defaultRequest: { cpu: 100m, memory: 128Mi }
EOF
Verify: the chart lints cleanly.
helm lint ./charts/hermes-user --set user=alice | tail -1
1 chart(s) linted, 0 chart(s) failed
Step 4: Add the first user
Each user gets a namespace (labelled as a tenant, which the gateway uses in Step 7) and a Secret with four values: the OpenRouter key, a random API key for their agent, and a random dashboard password plus session-signing secret. The Secret is created outside Helm, so the keys never land in a values file or in Helm’s release history.
To keep the example short, every user gets the same OpenRouter key. Each agent can read the keys in its own environment, so on a real team give each person their own OpenRouter key (see Hardening).
u=alice
kubectl create namespace hermes-$u
kubectl label namespace hermes-$u hermes.example.com/tenant=true
kubectl -n hermes-$u create secret generic hermes-secrets \
--from-literal=OPENROUTER_API_KEY="$OPENROUTER_API_KEY" \
--from-literal=API_SERVER_KEY="$(openssl rand -hex 32)" \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD="$(openssl rand -hex 16)" \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_SECRET="$(openssl rand -hex 32)"
helm install hermes ./charts/hermes-user -n hermes-$u --set user=$u --wait --timeout 10m >/dev/null
The first start pulls a large image, so --wait can take a few minutes.
Verify: the pod is running and the init container pinned the model.
kubectl -n hermes-alice get pods
kubectl -n hermes-alice logs deploy/hermes -c set-model
NAME READY STATUS RESTARTS AGE
hermes-6d9f8b7c4d-x2k8p 1/1 Running 0 3m
✓ Set model.provider = openrouter in /opt/data/config.yaml
✓ Set model.default = google/gemini-2.5-flash in /opt/data/config.yaml
Step 5: Talk to your agent
Before any public hostname exists, reach Alice’s agent through a port-forward. Her API key is in her Secret:
kubectl -n hermes-alice port-forward svc/hermes 8642:8642 >/dev/null &
PF=$!
sleep 3
ALICE_KEY=$(kubectl -n hermes-alice get secret hermes-secrets -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)
curl -s localhost:8642/health; echo
curl -s -o /dev/null -w 'without a key: %{http_code}\n' localhost:8642/v1/models
{"status": "ok", "platform": "hermes-agent", "version": "0.21.5"}
without a key: 401
Now a real request. The API speaks the OpenAI chat format, so any OpenAI-compatible client works too:
curl -s localhost:8642/v1/chat/completions \
-H "Authorization: Bearer $ALICE_KEY" -H 'Content-Type: application/json' \
-d '{"model": "hermes-alice", "messages": [{"role": "user", "content": "Reply with exactly: hello from alice"}]}' \
| python3 -c 'import sys, json; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
kill $PF
Verify: the agent answers through the model.
hello from alice
The dashboard works the same way on port 9119. Her username is alice, and the password is HERMES_DASHBOARD_BASIC_AUTH_PASSWORD in her Secret.
Step 6: Lock down the network
Right now any pod in the cluster can reach Alice’s agent, and her agent can reach anything. Three policies fix that: deny everything, then allow DNS and HTTPS to the public internet out, and allow only the gateway’s proxies in.
The outbound rule is deliberately broad. Hermes is a general assistant whose web tools need the open internet, and a standard NetworkPolicy cannot match hostnames anyway. What it does exclude matters more: every private range, so no other tenant’s pod, no cluster-internal service address and no cloud metadata endpoint (169.254.169.254, where node credentials live on most clouds).
Two things this policy does not stop. Through the public Gateway, one user’s agent can reach another user’s hostname like any client on the internet, so there the per-user API key and dashboard password are what protect it. And if your cluster’s API server has a public endpoint, the agent can reach it on port 443; the missing token and RBAC are what stop it there. To close the first gap, add your Gateway’s load balancer address to the except list.
cat >> charts/hermes-user/values.yaml <<'EOF'
networkPolicy:
enabled: true
# Namespace of the gateway's proxy pods, the only allowed inbound source.
proxyNamespace: envoy-gateway-system
EOF
cat > charts/hermes-user/templates/networkpolicy.yaml <<'EOF'
{{- if .Values.networkPolicy.enabled }}
# 1. Nothing in or out by default.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
# 2. Out: DNS, and HTTPS to the public internet only (model API, web tools).
# Private ranges are excluded: no other tenant, no cluster service, no cloud metadata.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-egress
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels: {{- include "hermes.selector" . | nindent 6 }}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: { kubernetes.io/metadata.name: kube-system }
podSelector:
matchLabels: { k8s-app: kube-dns }
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
- to:
- ipBlock:
cidr: 0.0.0.0/0
except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16]
ports:
- { protocol: TCP, port: 443 }
---
# 3. In: only from the gateway's proxies.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-ingress
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels: {{- include "hermes.selector" . | nindent 6 }}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: { kubernetes.io/metadata.name: {{ .Values.networkPolicy.proxyNamespace }} }
ports:
- { protocol: TCP, port: 8642 }
- { protocol: TCP, port: 9119 }
{{- end }}
EOF
helm upgrade hermes ./charts/hermes-user -n hermes-alice --set user=alice --wait >/dev/null
Verify: the policies exist, the agent still reaches the internet, and the metadata address times out (curl exit code 28).
kubectl -n hermes-alice get networkpolicy
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
curl -s -m 8 -o /dev/null -w 'public HTTPS: %{http_code}\n' https://openrouter.ai/api/v1/models
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
sh -c 'curl -s -m 5 http://169.254.169.254/ || echo "metadata: blocked (exit $?)"'
NAME POD-SELECTOR AGE
default-deny <none> 5s
hermes-egress app.kubernetes.io/instance=hermes,app.kubernetes.io/name=hermes 5s
hermes-ingress app.kubernetes.io/instance=hermes,app.kubernetes.io/name=hermes 5s
public HTTPS: 200
metadata: blocked (exit 28)
Step 7: Expose each user on their own hostname
One shared Gateway terminates TLS for *.hermes.example.com and accepts routes only from namespaces labelled as tenants. Each user’s release adds an HTTPRoute for their own hostname: /v1 goes to the API, everything else to the dashboard.
For practice, cert-manager creates a small private CA and signs the wildcard certificate with it. In production, replace the hermes-ca ClusterIssuer with Let’s Encrypt and a DNS-01 solver (a wildcard certificate needs DNS-01), and point *.hermes.example.com at the Gateway’s address.
mkdir -p platform
cat > platform/gateway.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: v1
kind: Namespace
metadata:
name: hermes-gateway
---
# Practice CA: a self-signed root that signs the wildcard certificate.
# In production, replace the "hermes-ca" ClusterIssuer with Let's Encrypt and a DNS-01 solver
# (a wildcard certificate needs DNS-01).
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: hermes-practice-ca
namespace: cert-manager
spec:
isCA: true
commonName: hermes-practice-ca
secretName: hermes-practice-ca
privateKey: { algorithm: ECDSA, size: 256 }
issuerRef: { name: selfsigned, kind: ClusterIssuer }
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: hermes-ca
spec:
ca:
secretName: hermes-practice-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: hermes-wildcard
namespace: hermes-gateway
spec:
secretName: hermes-wildcard-tls
commonName: "*.hermes.example.com"
dnsNames: ["*.hermes.example.com"]
issuerRef: { name: hermes-ca, kind: ClusterIssuer }
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: hermes
namespace: hermes-gateway
spec:
gatewayClassName: eg
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "*.hermes.example.com"
tls:
mode: Terminate
certificateRefs:
- name: hermes-wildcard-tls
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
hermes.example.com/tenant: "true"
EOF
kubectl apply -f platform/gateway.yaml
kubectl -n hermes-gateway wait --for=condition=Ready certificate/hermes-wildcard --timeout=120s
kubectl -n hermes-gateway wait --for=condition=Accepted gateway/hermes --timeout=120s
Now the route template, and Alice’s release with the gateway turned on:
cat > charts/hermes-user/templates/httproute.yaml <<'EOF'
{{- if .Values.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hermes
labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
parentRefs:
- name: {{ .Values.gateway.name }}
namespace: {{ .Values.gateway.namespace }}
sectionName: https
hostnames:
- {{ printf "%s.%s" .Values.user .Values.gateway.domain | quote }}
rules:
# OpenAI-compatible API
- matches:
- path: { type: PathPrefix, value: /v1 }
backendRefs:
- { name: hermes, port: 8642 }
# Everything else: the web dashboard
- backendRefs:
- { name: hermes, port: 9119 }
{{- end }}
EOF
helm upgrade hermes ./charts/hermes-user -n hermes-alice --set user=alice --set gateway.enabled=true --wait >/dev/null
On a cloud cluster the Gateway gets a load balancer address (kubectl -n hermes-gateway get gateway hermes). kind has no load balancer, so the Gateway stays Programmed=False with reason AddressNotAssigned, and you reach the proxy through a port-forward instead. curl --resolve sends the real hostname, so TLS and routing are exercised exactly as in production:
kubectl -n envoy-gateway-system wait --for=condition=Ready pod \
-l gateway.envoyproxy.io/owning-gateway-name=hermes --timeout=180s >/dev/null
kubectl -n hermes-gateway get secret hermes-wildcard-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > hermes-ca.crt
GW_SVC=$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=hermes -o name)
kubectl -n envoy-gateway-system port-forward "$GW_SVC" 8443:443 >/dev/null &
GW_PF=$!
sleep 3
hcurl() { curl -s --cacert hermes-ca.crt --resolve "$1.hermes.example.com:8443:127.0.0.1" "${@:2}"; }
hcurl alice https://alice.hermes.example.com:8443/v1/chat/completions \
-H "Authorization: Bearer $ALICE_KEY" -H 'Content-Type: application/json' \
-d '{"model": "hermes-alice", "messages": [{"role": "user", "content": "Reply with exactly: hello over https"}]}' \
| python3 -c 'import sys, json; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
hcurl alice -o /dev/null -w 'dashboard: %{http_code}\n' https://alice.hermes.example.com:8443/
Verify: the API answers over HTTPS, and the dashboard redirects to its login page.
hello over https
dashboard: 302
Step 8: Add more users
Steps 4 and 7 for one user fit in a small script. It is safe to re-run: it keeps an existing Secret, so a user’s keys do not change.
cat > add-user.sh <<'EOF'
#!/usr/bin/env bash
# Usage: ./add-user.sh <user> (needs OPENROUTER_API_KEY in the environment)
set -euo pipefail
u="$1"
ns="hermes-$u"
kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace "$ns" hermes.example.com/tenant=true --overwrite
if ! kubectl -n "$ns" get secret hermes-secrets >/dev/null 2>&1; then
kubectl -n "$ns" create secret generic hermes-secrets \
--from-literal=OPENROUTER_API_KEY="$OPENROUTER_API_KEY" \
--from-literal=API_SERVER_KEY="$(openssl rand -hex 32)" \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD="$(openssl rand -hex 16)" \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_SECRET="$(openssl rand -hex 32)"
fi
helm upgrade --install hermes ./charts/hermes-user -n "$ns" \
--set user="$u" --set gateway.enabled=true --wait --timeout 10m
EOF
chmod +x add-user.sh
./add-user.sh bob >/dev/null
Verify: two agents, one per namespace.
kubectl get pods -A -l app.kubernetes.io/name=hermes
NAMESPACE NAME READY STATUS RESTARTS AGE
hermes-alice hermes-7c5d9b8f6-q4wzl 1/1 Running 0 6m
hermes-bob hermes-5f8b6c7d9-m2xkj 1/1 Running 0 1m
To remove a user, uninstall the release and delete the namespace. The keep policy on the volume means helm uninstall alone keeps their memory, and deleting the namespace removes it for good:
helm uninstall hermes -n hermes-bob
kubectl delete namespace hermes-bob
Step 9: Prove the isolation
Five checks, each one a boundary that matters.
1. Alice’s agent cannot reach Bob’s. Her egress policy blocks private addresses, and his ingress policy accepts only the gateway. curl gives up after 5 seconds (exit code 28):
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
sh -c 'curl -s -m 5 http://hermes.hermes-bob:8642/health || echo "alice -> bob: blocked (exit $?)"'
2. Nothing else in the cluster can reach Bob’s agent either, not even a pod in a namespace with no policies:
kubectl run probe -n default --rm -i --restart=Never --quiet --image=curlimages/curl:8.17.0 --command -- \
sh -c 'curl -s -m 5 http://hermes.hermes-bob:8642/health || echo "default -> bob: blocked (exit $?)"'
3. Alice’s key does not open Bob’s agent:
hcurl bob -o /dev/null -w 'alice key on bob: %{http_code}\n' \
-H "Authorization: Bearer $ALICE_KEY" https://bob.hermes.example.com:8443/v1/models
4. The agent has no Kubernetes credentials. No token is mounted, and the service account may not read anything:
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
sh -c 'ls /var/run/secrets/kubernetes.io/serviceaccount 2>/dev/null || echo "no service account token"'
kubectl auth can-i get secrets -n hermes-bob --as=system:serviceaccount:hermes-alice:hermes || true
5. Bob’s own key still works for Bob:
BOB_KEY=$(kubectl -n hermes-bob get secret hermes-secrets -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)
hcurl bob -o /dev/null -w 'bob key on bob: %{http_code}\n' \
-H "Authorization: Bearer $BOB_KEY" https://bob.hermes.example.com:8443/v1/models
kill $GW_PF
Verify: all five, in order.
alice -> bob: blocked (exit 28)
default -> bob: blocked (exit 28)
alice key on bob: 401
no service account token
no
bob key on bob: 200
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Network policies exist but Step 9 check 1 or 2 connects | The CNI does not enforce NetworkPolicy (kind’s default kindnet, some minimal setups) | Use Calico or Cilium, as in Step 1 |
Pod exits at start with [stage2] ERROR: container started with --user | runAsUser/runAsNonRoot set on the pod | Remove them; the image’s init must start as root and drops to UID 10000 itself (Step 3) |
| Agent answers, but your OpenRouter bill shows an expensive model | The gateway reads the model from config.yaml, not from HERMES_INFERENCE_MODEL | Keep the set-model init container; check with kubectl exec deploy/hermes -c hermes -- hermes config get model.default |
Gateway shows Programmed=False, reason AddressNotAssigned | No load balancer (kind, bare metal) | Expected on kind; use the port-forward in Step 7, or install MetalLB or cloud-provider-kind |
curl: (60) or macOS curl couldn't get X509-issuer name | The client does not trust the practice CA, or the certificate is a subject-less self-signed leaf | Pass --cacert hermes-ca.crt; keep the CA issuer from Step 7 instead of signing the wildcard with selfSigned directly |
API returns 401 with the right key | An API_SERVER_KEY line in /opt/data/.env wins over the pod environment (the logs warn about it) | Remove the line from .env on the volume, or rotate the key in the Secret and restart |
| Dashboard does not come up, logs mention a missing auth provider | The dashboard fails closed on a non-loopback bind without auth | Keep the HERMES_DASHBOARD_BASIC_AUTH_* values, or configure OIDC |
| The agent cannot resolve any hostname after Step 6 | The cluster runs NodeLocal DNSCache (169.254.20.10), which is inside the excluded 169.254.0.0/16 range | Add an egress rule for 169.254.20.10/32 on port 53 UDP and TCP |
helm install --wait times out on the first user | The image is large and the first pull is slow | Raise --timeout; check kubectl -n hermes-alice describe pod for pull progress |
Hardening for production
- Restrict egress further if your users do not need the open web. Standard NetworkPolicy matches IP ranges only. Cilium’s
toFQDNsrules can limit egress toopenrouter.aiby name. - Use OIDC for the dashboard instead of the shared username and password. Hermes supports a self-hosted OpenID Connect provider (
HERMES_DASHBOARD_OIDC_ISSUER,HERMES_DASHBOARD_OIDC_CLIENT_ID). Exposed, unauthenticated Hermes dashboards and API servers were scanned and abused on the internet in June 2026, which is why the dashboard now refuses to start without auth. Treat every agent endpoint as a remote shell, because that is what it is. - One OpenRouter key per user, each with its own credit limit. A leaked key then exposes one person, one person’s runaway loop cannot drain everyone’s budget, and spend is visible per person.
- Back up the volumes.
/opt/dataholds each user’s memory, sessions and skills. A volume snapshot schedule (or Velero) is the difference between a pod restart and a lost assistant. - An operator instead of Helm. A community Hermes operator exists on OperatorHub. A plain Helm release per user is easier to read and debug, which is why this tutorial uses one.
Cleanup
Delete the practice cluster and everything in it:
cd ..
kind delete cluster --name hermes