# Multi-user Hermes Agent on Kubernetes

> Give every person on your team their own Hermes Agent on one Kubernetes cluster. Each user gets a namespace, a Helm release, a volume, a secret, a network policy and an HTTPS hostname, so no agent can read another's memory or files, use another's API key or dashboard password, or connect to another agent directly inside the cluster. Every command here was run end to end on a fresh cluster.

- URL: https://ashkmb.com/tutorials/multi-user-hermes-agent-kubernetes/
- Author: Ashkan Kamyab (https://ashkmb.com)
- Published: 2026-11-20
- Language: en
- Difficulty: intermediate
- Duration: 60 min
- Prerequisites: Docker, for a local practice cluster with kind (or any cluster whose CNI enforces NetworkPolicy); kubectl 1.36 or newer, Helm 3.8 or newer (OCI charts), kind, curl, openssl and python3; An OpenRouter API key, ideally a separate one with a small credit limit
- Tested: 2026-11-20; kubernetes 1.37.0 (kind 0.33.0); hermes-agent v2026.9.24 (0.21.5); helm 4.3.1; calico 3.33.1; envoy-gateway 1.10.0 (Gateway API 1.6.1); cert-manager 1.21.3

By the end of this tutorial, every person on your team has their own [Hermes Agent](https://github.com/NousResearch/hermes-agent) running on one Kubernetes cluster, at their own HTTPS address such as `alice.hermes.example.com`. Each agent lives in its own namespace with its own volume, secret and API key. Network policy stops any agent from connecting to another directly, and none of them holds credentials for the Kubernetes API. Adding a person is one command.

I ran every command below in order on a fresh local cluster before publishing. The versions are in the box above. If you have a cluster already, the only hard requirement is a CNI that enforces NetworkPolicy, which kind's default one does not.

## Why one Hermes per user

Hermes can serve several people from one process. Its messaging gateway has user allowlists, and its profiles give each agent its own config and memory. Both share one process and one filesystem, though. The agent has a real shell (the default terminal backend is `local`, so it runs commands inside its own container), and anything one person's agent can read, another person's agent can read too.

So the unit of isolation here is the pod: one namespace, one Deployment, one volume per person. Kubernetes then gives you hard boundaries for free: separate filesystems, separate secrets, network policy and resource quotas.

Is a general agent per person the right tool at all? I moved my own automation from Hermes to [fixed pipelines](https://ashkmb.com/blog/why-i-left-hermes-agent/), because my jobs were known in advance. A team that wants a general assistant, one that answers open questions, browses and runs tools on request, is exactly where a general agent fits. If you only need one agent for yourself, the [single-user version on EKS](https://ashkmb.com/blog/hermes-agent-eks-deployment/) is simpler.

## What you'll build

```mermaid
flowchart TB
  alice(["https://alice.hermes.example.com"])
  bob(["https://bob.hermes.example.com"])
  gw["Gateway hermes<br/>Envoy Gateway, wildcard TLS from cert-manager"]
  alice --> gw
  bob --> gw

  subgraph A["namespace hermes-alice"]
    adep["Deployment hermes, 1 pod<br/>:8642 API (/v1), :9119 dashboard (/)"]
    apvc[("PVC hermes-data<br/>/opt/data: memory")]
    asec[/"Secret hermes-secrets<br/>keys, password"/]
    apol["NetworkPolicy, quota<br/>in: gateway only<br/>out: DNS + public HTTPS"]
    adep --- apvc
    adep --- asec
    adep --- apol
  end

  subgraph B["namespace hermes-bob"]
    bdep["Deployment hermes, 1 pod<br/>same chart, own values"]
    bown["own PVC, Secret,<br/>NetworkPolicy, quota"]
    bdep --- bown
  end

  gw --> adep
  gw --> bdep
```

Everything per user comes from one small Helm chart that you write in Step 3. The gateway and the certificate are shared and installed once.

Before you start, put your OpenRouter key in your shell. All later commands read it from there:

```bash no-run
export OPENROUTER_API_KEY='sk-or-...'
```

Create a working directory for the files you are about to write:

```bash
mkdir -p hermes-k8s && cd hermes-k8s
```

## Step 1: Create a practice cluster

Skip this step if you already have a cluster whose CNI enforces NetworkPolicy (Calico, Cilium and most managed offerings do).

kind's default CNI accepts NetworkPolicy objects and then ignores them: the policies look applied while every connection still goes through, and the isolation checks in Step 9 fail. So the cluster starts without a CNI and gets [Calico](https://docs.tigera.io/calico/latest/about/):

```bash
cat > kind-hermes.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true   # kind's default CNI ignores NetworkPolicy; Calico enforces it
  podSubnet: 192.168.0.0/16
nodes:
  - role: control-plane
  - role: worker
EOF
kind create cluster --name hermes --config kind-hermes.yaml --image kindest/node:v1.37.0
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.33.1/manifests/calico.yaml >/dev/null
kubectl -n kube-system rollout status ds/calico-node --timeout=300s
kubectl wait --for=condition=Ready nodes --all --timeout=180s
```

**Verify:** both nodes are `Ready`.

```bash
kubectl get nodes
```

```text
NAME                   STATUS   ROLES           AGE   VERSION
hermes-control-plane   Ready    control-plane   2m    v1.37.0
hermes-worker          Ready    <none>          2m    v1.37.0
```

## Step 2: Install Envoy Gateway and cert-manager

Users reach their agent over HTTPS through the [Gateway API](https://gateway-api.sigs.k8s.io/). **Envoy Gateway** implements it and installs the Gateway API CRDs with its chart. I picked the Gateway API over an Ingress controller on purpose: the community ingress-nginx project was archived in March 2026 ([what that meant in practice](https://ashkmb.com/blog/ingress-nginx-archived-upstream/)). **cert-manager** issues the TLS certificate.

```bash
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.10.0 \
  -n envoy-gateway-system --create-namespace --wait >/dev/null
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 \
  -n cert-manager --create-namespace --set crds.enabled=true --wait >/dev/null
```

**Verify:** both controllers are running and the Gateway API CRDs exist.

```bash
kubectl get deploy -n envoy-gateway-system envoy-gateway -o jsonpath='{.status.readyReplicas}{"\n"}'
kubectl get deploy -n cert-manager cert-manager -o jsonpath='{.status.readyReplicas}{"\n"}'
kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'
```

```text
1
1
v1.6.1
```

## Step 3: Write the per-user Helm chart

The chart creates everything one user needs inside their namespace. Two facts about the official image shape it.

**The image starts as root and then drops to an unprivileged user.** Its init system (s6-overlay) fixes ownership of the data volume, then runs every Hermes process as user `hermes` (UID 10000). Setting `runAsNonRoot` breaks that start, and the image refuses an arbitrary `--user`. So the pod keeps the default entrypoint, but with privilege escalation off, every Linux capability dropped except the five the init needs, and the default seccomp profile.

**The model is pinned in Hermes' own config.** The gateway reads the model from `config.yaml` on the volume. The image's default is a large, expensive model, and an environment variable does not override it for the gateway. An init container runs `hermes config set` on every start, so the Helm value stays the source of truth.

The files:

```bash
mkdir -p charts/hermes-user/templates
cat > charts/hermes-user/Chart.yaml <<'EOF'
apiVersion: v2
name: hermes-user
description: One isolated Hermes Agent for one user
type: application
version: 0.1.0
appVersion: "v2026.9.24"
EOF
cat > charts/hermes-user/values.yaml <<'EOF'
# The user this release belongs to. Used in labels and the hostname.
user: ""

image:
  repository: nousresearch/hermes-agent
  tag: v2026.9.24
  pullPolicy: IfNotPresent

# Model the agent uses, through OpenRouter.
model: google/gemini-2.5-flash

# Secret (created outside Helm) holding OPENROUTER_API_KEY, API_SERVER_KEY,
# HERMES_DASHBOARD_BASIC_AUTH_PASSWORD and HERMES_DASHBOARD_BASIC_AUTH_SECRET.
existingSecret: hermes-secrets

storage:
  size: 5Gi
  storageClassName: ""   # empty = cluster default

resources:
  requests: { cpu: 250m, memory: 512Mi }
  limits: { cpu: "2", memory: 2Gi }

quota:
  cpu: "2"
  memory: 4Gi
  storage: 20Gi

gateway:
  enabled: false
  name: hermes
  namespace: hermes-gateway
  domain: hermes.example.com   # the user gets <user>.<domain>
EOF
cat > charts/hermes-user/templates/_helpers.tpl <<'EOF'
{{- define "hermes.labels" -}}
app.kubernetes.io/name: hermes
app.kubernetes.io/instance: {{ .Release.Name }}
hermes.example.com/user: {{ required "set --set user=<name>" .Values.user | quote }}
{{- end }}
{{- define "hermes.selector" -}}
app.kubernetes.io/name: hermes
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
EOF
```

The ServiceAccount mounts no token, so the agent's shell has no credentials for the Kubernetes API. The volume is kept when the release is uninstalled, because it holds the user's memory:

```bash
cat > charts/hermes-user/templates/serviceaccount.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
automountServiceAccountToken: false
EOF
cat > charts/hermes-user/templates/pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hermes-data
  labels: {{- include "hermes.labels" . | nindent 4 }}
  annotations:
    helm.sh/resource-policy: keep   # uninstalling the release keeps the user's memory
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: {{ .Values.storage.size }}
  {{- with .Values.storage.storageClassName }}
  storageClassName: {{ . }}
  {{- end }}
EOF
```

The Deployment runs the gateway with the OpenAI-compatible API server (port 8642) and the web dashboard (port 9119) in one pod. The dashboard refuses to start on a non-loopback address without an auth provider, so it gets the built-in username and password provider:

```bash
cat > charts/hermes-user/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  replicas: 1
  strategy:
    type: Recreate          # the volume is ReadWriteOnce: never two pods at once
  selector:
    matchLabels: {{- include "hermes.selector" . | nindent 6 }}
  template:
    metadata:
      labels: {{- include "hermes.labels" . | nindent 8 }}
    spec:
      serviceAccountName: hermes
      automountServiceAccountToken: false
      securityContext:
        seccompProfile: { type: RuntimeDefault }
      initContainers:
        # Pin the model in the user's config.yaml on every start, so the Helm value is the source of truth.
        - name: set-model
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: ["sh", "-c"]
          args:
            - |
              hermes config set model.provider openrouter &&
              hermes config set model.default "$MODEL"
          env:
            - { name: MODEL, value: {{ .Values.model | quote }} }
            - { name: HERMES_DOCKER_EXEC_AS_ROOT, value: "1" }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
              add: [CHOWN, DAC_OVERRIDE, FOWNER]
          volumeMounts:
            - { name: data, mountPath: /opt/data }
      containers:
        - name: hermes
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          args: ["gateway", "run"]
          env:
            - { name: API_SERVER_ENABLED, value: "true" }
            - { name: API_SERVER_HOST, value: "0.0.0.0" }
            - { name: API_SERVER_MODEL_NAME, value: {{ printf "hermes-%s" .Values.user | quote }} }
            - { name: HERMES_DASHBOARD, value: "1" }
            - { name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME, value: {{ .Values.user | quote }} }
          envFrom:
            - secretRef:
                name: {{ .Values.existingSecret }}
          ports:
            - { name: api, containerPort: 8642 }
            - { name: dashboard, containerPort: 9119 }
          readinessProbe:
            httpGet: { path: /health, port: api }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /health, port: api }
            initialDelaySeconds: 60
            periodSeconds: 30
          resources: {{- toYaml .Values.resources | nindent 12 }}
          securityContext:
            # The image's init starts as root, fixes ownership of /opt/data,
            # then runs every Hermes process as the unprivileged user hermes (UID 10000).
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
              add: [CHOWN, DAC_OVERRIDE, FOWNER, SETUID, SETGID]
          volumeMounts:
            - { name: data, mountPath: /opt/data }
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: hermes-data
EOF
cat > charts/hermes-user/templates/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  selector: {{- include "hermes.selector" . | nindent 4 }}
  ports:
    - { name: api, port: 8642, targetPort: api }
    - { name: dashboard, port: 9119, targetPort: dashboard }
EOF
```

A quota caps what one user can consume, so one runaway agent cannot starve the others:

```bash
cat > charts/hermes-user/templates/quota.yaml <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  hard:
    limits.cpu: {{ .Values.quota.cpu | quote }}
    limits.memory: {{ .Values.quota.memory }}
    requests.storage: {{ .Values.quota.storage }}
    persistentvolumeclaims: "2"
    pods: "4"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  limits:
    - type: Container
      default: { cpu: 500m, memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 128Mi }
EOF
```

**Verify:** the chart lints cleanly.

```bash
helm lint ./charts/hermes-user --set user=alice | tail -1
```

```text
1 chart(s) linted, 0 chart(s) failed
```

## Step 4: Add the first user

Each user gets a namespace (labelled as a tenant, which the gateway uses in Step 7) and a Secret with four values: the OpenRouter key, a random API key for their agent, and a random dashboard password plus session-signing secret. The Secret is created outside Helm, so the keys never land in a values file or in Helm's release history.

To keep the example short, every user gets the same OpenRouter key. Each agent can read the keys in its own environment, so on a real team give each person their own OpenRouter key (see Hardening).

```bash
u=alice
kubectl create namespace hermes-$u
kubectl label namespace hermes-$u hermes.example.com/tenant=true
kubectl -n hermes-$u create secret generic hermes-secrets \
  --from-literal=OPENROUTER_API_KEY="$OPENROUTER_API_KEY" \
  --from-literal=API_SERVER_KEY="$(openssl rand -hex 32)" \
  --from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD="$(openssl rand -hex 16)" \
  --from-literal=HERMES_DASHBOARD_BASIC_AUTH_SECRET="$(openssl rand -hex 32)"
helm install hermes ./charts/hermes-user -n hermes-$u --set user=$u --wait --timeout 10m >/dev/null
```

The first start pulls a large image, so `--wait` can take a few minutes.

**Verify:** the pod is running and the init container pinned the model.

```bash
kubectl -n hermes-alice get pods
kubectl -n hermes-alice logs deploy/hermes -c set-model
```

```text
NAME                      READY   STATUS    RESTARTS   AGE
hermes-6d9f8b7c4d-x2k8p   1/1     Running   0          3m
✓ Set model.provider = openrouter in /opt/data/config.yaml
✓ Set model.default = google/gemini-2.5-flash in /opt/data/config.yaml
```

## Step 5: Talk to your agent

Before any public hostname exists, reach Alice's agent through a port-forward. Her API key is in her Secret:

```bash
kubectl -n hermes-alice port-forward svc/hermes 8642:8642 >/dev/null &
PF=$!
sleep 3
ALICE_KEY=$(kubectl -n hermes-alice get secret hermes-secrets -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)
curl -s localhost:8642/health; echo
curl -s -o /dev/null -w 'without a key: %{http_code}\n' localhost:8642/v1/models
```

```text
{"status": "ok", "platform": "hermes-agent", "version": "0.21.5"}
without a key: 401
```

Now a real request. The API speaks the OpenAI chat format, so any OpenAI-compatible client works too:

```bash
curl -s localhost:8642/v1/chat/completions \
  -H "Authorization: Bearer $ALICE_KEY" -H 'Content-Type: application/json' \
  -d '{"model": "hermes-alice", "messages": [{"role": "user", "content": "Reply with exactly: hello from alice"}]}' \
  | python3 -c 'import sys, json; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
kill $PF
```

**Verify:** the agent answers through the model.

```text
hello from alice
```

The dashboard works the same way on port 9119. Her username is `alice`, and the password is `HERMES_DASHBOARD_BASIC_AUTH_PASSWORD` in her Secret.

## Step 6: Lock down the network

Right now any pod in the cluster can reach Alice's agent, and her agent can reach anything. Three policies fix that: deny everything, then allow DNS and HTTPS to the public internet out, and allow only the gateway's proxies in.

The outbound rule is deliberately broad. Hermes is a general assistant whose web tools need the open internet, and a standard NetworkPolicy cannot match hostnames anyway. What it does exclude matters more: every private range, so no other tenant's pod, no cluster-internal service address and no cloud metadata endpoint (`169.254.169.254`, where node credentials live on most clouds).

Two things this policy does not stop. Through the public Gateway, one user's agent can reach another user's hostname like any client on the internet, so there the per-user API key and dashboard password are what protect it. And if your cluster's API server has a public endpoint, the agent can reach it on port 443; the missing token and RBAC are what stop it there. To close the first gap, add your Gateway's load balancer address to the `except` list.

```bash
cat >> charts/hermes-user/values.yaml <<'EOF'

networkPolicy:
  enabled: true
  # Namespace of the gateway's proxy pods, the only allowed inbound source.
  proxyNamespace: envoy-gateway-system
EOF
cat > charts/hermes-user/templates/networkpolicy.yaml <<'EOF'
{{- if .Values.networkPolicy.enabled }}
# 1. Nothing in or out by default.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. Out: DNS, and HTTPS to the public internet only (model API, web tools).
#    Private ranges are excluded: no other tenant, no cluster service, no cloud metadata.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hermes-egress
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  podSelector:
    matchLabels: {{- include "hermes.selector" . | nindent 6 }}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16]
      ports:
        - { protocol: TCP, port: 443 }
---
# 3. In: only from the gateway's proxies.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hermes-ingress
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  podSelector:
    matchLabels: {{- include "hermes.selector" . | nindent 6 }}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: {{ .Values.networkPolicy.proxyNamespace }} }
      ports:
        - { protocol: TCP, port: 8642 }
        - { protocol: TCP, port: 9119 }
{{- end }}
EOF
helm upgrade hermes ./charts/hermes-user -n hermes-alice --set user=alice --wait >/dev/null
```

**Verify:** the policies exist, the agent still reaches the internet, and the metadata address times out (`curl` exit code 28).

```bash
kubectl -n hermes-alice get networkpolicy
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
  curl -s -m 8 -o /dev/null -w 'public HTTPS: %{http_code}\n' https://openrouter.ai/api/v1/models
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
  sh -c 'curl -s -m 5 http://169.254.169.254/ || echo "metadata: blocked (exit $?)"'
```

```text
NAME             POD-SELECTOR                                                      AGE
default-deny     <none>                                                            5s
hermes-egress    app.kubernetes.io/instance=hermes,app.kubernetes.io/name=hermes   5s
hermes-ingress   app.kubernetes.io/instance=hermes,app.kubernetes.io/name=hermes   5s
public HTTPS: 200
metadata: blocked (exit 28)
```

## Step 7: Expose each user on their own hostname

One shared Gateway terminates TLS for `*.hermes.example.com` and accepts routes only from namespaces labelled as tenants. Each user's release adds an HTTPRoute for their own hostname: `/v1` goes to the API, everything else to the dashboard.

For practice, cert-manager creates a small private CA and signs the wildcard certificate with it. **In production**, replace the `hermes-ca` ClusterIssuer with Let's Encrypt and a DNS-01 solver (a wildcard certificate needs DNS-01), and point `*.hermes.example.com` at the Gateway's address.

```bash
mkdir -p platform
cat > platform/gateway.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: v1
kind: Namespace
metadata:
  name: hermes-gateway
---
# Practice CA: a self-signed root that signs the wildcard certificate.
# In production, replace the "hermes-ca" ClusterIssuer with Let's Encrypt and a DNS-01 solver
# (a wildcard certificate needs DNS-01).
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hermes-practice-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: hermes-practice-ca
  secretName: hermes-practice-ca
  privateKey: { algorithm: ECDSA, size: 256 }
  issuerRef: { name: selfsigned, kind: ClusterIssuer }
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: hermes-ca
spec:
  ca:
    secretName: hermes-practice-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hermes-wildcard
  namespace: hermes-gateway
spec:
  secretName: hermes-wildcard-tls
  commonName: "*.hermes.example.com"
  dnsNames: ["*.hermes.example.com"]
  issuerRef: { name: hermes-ca, kind: ClusterIssuer }
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: hermes
  namespace: hermes-gateway
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.hermes.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: hermes-wildcard-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              hermes.example.com/tenant: "true"
EOF
kubectl apply -f platform/gateway.yaml
kubectl -n hermes-gateway wait --for=condition=Ready certificate/hermes-wildcard --timeout=120s
kubectl -n hermes-gateway wait --for=condition=Accepted gateway/hermes --timeout=120s
```

Now the route template, and Alice's release with the gateway turned on:

```bash
cat > charts/hermes-user/templates/httproute.yaml <<'EOF'
{{- if .Values.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: hermes
  labels: {{- include "hermes.labels" . | nindent 4 }}
spec:
  parentRefs:
    - name: {{ .Values.gateway.name }}
      namespace: {{ .Values.gateway.namespace }}
      sectionName: https
  hostnames:
    - {{ printf "%s.%s" .Values.user .Values.gateway.domain | quote }}
  rules:
    # OpenAI-compatible API
    - matches:
        - path: { type: PathPrefix, value: /v1 }
      backendRefs:
        - { name: hermes, port: 8642 }
    # Everything else: the web dashboard
    - backendRefs:
        - { name: hermes, port: 9119 }
{{- end }}
EOF
helm upgrade hermes ./charts/hermes-user -n hermes-alice --set user=alice --set gateway.enabled=true --wait >/dev/null
```

On a cloud cluster the Gateway gets a load balancer address (`kubectl -n hermes-gateway get gateway hermes`). kind has no load balancer, so the Gateway stays `Programmed=False` with reason `AddressNotAssigned`, and you reach the proxy through a port-forward instead. `curl --resolve` sends the real hostname, so TLS and routing are exercised exactly as in production:

```bash
kubectl -n envoy-gateway-system wait --for=condition=Ready pod \
  -l gateway.envoyproxy.io/owning-gateway-name=hermes --timeout=180s >/dev/null
kubectl -n hermes-gateway get secret hermes-wildcard-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > hermes-ca.crt
GW_SVC=$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=hermes -o name)
kubectl -n envoy-gateway-system port-forward "$GW_SVC" 8443:443 >/dev/null &
GW_PF=$!
sleep 3
hcurl() { curl -s --cacert hermes-ca.crt --resolve "$1.hermes.example.com:8443:127.0.0.1" "${@:2}"; }
hcurl alice https://alice.hermes.example.com:8443/v1/chat/completions \
  -H "Authorization: Bearer $ALICE_KEY" -H 'Content-Type: application/json' \
  -d '{"model": "hermes-alice", "messages": [{"role": "user", "content": "Reply with exactly: hello over https"}]}' \
  | python3 -c 'import sys, json; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
hcurl alice -o /dev/null -w 'dashboard: %{http_code}\n' https://alice.hermes.example.com:8443/
```

**Verify:** the API answers over HTTPS, and the dashboard redirects to its login page.

```text
hello over https
dashboard: 302
```

## Step 8: Add more users

Steps 4 and 7 for one user fit in a small script. It is safe to re-run: it keeps an existing Secret, so a user's keys do not change.

```bash
cat > add-user.sh <<'EOF'
#!/usr/bin/env bash
# Usage: ./add-user.sh <user>   (needs OPENROUTER_API_KEY in the environment)
set -euo pipefail
u="$1"
ns="hermes-$u"
kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace "$ns" hermes.example.com/tenant=true --overwrite
if ! kubectl -n "$ns" get secret hermes-secrets >/dev/null 2>&1; then
  kubectl -n "$ns" create secret generic hermes-secrets \
    --from-literal=OPENROUTER_API_KEY="$OPENROUTER_API_KEY" \
    --from-literal=API_SERVER_KEY="$(openssl rand -hex 32)" \
    --from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD="$(openssl rand -hex 16)" \
    --from-literal=HERMES_DASHBOARD_BASIC_AUTH_SECRET="$(openssl rand -hex 32)"
fi
helm upgrade --install hermes ./charts/hermes-user -n "$ns" \
  --set user="$u" --set gateway.enabled=true --wait --timeout 10m
EOF
chmod +x add-user.sh
./add-user.sh bob >/dev/null
```

**Verify:** two agents, one per namespace.

```bash
kubectl get pods -A -l app.kubernetes.io/name=hermes
```

```text
NAMESPACE      NAME                      READY   STATUS    RESTARTS   AGE
hermes-alice   hermes-7c5d9b8f6-q4wzl    1/1     Running   0          6m
hermes-bob     hermes-5f8b6c7d9-m2xkj    1/1     Running   0          1m
```

To remove a user, uninstall the release and delete the namespace. The `keep` policy on the volume means `helm uninstall` alone keeps their memory, and deleting the namespace removes it for good:

```bash no-run
helm uninstall hermes -n hermes-bob
kubectl delete namespace hermes-bob
```

## Step 9: Prove the isolation

Five checks, each one a boundary that matters.

**1. Alice's agent cannot reach Bob's.** Her egress policy blocks private addresses, and his ingress policy accepts only the gateway. `curl` gives up after 5 seconds (exit code 28):

```bash
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
  sh -c 'curl -s -m 5 http://hermes.hermes-bob:8642/health || echo "alice -> bob: blocked (exit $?)"'
```

**2. Nothing else in the cluster can reach Bob's agent either**, not even a pod in a namespace with no policies:

```bash
kubectl run probe -n default --rm -i --restart=Never --quiet --image=curlimages/curl:8.17.0 --command -- \
  sh -c 'curl -s -m 5 http://hermes.hermes-bob:8642/health || echo "default -> bob: blocked (exit $?)"'
```

**3. Alice's key does not open Bob's agent:**

```bash
hcurl bob -o /dev/null -w 'alice key on bob: %{http_code}\n' \
  -H "Authorization: Bearer $ALICE_KEY" https://bob.hermes.example.com:8443/v1/models
```

**4. The agent has no Kubernetes credentials.** No token is mounted, and the service account may not read anything:

```bash
kubectl -n hermes-alice exec deploy/hermes -c hermes -- \
  sh -c 'ls /var/run/secrets/kubernetes.io/serviceaccount 2>/dev/null || echo "no service account token"'
kubectl auth can-i get secrets -n hermes-bob --as=system:serviceaccount:hermes-alice:hermes || true
```

**5. Bob's own key still works for Bob:**

```bash
BOB_KEY=$(kubectl -n hermes-bob get secret hermes-secrets -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)
hcurl bob -o /dev/null -w 'bob key on bob: %{http_code}\n' \
  -H "Authorization: Bearer $BOB_KEY" https://bob.hermes.example.com:8443/v1/models
kill $GW_PF
```

**Verify:** all five, in order.

```text
alice -> bob: blocked (exit 28)
default -> bob: blocked (exit 28)
alice key on bob: 401
no service account token
no
bob key on bob: 200
```

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| Network policies exist but Step 9 check 1 or 2 connects | The CNI does not enforce NetworkPolicy (kind's default kindnet, some minimal setups) | Use Calico or Cilium, as in Step 1 |
| Pod exits at start with `[stage2] ERROR: container started with --user` | `runAsUser`/`runAsNonRoot` set on the pod | Remove them; the image's init must start as root and drops to UID 10000 itself (Step 3) |
| Agent answers, but your OpenRouter bill shows an expensive model | The gateway reads the model from `config.yaml`, not from `HERMES_INFERENCE_MODEL` | Keep the `set-model` init container; check with `kubectl exec deploy/hermes -c hermes -- hermes config get model.default` |
| Gateway shows `Programmed=False`, reason `AddressNotAssigned` | No load balancer (kind, bare metal) | Expected on kind; use the port-forward in Step 7, or install MetalLB or cloud-provider-kind |
| `curl: (60)` or macOS curl `couldn't get X509-issuer name` | The client does not trust the practice CA, or the certificate is a subject-less self-signed leaf | Pass `--cacert hermes-ca.crt`; keep the CA issuer from Step 7 instead of signing the wildcard with `selfSigned` directly |
| API returns `401` with the right key | An `API_SERVER_KEY` line in `/opt/data/.env` wins over the pod environment (the logs warn about it) | Remove the line from `.env` on the volume, or rotate the key in the Secret and restart |
| Dashboard does not come up, logs mention a missing auth provider | The dashboard fails closed on a non-loopback bind without auth | Keep the `HERMES_DASHBOARD_BASIC_AUTH_*` values, or configure OIDC |
| The agent cannot resolve any hostname after Step 6 | The cluster runs NodeLocal DNSCache (`169.254.20.10`), which is inside the excluded `169.254.0.0/16` range | Add an egress rule for `169.254.20.10/32` on port 53 UDP and TCP |
| `helm install --wait` times out on the first user | The image is large and the first pull is slow | Raise `--timeout`; check `kubectl -n hermes-alice describe pod` for pull progress |

## Hardening for production

- **Restrict egress further** if your users do not need the open web. Standard NetworkPolicy matches IP ranges only. Cilium's `toFQDNs` rules can limit egress to `openrouter.ai` by name.
- **Use OIDC for the dashboard** instead of the shared username and password. Hermes supports a self-hosted OpenID Connect provider (`HERMES_DASHBOARD_OIDC_ISSUER`, `HERMES_DASHBOARD_OIDC_CLIENT_ID`). Exposed, unauthenticated Hermes dashboards and API servers were scanned and abused on the internet in June 2026, which is why the dashboard now refuses to start without auth. Treat every agent endpoint as a remote shell, because that is what it is.
- **One OpenRouter key per user**, each with its own credit limit. A leaked key then exposes one person, one person's runaway loop cannot drain everyone's budget, and spend is visible per person.
- **Back up the volumes.** `/opt/data` holds each user's memory, sessions and skills. A volume snapshot schedule (or Velero) is the difference between a pod restart and a lost assistant.
- **An operator instead of Helm.** A community Hermes operator exists on OperatorHub. A plain Helm release per user is easier to read and debug, which is why this tutorial uses one.

## Cleanup

Delete the practice cluster and everything in it:

```bash
cd ..
kind delete cluster --name hermes
```
