fix(hermes): read the agent credentials from the managed scope #387
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/hermes-managed-scope"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was wrong
Since 0.21.6 the gateway multiplexes every profile by default (
gateway.multiplex_profilescannot be turned off) and resolves a credential only through the profile secret scope: the profile's own.envplus the managed layer.gateway/config.py::_getenvreads throughagent/secret_scope.get_secret, which fails closed while multiplexing, so a value that exists ONLY as a variable of the pod is invisible.api_serveris then never enabled, port 8642 stays closed, the readiness probe never passes and the Helm upgrade times out — that is how the 0.21.6 rollout ofgrachevaeufailed, with Flux rolling the release back every five minutes.What this fixes
kubernetes/apps/{grachevaeu,grachevko}/hermes/secrets/ks.yamland…/hermes/env/ks.yaml— new: the credentials are Kustomizations of their own, in the order this repository keeps for a CR of an operator, the Secret the CR reads and the application that reads the Secret.secrets/holds the generator of the API server key and the read point of the generated Secrets,env/holds the ExternalSecret that renders the.envand reads both, and the Kustomization of the pod depends onenv— the file it mounts has to exist before the container is created.kubernetes/apps/{grachevaeu,grachevko}/hermes/secrets/app/secretstore-kubernetes.yaml— new: the read point for the values the cluster makes itself. A namespacedSecretStoreof the kubernetes provider, constrained to one namespace, with no new permission — the operator's chart already carriessecrets:getandserviceaccounts/token:create; the CA is the one Kubernetes maintains in every namespace (kube-root-ca.crt), not the internal CA of the gateway domains.kubernetes/apps/{grachevaeu,grachevko}/hermes/env/app/externalsecret-managed-env.yaml— new: the credentials of the instance as one Secret,hermes-managed-env, rendered generically. Every property of the record of that instance in Vault becomes oneKEY=VALUEline, the property name being the variable name;HINDSIGHT_API_KEYcomes from the shared recordai/hindsight,API_SERVER_KEYfrom the generated Secret of the cluster.refreshInterval: 1m, so a key written in Vault reaches the pod within a minute and no commit is needed for a new key.kubernetes/apps/{grachevaeu,grachevko}/hermes/app/helmrelease.yaml— the Secret is mounted read-only as/etc/hermes/.env, the file of the managed scope, in the directory the image already treats as its own: noHERMES_MANAGED_DIR(the variable only relocates the same directory, and with noconfig.yamlbeside it the config half of the scope stays inert), no init container, no emptyDir.subPath, not a directory mount: a directory mount would hide the two files the image keeps in/etc/hermes(agent-browser-executable-path,image-provenance.json). The pod keeps no key of its own and nothing inside it is a source of truth: Hermes still writes its own$HERMES_HOME/.env, while the mounted file wins for every key it carries, so a stale line in the volume file changes nothing and needs no cleanup. The reloader annotation stays and now watches the Secret that carries the file.kubernetes/apps/grachevaeu/hermes/app/externalsecret-keys.yaml(deleted) — the keys now arrive as.envlines, so theenvFromSecret and that ExternalSecret are gone.externalsecret-api-key.yamlis moved, unchanged apart from its comment, intosecrets/app/: the key of the API server is generated once in the cluster and the value is read back through the kubernetes store instead of being handed to the pod as a variable — reading a Secret is not reading a generator, so the refresh cannot rotate it.Why the managed path and not
$HERMES_HOME/.envdirectly: Hermes writes that file itself (settings), a mount would freeze it, and then two files would disagree about which one wins.Verified
flate test all -p ./kubernetes/flux/cluster— 308 passed;flate build ks secrets -n grachevaeu|grachevkoandflate build ks env -n …render the objects in the right namespaces.flate build hr hermes -p ./kubernetes/flux/cluster -n grachevaeuand-n grachevkorender; in both, the app container and the pre-existingfix-runinit container carrymountPath: /etc/hermes/.envwithsubPath: .env,readOnly: true, from the Secrethermes-managed-env; noHERMES_MANAGED_DIR, noassemble-managed-env, noenvFromwith the credentials.data=API_SERVER_KEY(per-entrysourceRefto thekubernetesstore,remoteRef.key: hermes-api-key) +HINDSIGHT_API_KEY(Vault,ai/hindsight);dataFrom.extract= the record of the instance. A per-entrysourceRefis part of the API (ExternalSecretData.SourceRefin the operator's own types), which is what lets one ExternalSecret read two stores.sortAlpha), so the Secret does not change on a refresh and the reloader does not roll the pods with it; same pattern askubernetes/apps/ai/hindsight/app/externalsecret.yaml, andkeysis not usable there because ESO hands overmap[string]string.kube-root-ca.crtexists in both namespaces, andvaultis the only other store in the cluster; the account both stores authenticate as is the same operator.After the merge
kv/grachevko/hermesandkv/grachevaeu/hermesmust hold the keys under the variable names —DEEPSEEK_API_KEY,TELEGRAM_BOT_TOKEN, ...kv/grachevaeu/hermesstill holdsdeepseek_keyandtelegram_bot_token; until it is rewritten the ExternalSecret stays inSecretSyncedErrorand the instance gets noDEEPSEEK_API_KEY(the sync repeats every minute, so the fix needs no nudge): Property names only, without the values — the same read from inside the cluster, wherevault readskips the mount lookup that thekvcommands perform:kubectl -n security exec vault-0 -- vault read -format=json kv/data/grachevko/hermes | jq -r '.data.data | keys[]'hermes-keysingrachevaeu: nothing creates it any more..envstill holds is overridden by the mounted file.Rollback
git revertof the merge commit: the mount, the Secret name and the generator are unchanged frommain, so the previous shape comes back on the next reconcile.Failed to update PR description
Rendered diff of the HelmReleases in this pull request:
PR Reviewer Guide 🔍
Here are some key observations to aid the review process:
PR Code Suggestions ✨
Explore these optional code suggestions:
Исправить имя контроллера в advancedMounts
Первый уровень
advancedMountsв app-template — это имя контроллера, а не имяприложения. По видимому манифесту контроллер называется
main, а контейнер —app,поэтому ключ
hermesне совпадает: монтирования не будут созданы, init-контейнерупадёт при записи
/etc/hermes/managed/.env. Заменитеhermesнаmainво всех трёхновых записях
managed-env,managed-env-seedиmanaged-env-key; аналогично исправьтев
kubernetes/apps/grachevko/hermes/app/helmrelease.yaml.kubernetes/apps/grachevaeu/hermes/app/helmrelease.yaml [204-214]
Suggestion importance[1-10]: 9
__
Why: The suggestion correctly identifies that
advancedMountskeys must match the app-template controller name (main), not the application namehermes; usinghermeswould prevent the init container from mounting/etc/hermes/managedand break startup. This is a high-impact correctness fix, with a parallel issue in the grachevko file.976ed2041c80ab892c8880ab892c88643c826f17643c826f17a70fe57b68a70fe57b68fc5f66df9bfc5f66df9b9b0ab97d569b0ab97d568815e40aca8815e40aca3f83c3ecb03f83c3ecb09a0667df76