fix(hermes): give the store of generated secrets an account of its own namespace #397

Merged

grachevko merged 1 commit from fix/hermes-store-kubernetes into main 2026-10-12 16:04:28 +00:00

Conversation 3 Commits 1 Files changed 2 +75 -14

hermes commented 2026-10-12 14:54:34 +00:00

Collaborator

• Copy link
• Copy Markdown

What was wrong

Both secrets Kustomizations failed at admission and never became ready, so env and hermes waited on them — the ExternalSecrets of the mounted .env were never created and the new specification of the pod was never applied (that is why the pods did not roll). The operator's webhook named the rule:

SecretStore/grachevaeu/kubernetes dry-run failed (Forbidden): admission webhook
"validate.secretstore.external-secrets.io" denied the request: namespace should either be empty or
match the namespace of the SecretStore for a namespaced SecretStore

The store of the previous change named an account in security — the namespace of the operator — while a namespaced store may authenticate only as an account of its own namespace, and by design never as the operator's mounted one.

What this fixes

• kubernetes/apps/{grachevaeu,grachevko}/hermes/secrets/app/secretstore-kubernetes.yaml — the account the store authenticates as moves into the namespace of the store: a ServiceAccount of the instance, a Role with the two permissions the operator documents for a store (read the secrets of that namespace — neither list nor watch can be narrowed by resourceNames — and review its own rules through SelfSubjectRulesReview) and a RoleBinding that binds them. The SecretStore itself keeps the provider, the pinned remoteNamespace and the CA of the API server; it now reads exactly the namespace it belongs to.
• nothing in the application changes: the ExternalSecret still reads hermes-api-key through the store named kubernetes, and the mount, the Secret name and the generator stay as they are.

Verified

• flate test all -p ./kubernetes/flux/cluster — 308 passed.
• flate build ks secrets -p ./kubernetes/flux/cluster -n grachevaeu and -n grachevko render ServiceAccount, Role, RoleBinding, SecretStore and both ExternalSecrets of the instance.
• the rule is taken from the operator's own webhook message (quoted above), and the account in the same namespace is the shape its documentation prescribes for access within one cluster.

After the merge

1. kv/grachevaeu/hermes must hold the keys under the variable names — DEEPSEEK_API_KEY, TELEGRAM_BOT_TOKEN, ... It still holds deepseek_key and telegram_bot_token; once the store works, the ExternalSecret of that instance stays in SecretSyncedError until the record is rewritten (the sync repeats every minute, so the fix needs no nudge):

   vault kv get -tls-skip-verify -format=json kv/grachevaeu/hermes \
     | jq -c '.data.data | .DEEPSEEK_API_KEY = .deepseek_key | .TELEGRAM_BOT_TOKEN = .telegram_bot_token | del(.deepseek_key, .telegram_bot_token)' \
     | vault kv put -tls-skip-verify kv/grachevaeu/hermes -
   

2. Force the two waiting Kustomizations once, instead of waiting for the hourly interval: flux reconcile kustomization secrets -n grachevaeu (then env, then hermes, and the same for grachevko); or wait for the interval.
3. Delete the leftover Secret hermes-keys in grachevaeu (nothing creates it any more).

Rollback

git revert of the merge commit: the previous store shape is rejected at admission, so a rollback returns to the state where secrets/env do not reconcile and the pods keep their old specification — the running pods are unaffected either way.

## What was wrong Both `secrets` Kustomizations failed at admission and never became ready, so `env` and `hermes` waited on them — the ExternalSecrets of the mounted `.env` were never created and the new specification of the pod was never applied (that is why the pods did not roll). The operator's webhook named the rule: ``` SecretStore/grachevaeu/kubernetes dry-run failed (Forbidden): admission webhook "validate.secretstore.external-secrets.io" denied the request: namespace should either be empty or match the namespace of the SecretStore for a namespaced SecretStore ``` The store of the previous change named an account in `security` — the namespace of the operator — while a namespaced store may authenticate only as an account of its own namespace, and by design never as the operator's mounted one. ## What this fixes - `kubernetes/apps/{grachevaeu,grachevko}/hermes/secrets/app/secretstore-kubernetes.yaml` — the account the store authenticates as moves into the namespace of the store: a `ServiceAccount` of the instance, a `Role` with the two permissions the operator documents for a store (read the secrets of that namespace — neither `list` nor `watch` can be narrowed by `resourceNames` — and review its own rules through `SelfSubjectRulesReview`) and a `RoleBinding` that binds them. The `SecretStore` itself keeps the provider, the pinned `remoteNamespace` and the CA of the API server; it now reads exactly the namespace it belongs to. - nothing in the application changes: the ExternalSecret still reads `hermes-api-key` through the store named `kubernetes`, and the mount, the Secret name and the generator stay as they are. ## Verified - `flate test all -p ./kubernetes/flux/cluster` — 308 passed. - `flate build ks secrets -p ./kubernetes/flux/cluster -n grachevaeu` and `-n grachevko` render `ServiceAccount`, `Role`, `RoleBinding`, `SecretStore` and both `ExternalSecret`s of the instance. - the rule is taken from the operator's own webhook message (quoted above), and the account in the same namespace is the shape its documentation prescribes for access within one cluster. ## After the merge 1. `kv/grachevaeu/hermes` must hold the keys under the variable names — `DEEPSEEK_API_KEY`, `TELEGRAM_BOT_TOKEN`, ... It still holds `deepseek_key` and `telegram_bot_token`; once the store works, the ExternalSecret of that instance stays in `SecretSyncedError` until the record is rewritten (the sync repeats every minute, so the fix needs no nudge): ``` vault kv get -tls-skip-verify -format=json kv/grachevaeu/hermes \ | jq -c '.data.data | .DEEPSEEK_API_KEY = .deepseek_key | .TELEGRAM_BOT_TOKEN = .telegram_bot_token | del(.deepseek_key, .telegram_bot_token)' \ | vault kv put -tls-skip-verify kv/grachevaeu/hermes - ``` 2. Force the two waiting Kustomizations once, instead of waiting for the hourly interval: `flux reconcile kustomization secrets -n grachevaeu` (then `env`, then `hermes`, and the same for `grachevko`); or wait for the interval. 3. Delete the leftover Secret `hermes-keys` in `grachevaeu` (nothing creates it any more). ## Rollback `git revert` of the merge commit: the previous store shape is rejected at admission, so a rollback returns to the state where `secrets`/`env` do not reconcile and the pods keep their old specification — the running pods are unaffected either way.

hermes added 1 commit 2026-10-12 14:54:34 +00:00

fix(hermes): give the store of generated secrets an account of its own namespace ... 15aad36923

The Kubernetes provider of external-secrets refused the store of both instances at admission:

  SecretStore/grachevaeu/kubernetes dry-run failed (Forbidden): admission webhook
  "validate.secretstore.external-secrets.io" denied the request: namespace should either be empty or
  match the namespace of the SecretStore for a namespaced SecretStore

The store named an account in `security`, the namespace of the operator, while a namespaced store may
authenticate only as an account of its own namespace - and by design never as the operator's mounted
one. So the Kustomization `secrets` never became ready, `env` and `hermes` waited on it, and neither
the ExternalSecrets of the mounted .env nor the new specification of the pod were ever applied.

The store keeps its namespace; the account moves into it: a ServiceAccount of the instance with a
Role for the two permissions the operator documents for a store (read the secrets of that namespace -
neither list nor watch can be narrowed by resourceNames - and review its own rules) and a RoleBinding
that binds them. The store then reads exactly the namespace it belongs to, which is what its
`remoteNamespace` says.

Nothing in the application changes: the ExternalSecret still reads `hermes-api-key` through the store
named `kubernetes`.

pr-agent commented 2026-10-12 14:55:19 +00:00

Collaborator

• Copy link
• Copy Markdown

Failed to update PR description

Failed to update PR description

pr-agent commented 2026-10-12 14:56:51 +00:00

Collaborator

• Copy link
• Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ No major issues detected

## PR Reviewer Guide 🔍 Here are some key observations to aid the review process: <table> <tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 2 🔵🔵⚪⚪⚪</td></tr> <tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr> <tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr> <tr><td>⚡&nbsp;<strong>No major issues detected</strong></td></tr> </table>

pr-agent commented 2026-10-12 15:01:17 +00:00

Collaborator

• Copy link
• Copy Markdown

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Move selfsubjectrulesreviews to a ClusterRole selfsubjectrulesreviews is a cluster-scoped authorization.k8s.io resource, so it cannot be granted by a namespaced Role. Keep the namespace-scoped secret access in the Role, and move the self-subject rules review permission to a ClusterRole bound to the ServiceAccount with a ClusterRoleBinding that also covers the grachevko instance. kubernetes/apps/grachevaeu/hermes/secrets/app/secretstore-kubernetes.yaml [19-29] apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-reader rules: - apiGroups: [""] resources: [secrets] verbs: [get, list, watch] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: secrets-reader-selfsubjectrulesreview +rules: - apiGroups: [authorization.k8s.io] resources: [selfsubjectrulesreviews] verbs: [create] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: secrets-reader-selfsubjectrulesreview +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: secrets-reader-selfsubjectrulesreview +subjects: + - kind: ServiceAccount + name: secrets-reader + namespace: grachevaeu + - kind: ServiceAccount + name: secrets-reader + namespace: grachevko Suggestion importance[1-10]: 9 __ Why: This fixes a critical RBAC validity issue: selfsubjectrulesreviews is cluster-scoped and cannot be granted by a namespaced Role. The proposed ClusterRole and ClusterRoleBinding correctly restore the needed permission across both namespaces.	High
	Remove invalid namespaced role rule Remove the cluster-scoped selfsubjectrulesreviews rule from this namespaced Role; it belongs in a ClusterRole. The first file's ClusterRoleBinding already binds this namespace's secrets-reader ServiceAccount to the shared secrets-reader-selfsubjectrulesreview ClusterRole. kubernetes/apps/grachevko/hermes/secrets/app/secretstore-kubernetes.yaml [10-20] apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-reader rules: - apiGroups: [""] resources: [secrets] verbs: [get, list, watch] - - apiGroups: [authorization.k8s.io] - resources: [selfsubjectrulesreviews] - verbs: [create] Suggestion importance[1-10]: 6 __ Why: The removal is correct because selfsubjectrulesreviews cannot be placed in a namespaced Role, but this change relies on a ClusterRoleBinding that is not present in the current diff. Applied alone, it would drop the required permission.	Low

## PR Code Suggestions ✨ <!-- pr-agent:improve:summary --> <!-- 15aad36 --> Explore these optional code suggestions: <table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=2>Possible issue</td> <td> <details><summary>Move selfsubjectrulesreviews to a ClusterRole</summary> ___ **<code>selfsubjectrulesreviews</code> is a cluster-scoped authorization.k8s.io resource, so it <br>cannot be granted by a namespaced <code>Role</code>. Keep the namespace-scoped secret access in <br>the <code>Role</code>, and move the self-subject rules review permission to a <code>ClusterRole</code> bound <br>to the ServiceAccount with a <code>ClusterRoleBinding</code> that also covers the grachevko <br>instance.** [kubernetes/apps/grachevaeu/hermes/secrets/app/secretstore-kubernetes.yaml [19-29]](https://forgejo.grachevko.ru/grachevko/home-ops/src/branch/fix/hermes-store-kubernetes/kubernetes/apps/grachevaeu/hermes/secrets/app/secretstore-kubernetes.yaml#L19-L29) ```diff apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-reader rules: - apiGroups: [""] resources: [secrets] verbs: [get, list, watch] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: secrets-reader-selfsubjectrulesreview +rules: - apiGroups: [authorization.k8s.io] resources: [selfsubjectrulesreviews] verbs: [create] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: secrets-reader-selfsubjectrulesreview +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: secrets-reader-selfsubjectrulesreview +subjects: + - kind: ServiceAccount + name: secrets-reader + namespace: grachevaeu + - kind: ServiceAccount + name: secrets-reader + namespace: grachevko ``` <details><summary>Suggestion importance[1-10]: 9</summary> __ Why: This fixes a critical RBAC validity issue: `selfsubjectrulesreviews` is cluster-scoped and cannot be granted by a namespaced `Role`. The proposed `ClusterRole` and `ClusterRoleBinding` correctly restore the needed permission across both namespaces. </details></details></td><td align=center>High </td></tr><tr><td> <details><summary>Remove invalid namespaced role rule</summary> ___ **Remove the cluster-scoped <code>selfsubjectrulesreviews</code> rule from this namespaced <code>Role</code>; it <br>belongs in a <code>ClusterRole</code>. The first file's <code>ClusterRoleBinding</code> already binds this <br>namespace's <code>secrets-reader</code> ServiceAccount to the shared <br><code>secrets-reader-selfsubjectrulesreview</code> ClusterRole.** [kubernetes/apps/grachevko/hermes/secrets/app/secretstore-kubernetes.yaml [10-20]](https://forgejo.grachevko.ru/grachevko/home-ops/src/branch/fix/hermes-store-kubernetes/kubernetes/apps/grachevko/hermes/secrets/app/secretstore-kubernetes.yaml#L10-L20) ```diff apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-reader rules: - apiGroups: [""] resources: [secrets] verbs: [get, list, watch] - - apiGroups: [authorization.k8s.io] - resources: [selfsubjectrulesreviews] - verbs: [create] ``` <details><summary>Suggestion importance[1-10]: 6</summary> __ Why: The removal is correct because `selfsubjectrulesreviews` cannot be placed in a namespaced `Role`, but this change relies on a `ClusterRoleBinding` that is not present in the current diff. Applied alone, it would drop the required permission. </details></details></td><td align=center>Low </td></tr></tr></tbody></table>

grachevko merged commit 2162806be7 into main 2026-10-12 16:04:28 +00:00

grachevko deleted branch fix/hermes-store-kubernetes 2026-10-12 16:04:28 +00:00

grachevko referenced this pull request from a commit 2026-10-12 16:04:29 +00:00

fix(hermes): give the store of generated secrets an account of its own namespace (#397)

hermes referenced this pull request 2026-10-12 16:26:53 +00:00

fix(hermes): give the source override the shape External Secrets 2.12.0 accepts #399

hermes referenced this pull request 2026-10-12 16:28:14 +00:00

CI: валидировать custom resources по схемам CRD развёрнутых версий #400

hermes referenced this pull request 2026-10-12 16:28:14 +00:00

CI: проверять admission-правила ESO (namespaced SecretStore ссылается на SA своего namespace) #401

grachevko referenced this pull request from a commit 2026-10-12 18:07:06 +00:00

fix(hermes): give the source override the shape External Secrets 2.12.0 accepts (#399)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

grachevko/home-ops!397

No description provided.