fix(hermes): give the source override the shape External Secrets 2.12.0 accepts #399

Merged

grachevko merged 2 commits from fix/eso-source-ref into main 2026-10-12 18:07:05 +00:00

Conversation 4 Commits 2 Files changed 2 +15 -4

hermes commented 2026-10-12 16:26:53 +00:00

Collaborator

• Copy link
• Copy Markdown

What was wrong

Both instances of the agent carry an ExternalSecret that renders their .env: most values come from Vault, one entry reads the generated key of the API server back through the store of the namespace, and that override was written flat:

    - secretKey: API_SERVER_KEY
      sourceRef:
        kind: SecretStore
        name: kubernetes

The API server refuses the object, and with it the whole rollout of the instances:

ExternalSecret/grachevko/hermes-managed-env dry-run failed: failed to create typed patch object
(grachevko/hermes-managed-env; external-secrets.io/v1, Kind=ExternalSecret):
.spec.data[0].sourceRef.kind: field not declared in schema

spec.data[].sourceRef is a StoreSourceRef: it takes exactly one of storeRef and generatorRef, and the kind and the name live inside storeRef. Written flat the object carries two properties the schema does not declare, so the env Kustomization of both namespaces stays NotReady, hermes waits on it, and the merged work of #397 and #398 does not reach the cluster at all — the file manager that now runs in the pod of the agent is waiting for the same chain.

What this fixes

kubernetes/apps/{grachevko,grachevaeu}/hermes/env/app/externalsecret-managed-env.yaml — the single-entry override takes the shape of the deployed version:

      sourceRef:
        storeRef:
          kind: SecretStore
          name: kubernetes

with the rule written next to it, because the flat form is the one that reads as correct.

Verified

• Go types of External Secrets v2.12.0 (the version the cluster runs, security/external-secrets): SourceRef *StoreSourceRef, and StoreSourceRef{ SecretStoreRef json:"storeRef"; *GeneratorRef json:"generatorRef" } with +kubebuilder:validation:MaxProperties=1 / MinProperties=1 (apis/externalsecrets/v1/externalsecret_types.go).
• CRD schema of the same version (bundle from the chart): spec.data[].sourceRef declares properties: [storeRef, generatorRef], maxProperties: 1, minProperties: 1; storeRef declares kind (SecretStore or ClusterSecretStore, default SecretStore) and name. The old shape fails the check on both counts, the new shape passes.
• flate test all — the two failures it reports are the unreachable network/k8s-gateway chart source (codeberg answers 503), untouched by this change.
• The same shape is already in use in this repository and running: dataFrom[].sourceRef.generatorRef in alexandrie, tg-ws-proxy, forgejo and pocket-id.

After the merge

Both chains resume by themselves (the env Kustomization stops failing the dry run); to pull the revision in without waiting for the source poll: flux reconcile source git flux-system, then flux reconcile kustomization env -n grachevko and -n grachevaeu.

Rollback

Reverting returns the object to the shape the API server rejects: both namespaces stay blocked exactly as they are now. The state before the merge is the broken one, so there is nothing to roll back to — the way out is forward, or by removing the single-entry override entirely (the API key then has to arrive another way).

## What was wrong Both instances of the agent carry an ExternalSecret that renders their `.env`: most values come from Vault, one entry reads the generated key of the API server back through the store of the namespace, and that override was written flat: ```yaml - secretKey: API_SERVER_KEY sourceRef: kind: SecretStore name: kubernetes ``` The API server refuses the object, and with it the whole rollout of the instances: ``` ExternalSecret/grachevko/hermes-managed-env dry-run failed: failed to create typed patch object (grachevko/hermes-managed-env; external-secrets.io/v1, Kind=ExternalSecret): .spec.data[0].sourceRef.kind: field not declared in schema ``` `spec.data[].sourceRef` is a `StoreSourceRef`: it takes **exactly one** of `storeRef` and `generatorRef`, and the kind and the name live **inside** `storeRef`. Written flat the object carries two properties the schema does not declare, so the `env` Kustomization of both namespaces stays NotReady, `hermes` waits on it, and the merged work of #397 and #398 does not reach the cluster at all — the file manager that now runs in the pod of the agent is waiting for the same chain. ## What this fixes `kubernetes/apps/{grachevko,grachevaeu}/hermes/env/app/externalsecret-managed-env.yaml` — the single-entry override takes the shape of the deployed version: ```yaml sourceRef: storeRef: kind: SecretStore name: kubernetes ``` with the rule written next to it, because the flat form is the one that reads as correct. ## Verified - Go types of External Secrets **v2.12.0** (the version the cluster runs, `security/external-secrets`): `SourceRef *StoreSourceRef`, and `StoreSourceRef{ SecretStoreRef `json:"storeRef"`; *GeneratorRef `json:"generatorRef"` }` with `+kubebuilder:validation:MaxProperties=1` / `MinProperties=1` (`apis/externalsecrets/v1/externalsecret_types.go`). - CRD schema of the same version (bundle from the chart): `spec.data[].sourceRef` declares `properties: [storeRef, generatorRef]`, `maxProperties: 1`, `minProperties: 1`; `storeRef` declares `kind` (SecretStore or ClusterSecretStore, default SecretStore) and `name`. The old shape fails the check on both counts, the new shape passes. - `flate test all` — the two failures it reports are the unreachable `network/k8s-gateway` chart source (codeberg answers 503), untouched by this change. - The same shape is already in use in this repository and running: `dataFrom[].sourceRef.generatorRef` in alexandrie, tg-ws-proxy, forgejo and pocket-id. ## After the merge Both chains resume by themselves (the `env` Kustomization stops failing the dry run); to pull the revision in without waiting for the source poll: `flux reconcile source git flux-system`, then `flux reconcile kustomization env -n grachevko` and `-n grachevaeu`. ## Rollback Reverting returns the object to the shape the API server rejects: both namespaces stay blocked exactly as they are now. The state before the merge is the broken one, so there is nothing to roll back to — the way out is forward, or by removing the single-entry override entirely (the API key then has to arrive another way).

hermes added 1 commit 2026-10-12 16:26:53 +00:00

fix(hermes): give the source override the shape External Secrets 2.12.0 accepts ... 77a3a30678

Both instances carry an ExternalSecret whose single entry reads the generated key of the API server
back through the store of this namespace. The override of the source was written as `sourceRef.kind`
plus `sourceRef.name`, and the API server refuses that object:

  ExternalSecret/grachevko/hermes-managed-env dry-run failed: failed to create typed patch object
  (external-secrets.io/v1, Kind=ExternalSecret): .spec.data[0].sourceRef.kind: field not declared
  in schema

The field is not declared because `spec.data[].sourceRef` is a StoreSourceRef: it takes exactly one
of `storeRef` and `generatorRef`, and the kind and the name live inside `storeRef` (the CRD of the
deployed version says maxProperties: 1, minProperties: 1, properties: [storeRef, generatorRef]; the
type is StoreSourceRef in apis/externalsecrets/v1/externalsecret_types.go of v2.12.0). Written flat,
the object carries two undeclared properties, the Kustomization that renders it never becomes Ready,
and the chain behind it - `env`, then `hermes` in both namespaces - does not start: the merged
rollout of the instances is blocked entirely, and the file manager that now runs in the pod of the
agent waits for the same chain.

Neither check of the pull request pipeline covers this class: `flate test all` reports the object as
fine, and the `Flux Local` render passes too, because neither validates custom resources against the
CRD schemas of the versions we deploy. Two issues are open for that gap (the schema check and the
admission-rule check); this commit only makes the object acceptable.

Verified against the primary sources of the deployed version: the Go types of v2.12.0
(`SourceRef *StoreSourceRef`, StoreSourceRef{storeRef, generatorRef}, kubebuilder MaxProperties=1)
and the CRD schema of v2.12.0 taken from the chart's bundle - `spec.data[].sourceRef` declares
[storeRef, generatorRef] with maxProperties 1, so the old shape fails on the undeclared `kind` and
on the count, while the new shape passes. The same shape is already in use in this repository, in
`dataFrom[].sourceRef.generatorRef` of alexandrie, tg-ws-proxy, forgejo and pocket-id.

After the merge: the two namespaces' `secrets -> env -> hermes` chains resume on their own (the
`env` Kustomization stops failing on the dry run). To pull it in without waiting for the source
poll: `flux reconcile source git flux-system` followed by
`flux reconcile kustomization env -n grachevko` (and the same for grachevaeu).

pr-agent commented 2026-10-12 16:27:16 +00:00

Collaborator

• Copy link
• Copy Markdown

Failed to update PR description

Failed to update PR description

pr-agent commented 2026-10-12 16:27:20 +00:00

Collaborator

• Copy link
• Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ No major issues detected

## PR Reviewer Guide 🔍 Here are some key observations to aid the review process: <table> <tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 1 🔵⚪⚪⚪⚪</td></tr> <tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr> <tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr> <tr><td>⚡&nbsp;<strong>No major issues detected</strong></td></tr> </table>

pr-agent commented 2026-10-12 16:27:56 +00:00

Collaborator

• Copy link
• Copy Markdown

PR Code Suggestions ✨

No code suggestions found for the PR.

## PR Code Suggestions ✨ <!-- pr-agent:improve:no-suggestions --> No code suggestions found for the PR.

hermes referenced this pull request 2026-10-12 16:28:14 +00:00

CI: валидировать custom resources по схемам CRD развёрнутых версий #400

grachevko force-pushed fix/eso-source-ref from 77a3a30678 to 57792fc93f 2026-10-12 17:39:30 +00:00 Compare

hermes added 1 commit 2026-10-12 17:59:58 +00:00

fix(hermes): state the shape of the source override once, in one instance ... b7c77b41e5

The explanation of `spec.data[].sourceRef` and of the flat form the API server refuses was copied
verbatim into both instances. In this pair a fact is written once, in the same file of the
neighbouring instance, and the other file points at it: the header of the grachevko manifest already
does exactly that for the rest of the mechanism, so the long form stays in grachevaeu and the
grachevko copy is a two-line pointer.

Nothing else changes: the nesting of the override is `sourceRef: { storeRef: { kind, name } }` in
both files. The review of the previous revision read `storeRef` as a sibling of `sourceRef` from the
diff, which does not reproduce: the rendered object of the Kustomization carries nothing beside
`sourceRef` in the entry, and the two `kind`/`name` columns sit two levels deeper.

hermes commented 2026-10-12 18:00:10 +00:00

Author

Collaborator

• Copy link
• Copy Markdown

The two Possible Bug findings of the last review do not reproduce

Both read the added storeRef as a sibling of sourceRef. In the file the columns are sourceRef (6 spaces) -> storeRef (8) -> kind/name (10), and the object this Kustomization renders carries nothing beside sourceRef in the entry:

$ flate build ks env -p ./kubernetes/flux/cluster -n grachevko
    - secretKey: API_SERVER_KEY
      sourceRef:
        storeRef:
          kind: SecretStore
          name: kubernetes

That render is what Flux sends to the API server, and it is exactly the shape the schema declares (spec.data[].sourceRef = StoreSourceRef, maxProperties: 1, properties [storeRef, generatorRef]). The flat reading comes from the diff context, not from the file - the same class as the ConfigMap nesting finding in #398.

The third finding is right and is fixed in b7c77b4

The explanation was copied verbatim into both instances, which the rule of this repository forbids: a fact is stated once, where it is not obvious from the code. The long form stays in grachevaeu and the grachevko copy is a two-line pointer - the header of that file already does this for the rest of the mechanism.

### The two `Possible Bug` findings of the last review do not reproduce Both read the added `storeRef` as a sibling of `sourceRef`. In the file the columns are `sourceRef` (6 spaces) -> `storeRef` (8) -> `kind`/`name` (10), and the object this Kustomization renders carries nothing beside `sourceRef` in the entry: ``` $ flate build ks env -p ./kubernetes/flux/cluster -n grachevko - secretKey: API_SERVER_KEY sourceRef: storeRef: kind: SecretStore name: kubernetes ``` That render is what Flux sends to the API server, and it is exactly the shape the schema declares (`spec.data[].sourceRef` = `StoreSourceRef`, `maxProperties: 1`, properties `[storeRef, generatorRef]`). The flat reading comes from the diff context, not from the file - the same class as the ConfigMap nesting finding in #398. ### The third finding is right and is fixed in `b7c77b4` The explanation was copied verbatim into both instances, which the rule of this repository forbids: a fact is stated once, where it is not obvious from the code. The long form stays in `grachevaeu` and the `grachevko` copy is a two-line pointer - the header of that file already does this for the rest of the mechanism.

grachevko merged commit d2d4334aac into main 2026-10-12 18:07:05 +00:00

grachevko deleted branch fix/eso-source-ref 2026-10-12 18:07:06 +00:00

grachevko referenced this pull request from a commit 2026-10-12 18:07:06 +00:00

fix(hermes): give the source override the shape External Secrets 2.12.0 accepts (#399)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

grachevko/home-ops!399

No description provided.