Task · SFL-0067

helm: don't automount the ServiceAccount API token unless the pod actually talks to the API server

Status
To Do
Labels
followup, phase-3
Milestone
Security & deployment hardening
Updated
2026-08-14

Description

What

The Helm chart never sets automountServiceAccountToken, so Kubernetes applies its default (true) and projects a live, auto-refreshed ServiceAccount API token into every sf2loki pod at /var/run/secrets/kubernetes.io/serviceaccount — including the default single-instance install, which has no use for it.

The token has exactly one legitimate consumer, and it is HA-gated:

No other code in src/ reads /var/run/secrets or contacts kubernetes.default.svc. Chart defaults are ha.enabled: false (values.yaml:18-19), config.coordinate.type: noop (values.yaml:650) and networkPolicy.enabled: false (values.yaml:195), so the default render mounts an API credential that no code path uses and no policy constrains.

The chart already documents that this topology needs no control-plane access. deploy/helm/templates/networkpolicy.yaml puts the API-server egress rule inside {{- if .Values.ha.enabled }} with the comment “Omitted entirely when ha.enabled is false so a non-HA deployment doesn’t need control-plane reachability at all” — yet the credential for that control plane stays mounted in exactly that topology.

Nothing pins the current behaviour. The helm-chart CI job (.github/workflows/ci.yml:118-174) runs helm lint, three helm template | kubeconform permutations (ci.yml:162-165) and one render-guard assertion (ci.yml:167-174); kubeconform checks schema conformance and passes either way. tests/test_config_artifacts_drift.py is the only test touching deploy/helm, and it checks generated values drift, not pod-security fields.

Why it matters

sf2loki parses attacker-influenceable input: decoded Pub/Sub Avro payloads, SOQL result rows, EventLogFile CSVs and ApexLog bodies. A pod compromise through any of those paths currently hands the attacker a valid, kubelet-refreshed API-server token. Even with no RBAC bound to the ServiceAccount (the non-HA case renders no Role or RoleBinding — rbac.yaml:34-75 is entirely inside {{- if .Values.ha.enabled }}), that token authenticates as a member of system:authenticated and system:serviceaccounts, which grants API discovery, SelfSubjectRulesReview/SelfSubjectReview, TokenRequest probing, and whatever else a cluster has granted those groups — a common misconfiguration in shared clusters. With networkPolicy.enabled: false by default, nothing blocks the egress to the API server either.

This is defense-in-depth rather than a directly exploitable hole, which is why the severity is low. It is also a standard benchmark item that scanners flag on this chart today (CIS Kubernetes Benchmark 5.1.6, Checkov CKV_K8S_38, kube-score pod-probes/SA checks), and it is the one remaining gap in an otherwise thoroughly hardened pod spec: readOnlyRootFilesystem, allowPrivilegeEscalation: false, all capabilities dropped, RuntimeDefault seccomp, runAsNonRoot (values.yaml:162-174), and a resourceNames-scoped namespaced Role (rbac.yaml:55-58).

Proposed approach

Derive the value rather than hardcoding it, because the token IS required for k8s_lease. Gating on .Values.ha.enabled alone is too narrow: config.coordinate.type: k8s_lease can legitimately be set with ha.enabled: false (single replica, or externally managed RBAC via serviceAccount.create: false), and when configOverride is non-empty the template cannot see the coordinator type at all — the same blind spot the single-instance render guard at deployment.yaml:8 already handles by skipping itself.

  1. Add a helper to deploy/helm/templates/_helpers.tpl (alongside sf2loki.serviceAccountName at _helpers.tpl:48-55):

    {{/* Mount the SA API token only when something in the pod actually calls the API
         server: the k8s_lease coordinator (coordinate/k8s_lease.py load_incluster_config)
         is the only consumer. A raw configOverride hides the coordinator type from the
         template, so default to true there (operator owns correctness, same as the
         single-instance render guard). serviceAccount.automount overrides everything —
         set it true for an extraContainers sidecar that needs API access. */}}
    {{- define "sf2loki.automountServiceAccountToken" -}}
    {{- if not (kindIs "invalid" .Values.serviceAccount.automount) -}}
    {{- .Values.serviceAccount.automount -}}
    {{- else if .Values.configOverride -}}
    true
    {{- else if or .Values.ha.enabled (eq (dig "coordinate" "type" "noop" .Values.config) "k8s_lease") -}}
    true
    {{- else -}}
    false
    {{- end -}}
    {{- end -}}
  2. In deploy/helm/templates/deployment.yaml, add to the pod spec next to serviceAccountName (line 47):

    automountServiceAccountToken: {{ include "sf2loki.automountServiceAccountToken" . }}
  3. Mirror it on the created ServiceAccount in deploy/helm/templates/rbac.yaml (inside the serviceAccount.create block, lines 7-19) so any other pod referencing the SA inherits the same default. The pod-spec field takes precedence, so the two agreeing is intentional.

  4. Add the knob to deploy/helm/values.yaml under serviceAccount: (lines 42-52):

      # null (default) = derived: the API token is mounted only when the pod talks to the
      # API server (ha.enabled, config.coordinate.type: k8s_lease, or a raw configOverride
      # where the chart cannot tell). Set true when an extraContainers sidecar needs API
      # access, false to force it off.
      automount: null
  5. Extend the helm-chart CI job with a render assertion next to the existing render-guard step (.github/workflows/ci.yml:167-174), asserting automountServiceAccountToken: false in the default render and true in the HA, coordinate.type=k8s_lease, configOverride and explicit-override renders. Also add --set config.coordinate.type=k8s_lease as a fourth validate case (ci.yml:162-165) so that permutation is schema-checked.

  6. Document it in deploy/helm/README.md and docs/deployment/kubernetes.md, and note in docs/deployment/high-availability.md (near line 78, where the lease RBAC is described) that HA requires the mounted token. Include the workload-identity caveat: EKS IRSA injects its own projected volume at /var/run/secrets/eks.amazonaws.com/serviceaccount via the pod-identity webhook and is unaffected by automountServiceAccountToken: false, and GKE Workload Identity resolves ADC through the node metadata server, so the S3/GCS state-store paths that rely on serviceAccount.annotations (values.yaml:46-48) keep working. Verify both against a live render before asserting it in docs.


Imported from GitHub issue #151 on 2026-08-14, when this repo migrated from GitHub Issues to Backlog.md. The original issue has been deleted; its verbatim body, labels and comments are preserved in archive/issues-dump.json (jq '.[] | select(.number == 151)' archive/issues-dump.json).

Filed from the 2026-07-30 full-repo audit (11 finder lanes + adversarial verification per finding).

Acceptance Criteria

Definition of Done

References

View the source file on GitHub