Task · SKT-0005.17

Validate mandatory live Grafana configuration before startup

Description

Non-empty strings currently satisfy live configuration, so malformed sink URLs, non-numeric instance IDs, transposed endpoints, and unreachable DNS/TLS can survive startup and fail only during asynchronous delivery. Add a redacted preflight with clear boundaries between static validation and live authorization.

Acceptance Criteria

Definition of Done

Implementation Plan

  1. Define and test the redacted static and bounded network/auth preflight contract for mandatory live Prometheus, Loki, and OTLP configuration.

  2. Implement strict URL and identifier validation plus explicit opt-in network checks while keeping ordinary dry-run credential-free and offline.

  3. Verify malformed, DNS/unreachable, TLS, 401/403, path, and ready states against local fake endpoints without exposing tokens.

  4. Reproduce the Docker e2e failure, add a regression test at the real entry point, and repair the fake receiver path with trusted test TLS while preserving strict production HTTPS validation.

  5. Run focused e2e, make gate, and make e2e; record the exhausted CodeRabbit review budget and exact CI evidence before re-finalization.

Implementation Notes

Implemented strict redacted mandatory live configuration validation and an explicit -preflight command. Static startup checks require expected HTTPS sink paths and positive decimal instance IDs; opt-in probes classify ready, 401/403 unauthorized, DNS/TLS/timeout/connection unreachable, and endpoint-path/status failures with five-second per-request bounds and no endpoint, ID, token, or response-body output. Ordinary DRY_RUN remains credential-free and offline. Observed red-green tests use local TLS/fake transports for valid empty RW2/Loki/OTLP probes and every failure class. Focused Go tests and vet passed; no real Grafana endpoint was probed.

Root integration: CodeRabbit reviews 7 and 8 completed; GC_FARO_COLLECTOR was additionally classified as secret-bearing and redacted. Focused Go/race/vet, make gate, and offline default/focused dry runs passed. No live Grafana endpoint was called, and no schema regeneration was applicable.

Reopened after exact-head CI run 32372813999, e2e job 96437819784, failed because the Docker fake receiver supplies HTTP sink URLs while live startup now rejects GC_PROM_RW before emission with: GC_PROM_RW must be an HTTPS URL with path /api/prom/push. The receiver observed 0 metrics, 0 log sources, 0 trace services, and 0 sigil kinds. This is a Wave 3 integration regression. Resume boundary: preserve the production HTTPS-only contract; reproduce through the real e2e entry point; upgrade the e2e receiver/container trust path to TLS if feasible; verify focused e2e, make gate, and make e2e. CodeRabbit reviews 7 and 8 exhausted the user-authorized eight-review budget, so no ninth review may be run.

TLS e2e repair evidence: receiver and synthkit now use a one-hour generated CA/server certificate valid for Docker DNS name receiver; synthkit trusts only that CA through SSL_CERT_FILE; RW2 uses /api/prom/push, OTLP uses /otlp, and fixture user IDs satisfy positive-decimal validation. Root make e2e passed with all 664 declared metrics, 3 log sources, 3 trace services, and 3 Sigil kinds received. A subsequent least-privilege experiment with a root-owned container key at mode 0600 failed because the distroless receiver runs as uid 65532; current Testcontainers-Go v0.44.0 ContainerFile has no UID/GID field, so the ephemeral host key remains 0600 while the root-copied in-container key uses 0604. Final make gate and repeat make e2e both passed. CodeRabbit review 9 was intentionally skipped because the user-authorized eight-review budget was exhausted; reviews 7 and 8 covered the originating Wave 3 change but not this repair.

Final Summary

Added strict mandatory live configuration validation and explicit redacted bounded -preflight in 265ecd9 (pushed via dab60de), covering endpoint paths, positive identifiers, ready/auth/DNS/TLS/timeout/path classifications, and secret-safe output while keeping ordinary dry-run offline.

Repair completion: strict live HTTPS validation remains intact. Commit 12b1aca upgraded the real Docker e2e receiver to a short-lived generated TLS identity, supplied a scoped test CA to synthkit, aligned RW2/OTLP paths and numeric identifiers, and proved all 664 declared metrics plus 3 log sources, 3 trace services, and 3 Sigil kinds arrive. Local make gate and repeat make e2e passed; exact-SHA CI run 32374469397 and e2e job 96443059588 passed; publish run 32374470107 built, signed, attested, scanned, and published the multi-arch image. CodeRabbit review 9 was skipped because the authorized eight-review budget was exhausted.

References

View the source file on GitHub