Description
Found by the SKT-0008 producer validation, 2026-08-27.
web_service emits both target_info and traces_target_info; the app workload emits neither. Tempo enable_target_info defaults to false, so some real deployments legitimately have no target_info — but Grafana Cloud enables it, and signals/apm.md [slug: apm-target-info] states outright that both must be emitted.
The consequence is concrete rather than cosmetic: the entity-graph service to pod join keys on k8s_pod_name carried by target_info. App-workload services are therefore missing from that join, so they appear unconnected in the entity graph while web_service services appear correctly — an inconsistency within one synthetic estate that no real cluster would show.
Not fixable inside spanmetrics.go without pulling in placement identity, which is why it is its own task rather than folded into the validation.
Acceptance Criteria
- #1 The app workload emits target_info and traces_target_info with the same identity contract web_service uses
- #2 The emitted labels carry the pod identity the entity-graph service-to-pod join requires
- #3 App-workload services appear connected in the entity graph alongside web_service ones
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Plan
Write a failing app target-info contract test, then emit target_info and traces_target_info with the web_service identity and required pod join labels; validate the entity-graph join shape.
2026-09-03 wave: Start from the landed target_info families. Reproduce and correct the retained per-family query-identity boundary using explicit inventory scope/query labels, validate focused app/dashgen logic, then obtain or hand back the bounded live service-to-pod graph read-back without treating empty or premature queries as proof.
2026-09-04 live closeout: resolve the current Prometheus instance id at runtime, query target_info and traces_target_info with the read credential, and prove the app service-to-pod join on k8s_pod_name alongside web_service. Empty, stale, or unauthenticated results are not proof.
2026-09-05 wave plan: No lab selection change. After Lane A fixes profiling-demo emission and one metric interval passes, root queries target_info and traces_target_info for checkout, catalog and cart-store and proves the k8s_pod_name service-to-pod join alongside the existing web_service join; empty, stale, or unauthenticated results do not count.
2026-09-06 execution: after profiling-demo is enabled for 125 seconds, query target_info and traces_target_info for checkout, catalog, and cart-store and prove the k8s_pod_name join alongside the existing web_service edge.
Implementation Notes
2026-09-02 validation: app emits target_info and traces_target_info for non-external graph nodes with service, pod, deployment, node when present, and service-instance join identity; focused tests and integrated just check and just dump passed. AC3 remains unproven live: the safe eight-blueprint identity run authenticated and checked all inventory families, but acme-ai-platform, k8s-full-stack, and otlp-native stopped at non-queryable traces_host_info before entity-graph connectivity could be established. Resume by correcting the retained SKT-0031 per-family query-identity boundary, rerun the safe live verifier, then prove the service-to-pod graph edge.
2026-09-03 live read-back: the explicit per-family identity fix is locally verified, but the standing lab query could not be completed. The RKSY Grafana context is online; both datasource discovery and an explicit Prometheus datasource query returned Unauthorized, and the standing-cluster read path stopped because the AWS SSO session is expired. No empty or unauthenticated result was treated as proof. Resume with a fresh noninteractive RKSY metrics-read credential or refreshed read session, rerun the inventory-backed verifier, and prove the app service-to-pod entity-graph edge alongside web_service.
2026-09-02: the read blocker recorded on 2026-09-03 is resolved and this is now unparked work. See SKT-0049’s notes for the cause: the Mimir Basic-auth username is the Prometheus instance id, not the stack id, and the wrong value 401s with the same body a revoked credential produces. A read credential covering every instance in the org is stored outside this repository and is named in SKT-0049. Resume by querying target_info and traces_target_info for the app-workload services on the lab stack and proving the service-to-pod join on k8s_pod_name resolves, then check AC 2 and AC 3.
2026-09-04 live readback: runtime-resolved query identities authenticated successfully. Fresh target_info and traces_target_info existed for web services, and the web service-to-pod join on k8s_pod_name resolved to five series. No active instance contained app-workload samples, so the required app-workload live join remains unproven. No source change was justified. just check, just dump, and just e2e passed; chart and published-compose e2e cases were skipped because their opt-ins were absent. Resume by selecting one existing app workload on the deployment-owning lab, waiting one metric interval, and rerunning the exact join.
2026-09-03 root correction: the 2026-09-04 park reason is a SYMPTOM, not the boundary. ‘No active instance contained app-workload samples’ is true, but the cause is not a missing app lane on the lab - the lab already selects profiling-demo, which declares the only type: app workload in any selected blueprint. Live read-back at 2026-09-03T19:35Z shows profiling-demo emitting ZERO non-selfobs metric families on the lab, alongside k8s-full-stack, aws-cloud-services and netobs-enterprise. Filed as SKT-0050. So the resume boundary ‘select one existing app workload on the deployment-owning lab’ was already satisfied and is WRONG; no lab selection change is needed. AC#3 resumes behind SKT-0050: once profiling-demo emits, query target_info and traces_target_info for services checkout, catalog and cart-store and prove the k8s_pod_name join.
2026-09-05 authenticated live read-back: target_info and traces_target_info for checkout, catalog and cart-store returned zero rows, and the k8s_pod_name service-to-pod join returned zero. The identical current web-service query returned six target-info groups and six joined groups, proving the query shape. The blocker is now exact: profiling-demo is selected in desired state but disabled in persisted control state.
2026-09-05 reachable resume boundary: the deployment owner enables profiling-demo through the existing control API, waits two 60s intervals plus the 5s delivery deadline, then reruns both target-info queries and the k8s_pod_name join for checkout, catalog and cart-store. No infrastructure selection change is needed or authorised.
2026-09-06 live result after authorised profiling-demo enablement and the full wait: the app workload itself is live, but the deployed runtime inventory for profiling-app contains only app_queue_depth. The stack shows one gateway-generated target_info row for checkout-0, zero traces_target_info rows for checkout/catalog/cart-store, and zero service-to-pod joins; the matching web-service query returns four target-info rows and two joined pod groups. Current source contains w.tickTargetInfo() in internal/workload/app/metrics.go and both target-info emitters in spanmetrics.go, but that source contract is absent from the deployed runtime. No infrastructure/deployment write is authorised this wave.
2026-09-04 live closeout, AC#3 PROVEN. Root cause was deployed-image reachability exactly as the 2026-09-06 park recorded. Bumped the lab pin in rkps-awsinfra 408e6e0: applications/synthkit/values.yaml image.tag main-4fe898f (2026-09-01, 35 commits behind) to main-4b7a80a (exact synthkit main head). No selection or control-state change. After ArgoCD sync plus a 130s settle, authenticated Mimir read-back at 2026-09-04T09:07Z on prom instance 3529994 (resolved at run time from the Cloud API, not assumed): app target_info returns 4 fresh series across checkout, catalog and cart-store; traces_target_info returns 3, one per service, each carrying k8s_pod_name; the service-to-pod join on k8s_pod_name resolves for all three alongside the web edge, which returns 7 joined groups over 6 services. Sample age verified with time()-timestamp(), 14.1s for every row, so nothing here is a stale read. The pre-rollout checkout-0 row aged out and is absent from the fresh set; an instant query’s value timestamp is the evaluation time, not the sample time, so a first pass that appeared to include it was a measurement error, not live data. One observation retained rather than treated as a defect: checkout carries two fresh target_info rows for the same pod, one synthkit-shaped with k8s_node_name, cluster and service_instance_id, and one gateway-generated carrying telemetry_sdk_name=opentelemetry and no k8s_* attributes. Both are current.
Final Summary
2026-09-02: Parked with both target-info families and their join labels implemented and tested; live entity-graph connectivity remains unproven behind the existing per-family query-identity failure.
2026-09-03: Parked after implementing and testing explicit per-family query identities. Live entity-graph connectivity remains unproven at the expired/unauthorized standing-lab read boundary.
2026-09-04: Parked at live app-workload evidence. The equivalent web join resolved, but no active app-workload samples existed, so AC 3 remains unchecked.
2026-09-05: Parked at the corrected persisted-control-state boundary. Authenticated queries prove the web service-to-pod join still resolves, but the app target-info rows and join are absent because the already-selected profiling-demo blueprint is explicitly disabled. Resume by authorised control-plane enablement, wait 125s, and rerun the three-service target_info/traces_target_info join. No infrastructure selection edit is needed.
2026-09-06: Parked at exact deployed-image reachability, not control state. profiling-demo is enabled and its app metrics emit, but the installed runtime predates the app target-info implementation. Resume by deploying an immutable image whose inventory includes target_info and traces_target_info for the app construct, wait 125 seconds, then re-run the three-service k8s_pod_name join alongside the web edge.