Backlogsm7kni.io

Projects / sf2loki / SFL-0038

Task · SFL-0038

pubsub: wildcard topic discovery only matches *EventStream - the Event/EventStore RTEM family (Threat Detection anomaly channels) is never discovered

Status
To Do
Labels
followup, phase-3, roadmap
Milestone
Feature roadmap — operator ergonomics & platform capabilities
Updated
2026-08-14

Description

What

MetadataClient.list_event_stream_topics (src/sf2loki/salesforce/metadata_client.py:29-50) resolves the topics: ["*"] wildcard by filtering the describeGlobal response with a single predicate:

names = [
    str(s["name"])
    for s in response.json().get("sobjects", [])
    if str(s.get("name", "")).endswith("EventStream")   # metadata_client.py:48
]
return sorted(f"/event/{name}" for name in names)

That matches only the older Real-Time Event Monitoring generation, where the streaming channel is <Name>EventStream (LoginEventStream, ApiEventStream, ReportEventStream, …) and the big-object store is <Name>Event (LoginEvent, ApiEvent).

The newer RTEM generation inverts the naming: the streaming channel is <Name>Event and the big-object store is <Name>EventStore. This covers the Threat Detection anomaly channels (ApiAnomalyEvent/ApiAnomalyEventStore, CredentialStuffingEvent/CredentialStuffingEventStore, ReportAnomalyEvent/ReportAnomalyEventStore, SessionHijackingEvent/SessionHijackingEventStore) and the FileEvent/FileEventStore-shaped events. None of these ends in EventStream, so the wildcard can never discover them.

The repo already records this two-generation naming split everywhere except the discovery predicate:

tests/salesforce/test_metadata_client.py:42-59 pins the current behaviour and explicitly drops LoginEvent with the comment “stored object, not a stream” — correct for the old generation, but it means the <Name>Event streaming shape has no code path at all.

The config description overstates what discovery does. src/sf2loki/config.py:350-357:

'Explicit topics, or "*" to DISCOVER and subscribe to every RTEM stream the '
"org exposes (the *EventStream channels), re-filtered by include/exclude."

“every RTEM stream the org exposes” is the headline; the *EventStream scope is a parenthetical. The same text is generated into docs/config-reference.md:68, config.example.yaml:207 and deploy/helm/values.yaml:467. docs/sources/pubsub.md:36 is the only place that states the *EventStream limit plainly.

Nothing surfaces the shortfall at runtime:

Why it matters

An org with the Threat Detection add-on that configures the documented security-posture wildcard:

sources:
  pubsub:
    enabled: true
    topics: ["*"]

gets every *EventStream channel and silently gets none of the anomaly channels. Session hijacking, credential stuffing, report anomaly and API anomaly detections — the highest-signal security events in the RTEM catalogue — are never subscribed. Discovery reports success, sf2loki doctor reports a benign wildcard WARN, and the security dashboards in deploy/grafana/dashboards/ simply have no anomaly data. The operator has no signal distinguishing “no anomalies detected” from “never subscribed”.

The data is reachable today by listing /event/ApiAnomalyEvent explicitly (src/sf2loki/config.py:355 shows exactly that), which is why this is a wildcard-completeness gap rather than a missing capability — but the wildcard is the path an operator picks precisely to avoid having to know the channel inventory.

Proposed approach

Extend list_event_stream_topics with a second discriminator derived from the same describeGlobal response, so no extra API call is needed:

  1. Collect names = {sobject["name"]} once from the response.
  2. Keep the existing rule: every name ending EventStream → /event/<name>.
  3. Add: for every name ending EventStore, strip the trailing Store; if the stripped name is also present in names, emit /event/<stripped>.
  4. Union, de-duplicate, return sorted.

Why the twin guard is the right discriminator:

Optional hardening, only if verified live first: describeGlobal entries carry per-object flags, and a platform event is not SOQL-queryable while a big object is. Requiring queryable is False on the derived candidate would tighten the rule further. Do not implement this on assumption — confirm the flag values against a real describeGlobal response (the DEV org credentials in the gitignored .env.dev are the fastest check) before relying on it. The twin-existence guard alone is sufficient and is the required behaviour; the flag check is additive.

Downstream integration needs no change, but state it in the PR description so a reviewer can confirm:

Entitlement risk to handle explicitly: an org without the Threat Detection add-on may still expose the *EventStore and *Event sObjects in describeGlobal, in which case the derived topic is discovered but unsubscribable. src/sf2loki/sources/pubsub_source.py:641-644 handles a non-INVALID_ARGUMENT subscribe failure by logging a WARN and reconnecting with exponential backoff up to the configured maximum, so the failure mode is bounded log noise, not a crash or a hot loop. This risk already exists for *EventStream channels on non-Shield orgs, so it is not a new class of failure. Mitigate the observability side rather than the mechanism:

Documentation and generated artifacts to update:

Imported from GitHub issue #122 on 2026-08-14, when this repo migrated from GitHub Issues to Backlog.md. The original issue has been deleted; its verbatim body, labels and comments are preserved in archive/issues-dump.json (jq '.[] | select(.number == 122)' archive/issues-dump.json).

Filed from the 2026-07-30 full-repo audit (11 finder lanes + adversarial verification per finding).

Acceptance Criteria

Definition of Done

References

View the source file on GitHub