Start with the event chain you must reconstruct
Investigations commonly fail for two different reasons. A user action passes through an API gateway, several services, a queue, and an external platform, but their records cannot be joined. Alternatively, an attacker gains host privileges and can alter or delete the local evidence. The first is a correlation problem; the second is an integrity problem. Increasing log verbosity alone usually creates more search work without reliably answering who acted, through which entry point, on what resource, and with what result.
Before choosing technology, map the paths that matter: authentication, permission changes, data exports, ERP writes, administrator actions, and key use. Include synchronous calls and asynchronous jobs. Treat business events, operational telemetry, and security audit records as related but distinct datasets. A distributed trace is useful for diagnosing latency, yet an investigation also needs stable identities, authorization decisions, targets, and change details. The datasets should share correlation fields without pretending that one can replace the others.
Create identifiers at the trust boundary and preserve causality
Generate an internal correlation ID when traffic enters the trusted environment, or strictly validate any incoming value for format, length, and source. Blindly accepting a client-supplied identifier invites collisions, spoofing, and misleading searches. If a partner already provides a transaction number, retain it as an external reference rather than using it as the internal identifier. Correlation IDs are not credentials and should never encode email addresses, account names, or other personal data.
- Request context: Carry correlation ID, trace ID, and span ID through explicit HTTP headers or RPC metadata.
- Asynchronous context: Put the original correlation ID in the message envelope, then add message and job IDs plus a causal parent. Retries must preserve the original relationship.
- Identity context: Record actor ID, service identity, tenant, and authentication method. For delegated actions, distinguish the operator from the represented subject.
- Event context: Use structured fields for action, target, outcome, reason code, policy version, and data classification instead of relying on prose.
- Time context: Store both event and ingestion times in a consistent time standard, monitor clock drift, and account for delayed or replayed messages.
A shared middleware or SDK should inject this context so each service does not invent its own convention. When a downstream system cannot preserve headers, the integration layer can maintain a mapping between the internal correlation ID and the external transaction number. Avoid using high-cardinality IDs as metric labels, which can make monitoring expensive; keep them in logs or traces with fit-for-purpose indexes.
Make tampering detectable and deletion difficult
Forward important events off the originating host as soon as practical, then write them to centralized storage with append-only controls, object locking, or WORM retention. Application identities should be able to append records but not shorten retention or delete history. Separate authority for storage policy, log search, and signing keys. If one privileged account controls both the systems and their evidence, an immutable label offers little protection.
Hash chains can reveal inserted, removed, or modified records, while signed batches can demonstrate that an exported file has not changed since signing. Neither mechanism proves completeness when a compromised service simply stops emitting events. A signing key stored beside the application can also be abused to forge later entries, so use a separate KMS or HSM and retain key-version and rotation metadata. For critical streams, an ingestion service can assign an independent sequence number or receipt time, limiting the source system's ability to manufacture a convincing timeline.
Balance investigative value, privacy, and operating cost
Security logs should not contain passwords, access tokens, cookies, API keys, or unfiltered personal data. Prefer field allowlists, centrally managed redaction, and schema validation. Where investigators need to recognize the same subject across events, use controlled tokenization rather than storing the sensitive value. Routine diagnostic traffic may be sampled or kept briefly, but audit events covering authentication, authorization, key use, administrative actions, exports, and retention-policy changes should not disappear through sampling.
Validate the design with exercises, not configuration screenshots. Starting from a known action, confirm that investigators can follow it across APIs, workers, LINE, ERP, CRM, and cloud services. Remove a record in a test environment and verify that sequence or hash checks expose the gap. An evidence export should include its query, time boundaries, file hashes, signature, and key version. Choose tiered retention from legal obligations, threat exposure, investigation windows, and storage cost. Where integrations use inconsistent conventions, an experienced integration team can establish a common event model and strengthen the highest-risk paths first.