Event and Output Pipeline ​

Purpose and Scope ​

This page provides a deep architectural and implementation-level trace of the core TLS event and output pipeline in eCapture. It covers the complete lifecycle of captured OpenSSL/BoringSSL, GoTLS, and GnuTLS events—starting from kernel-space eBPF ring buffer extraction, through user-space decoding (BaseEvent.Decode) pkg/event_processor/base_event.go:89-120, asynchronous dispatch via EventProcessor pkg/event_processor/processor.go:70-93, worker aggregation in eventWorker pkg/event_processor/iworker.go:69-88, handler processing (Text, Keylog, and Pcapng), raw transport routing via ByteSink docs/agent/output-pipeline.md:33-42, and additive publication to typed eCaptureQ channels docs/agent/output-pipeline.md:57-59.

1. Event Decoding and Asynchronous Dispatch ​

When eBPF probes capture TLS plaintext or master secrets, raw byte buffers are pushed from kernel space to user space via Perf/Ring buffers. The event processor pipeline transforms these raw byte arrays into structured domain events and routes them to appropriate worker routines.

Key Components & Data Structures ​

• BaseEvent: Implements the base decoding logic for raw ring buffer payloads, extracting fields such as DataType, Timestamp, Pid, Tid, DataLen, Comm, Fd, and Version pkg/event_processor/base_event.go:76-87 via binary little-endian reading pkg/event_processor/base_event.go:89-120.
• EventProcessor: Manages incoming events, worker routing queues (workerQueue), and cleanup tasks (destroyConn) pkg/event_processor/processor.go:27-53. The Serve() loop continuously reads from incoming, dispatches events, and manages shutdown draining via drain() pkg/event_processor/processor.go:70-115.
• eventWorker: Each unique connection or stream (identified by UUID) is assigned an eventWorker goroutine pkg/event_processor/iworker.go:69-99. It aggregates payloads up to truncation limits or idle timeouts and applies format parsers like HTTP request/response detectors pkg/event_processor/iparser.go:85-115.

Sources: pkg/event_processor/base_event.go:76-120, pkg/event_processor/processor.go:27-115, pkg/event_processor/iworker.go:69-99, pkg/event_processor/iparser.go:85-115

2. Representation Handlers and Routing ​

Once events are processed by workers, they are classified and routed to representation handlers depending on the active capture mode (Text, Keylog, or Pcapng).

Handler Mechanics ​

• Text Handler & Parsers: Formats raw payload buffers into readable UTF-8 strings or hexadecimal dumps (BaseEvent.String() / StringHex()) pkg/event_processor/base_event.go:134-167 and inspects higher-layer protocols using registered parsers (HTTPRequest, HTTPResponse) pkg/event_processor/http_request.go:28-45, pkg/event_processor/http_response.go:28-37.
• KeylogHandler: Intercepts TLS master secrets, client randoms, and traffic secrets (MasterSecretEvent, GoTLSMasterSecretEvent, DirectTrafficSecretEvent) internal/probe/base/handlers/keylog_handler.go:37-79. It deduplicates keys using a seen-keys cache and formats records in the standard NSS Key Log Format internal/probe/base/handlers/keylog_handler.go:84-106.
• Pcapng Session & Handler: Manages packet blocks alongside embedded Decryption Secrets Blocks (DSB) within a synchronized session pipeline docs/agent/output-pipeline.md:33-55.

Sources: pkg/event_processor/base_event.go:134-167, pkg/event_processor/iparser.go:117-162, pkg/event_processor/http_request.go:28-45, pkg/event_processor/http_response.go:28-37, internal/probe/base/handlers/keylog_handler.go:37-106, docs/agent/output-pipeline.md:33-55

3. ByteSink Destinations and Transport Routing ​

Raw transports are decoupled from format representation using the ByteSink interface docs/agent/output-pipeline.md:33-42. This allows text, keylog, and pcapng encoders to write seamlessly to various underlying targets via a unified factory docs/agent/output-pipeline.md:95-114.

ByteSink Contract ​

type ByteSink interface {
    io.Writer
    Name() string
    Flush() error
    Close() error
}

Source: docs/agent/output-pipeline.md:33-42

Supported Transport Destinations ​

• Stdout / File: Direct stream or file-backed storage (supporting file URIs and optional log rotation for text/keylog streams) docs/agent/output-pipeline.md:79-83.
• TCP Sinks: Ordered raw byte stream over network sockets (tcp://host:port) docs/agent/output-pipeline.md:44-48.
• WebSocket Sinks: Ordered binary-frame WebSocket streams (ws:// or wss://), where receivers concatenate frame payloads in arrival order docs/agent/output-pipeline.md:44-48.

Sources: docs/agent/output-pipeline.md:33-55, internal/output/writers/pipeline_test.go:31-104

4. Additive eCaptureQ Publication ​

In addition to writing raw byte streams to primary ByteSink destinations, eCaptureQ operates as an additive, typed event publisher docs/agent/output-pipeline.md:57-65.

Separation of Concerns ​

• Isolation: Captured payloads and sensitive TLS secrets are strictly partitioned from operational logs and never enter --logaddr or PROCESS_LOG channels docs/agent/output-pipeline.md:27-30.
• Protocol: eCaptureQ messages are serialized as binary protobuf LogEntry structures containing metadata, timestamps, process information, and network tuples without reverse-parsing raw sink bytes docs/agent/output-pipeline.md:105-118.

Sources: docs/agent/output-pipeline.md:27-30, docs/agent/output-pipeline.md:105-118