Qpoint Data Schema Reference

Overview

Qpoint captures network traffic data in two distinct storage layers, each with different data structures and security characteristics:

EventStore: Anonymized connection metadata (summary information)
ObjectStore: Complete HTTP transaction payloads (may contain sensitive data)

This separation allows you to:

Store metadata in observability platforms (Axiom, OTLP backends)
Keep sensitive payloads in your own S3-compatible storage for data sovereignty
Link metadata to payloads using identifiers

EventStore Schema

EventStore contains anonymized metadata about network connections and HTTP transactions. These events do not include request/response bodies or sensitive headers.

Event Types

Event Type

Purpose

Sensitive Data

connection

TCP connection lifecycle and source attribution

artifact_record

Reference (and summary) for each stored HTTP artifact

issue

Error/anomaly detection

Connection Event

Establishes a network session with full source attribution via eBPF. Fields are grouped under meta, tags, source, destination, and other top-level properties:

{
  "meta": {
    "connectionId": "d3tsmvg7p3qggmf8pdi0",
    "endpointId": "httpbin.org",
    "tlsProbeTypesDetected": ["openssl"],
    "tlsProbeIntrospected": true
  },
  "tags": {
    "bin": ["curl"],
    "host": ["qpoint"],
    "ip": ["192.168.3.218"],
    "protocol": ["http2"],
    "strategy": ["observe"]
  },
  "finalized": true,
  "timestamp": "2025-10-24T18:54:54.510499628Z",
  "direction": "egress-external",
  "socketProtocol": "tcp",
  "l7Protocol": "http2",
  "tlsVersion": 772,
  "system": {
    "hostname": "qpoint",
    "agent": "tap",
    "agentInstance": "d3tsmqg7p3qggmf8pdfg"
  },
  "source": {
    "address": {
      "family": "ipv4",
      "ip": "192.168.3.218",
      "port": 58704
    },
    "hostname": "qpoint",
    "exe": "/usr/bin/curl",
    "userId": 1000
  },
  "destination": {
    "address": {
      "family": "ipv4",
      "ip": "34.231.103.76",
      "port": 443
    }
  },
  "bytesReceived": 4675,
  "bytesSent": 901,
  "labels": ["user-shell"]
}

Field Reference

Identifiers:

meta.connectionId (string): Unique identifier for this TCP connection. Links all events for this session.
meta.endpointId (string): Destination domain or IP observed for this connection.

Source (your application):

source.address.ip / source.address.port: IP:port of the originating process.
source.exe: Full executable path captured via eBPF.
source.userId: UID of the process owner.

Destination (external service):

destination.address.*: IP and port of the peer endpoint.

Protocol detection & TLS probes:

l7Protocol: http1, http2, or other if Qtap could not parse HTTP.
socketProtocol: Typically tcp.
tlsVersion: Numeric TLS version (772 = TLS 1.3, 771 = TLS 1.2).
meta.tlsProbeTypesDetected: TLS libraries hooked for this flow (openssl, gotls, etc.).

Context tags:

tags.bin: Process or container binary (e.g., curl, tailscaled).
tags.host, tags.ip, tags.protocol, tags.strategy: Additional grouping metadata added by Qtap.

Traffic metrics:

bytesSent, bytesReceived (optional): Directional byte counts for this socket. Only present when data transfer occurs on the connection.
direction: Capture perspective (egress-external, egress-internal, ingress, etc.).

ArtifactRecord Event

Each stored object produces an artifact_record entry. Multiple records can share the same connectionId/requestId pair:

item.type

Description

Typical content

http_transaction

Full request+response artifact

Includes summary block with method/status/duration

req-headers, res-headers

Standalone header blobs

digest points to JSON containing only headers

req-body, res-body

Base64 body blobs

digest points to raw body object

Example (`type: "http_transaction"`)

{
  "connectionId": "d3u253g7p3qi929fgpt0",
  "endpointId": "httpbin.org",
  "requestId": "d3u253g7p3qi929fgptg",
  "timestamp": "2025-10-25T01:06:23.30877375Z",
  "type": "http_transaction",
  "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c",
  "url": "http://127.0.0.1:9000/qtap-captures/35a712233f2a70e4842d83eb017f952ae09bf74c",
  "summary": {
    "connection_id": "d3u253g7p3qi929fgpt0",
    "direction": "egress-external",
    "duration_ms": 580,
    "process_exe": "/usr/bin/curl",
    "request_content_type": "application/json",
    "request_host": "httpbin.org",
    "request_method": "POST",
    "request_protocol": "http2",
    "request_scheme": "https",
    "response_content_type": "application/json",
    "response_status": 200
  }
}

Field Reference

Identifiers:

connectionId, endpointId, requestId: Match the corresponding connection and HTTP transaction.
digest: Object key in your S3-compatible store (sha1 hex string).
url: Direct fetch URL (Qtap substitutes {{DIGEST}} into the configured access_url).

Summary (only on http_transaction records):

summary.connection_id (string): Connection ID matching the parent connection event
summary.direction (string): Traffic direction (e.g., "egress-external")
summary.duration_ms (number): Request duration in milliseconds
summary.process_exe (string): Full path to the process executable
summary.request_host (string): Request authority/hostname
summary.request_method (string): HTTP method (GET, POST, etc.)
summary.request_protocol (string): HTTP version (http1, http2)
summary.request_scheme (string): Request scheme (http, https)
summary.request_content_type (string, optional): Content-Type of request body (only present when request includes a body)
summary.response_content_type (string): Content-Type of response
summary.response_status (number): HTTP status code

Use summary.response_status and summary.duration_ms to filter events without fetching full payloads from ObjectStore.

Other artifact types (req-headers, res-body, etc.) do not carry a summary block—they only include digest, type, timestamp, and the URL.

Issue Event

Issues are emitted by plugins such as detect_errors and appear as flat payloads (no nested issue/context objects). Example:

{
  "connectionId": "d3tsmvg7p3qggmf8pdi0",
  "endpointId": "httpbin.org",
  "requestId": "d3tsmvg7p3qggmf8pdig",
  "timestamp": "2025-10-24T18:54:54.509168641Z",
  "direction": "egress-external",
  "error": "Server Errors",
  "url": "https://httpbin.org/get",
  "path": "/get",
  "method": "GET",
  "status": 503,
  "triggerConditions": ["detect_errors:status"],
  "triggerReasons": ["response status code [503] matched [5xx]"],
  "tags": [
    "ip:192.168.3.218",
    "bin:curl",
    "strategy:observe",
    "host:qpoint",
    "protocol:http2"
  ]
}

Key fields:

error: Name of the rule or issue type (e.g., Server Errors).
triggerConditions / triggerReasons: Why the rule fired.
requestId: Use to pull the associated artifact.

ObjectStore Schema

ObjectStore contains complete HTTP transaction payloads, including headers and bodies. This data often contains:

Authentication tokens (JWTs, API keys)
Personally Identifiable Information (PII)
Business-sensitive data (financial info, user records)

Security: Store ObjectStore data in your own S3-compatible storage (MinIO, AWS S3, GCS) to maintain data sovereignty. Never send this to third-party observability platforms.

HttpTransaction Object

The complete HTTP request and response captured by Qtap.

Schema

{
  "metadata": {
    "process_id": "576496",
    "process_exe": "/usr/bin/curl",
    "bytes_sent": 41,
    "bytes_received": 232,
    "connection_id": "d3tsmvg7p3qggmf8pdi0",
    "endpoint_id": "httpbin.org"
  },
  "request": {
    "method": "GET",
    "url": "https://httpbin.org/get",
    "scheme": "https",
    "path": "/get",
    "authority": "httpbin.org",
    "protocol": "http2",
    "request_id": "d3tsmvg7p3qggmf8pdig",
    "user_agent": "curl/8.14.1",
    "headers": {
      ":authority": "httpbin.org",
      ":method": "GET",
      ":path": "/get",
      ":scheme": "https",
      "Accept": "*/*",
      "User-Agent": "curl/8.14.1"
    }
  },
  "response": {
    "status": 503,
    "content_type": "text/html",
    "headers": {
      ":status": "503",
      "Content-Length": "162",
      "Content-Type": "text/html",
      "Date": "Fri, 24 Oct 2025 18:54:54 GMT",
      "Server": "awselb/2.0"
    },
    "body": "PGh0bWw+DQo8aGVhZD48dGl0bGU+NTAzIFNlcnZpY2UgVGVtcG9yYXJpbHkgVW5hdmFpbGFibGU8L3RpdGxlPjwvaGVhZD4NCjxib2R5Pg0KPGNlbnRlcj48aDE+NTAzIFNlcnZpY2UgVGVtcG9yYXJpbHkgVW5hdmFpbGFibGU8L2gxPjwvY2VudGVyPg0KPC9ib2R5Pg0KPC9odG1sPg0K"
  },
  "transaction_time": "2025-10-24T18:54:54.508987323Z",
  "duration_ms": 76,
  "direction": "egress-external"
}

Optional Top-Level Fields: When using Docker logging drivers (e.g., with Fluent Bit), you may see additional top-level fields like container_id, container_name, and source added by the Docker logging infrastructure.

Field Reference

Metadata (Source Attribution):

metadata.process_id, metadata.process_exe: Process identity.
metadata.bytes_sent / metadata.bytes_received: Same byte counters as the connection event.
metadata.connection_id, metadata.endpoint_id: Identifiers for correlation.
Optional fields (only when Qtap observes them): metadata.container_name, metadata.pod_name, etc. These were not present in the observed captures but can appear in containerized/Kubernetes deployments.

Request:

request.method (string): HTTP method
request.url (string): Complete URL
request.scheme (string): http or https
request.path (string): URL path
request.authority (string): Host header
request.protocol (string): HTTP version (http1, http2)
request.request_id (string): Unique transaction ID
request.user_agent (string): User-Agent header
request.content_type (string): Content-Type header value (when request has a body)
request.headers (object): All request headers (including Authorization)
request.body (string): Base64-encoded request body (present when request includes body)

HTTP/2 Pseudo-Headers: Headers like :authority, :method, :path, :scheme are HTTP/2 pseudo-headers. For HTTP/1.1, these are represented as standard headers.

Response:

response.status, response.headers, response.body (base64). Body may be absent when capture level < full.

Timing & Direction:

transaction_time (string): ISO 8601 timestamp with nanosecond precision
duration_ms (number): Request duration in milliseconds
direction (string): Traffic direction (see Connection Event)

Decoding Bodies

Request and response bodies are base64-encoded to safely handle binary data:

import base64
import json

# Decode request body
request_body = base64.b64decode(transaction["request"]["body"])
request_data = json.loads(request_body)
print(request_data["name"])  # "John Doe"

# Decode response body
response_body = base64.b64decode(transaction["response"]["body"])
response_data = json.loads(response_body)
print(response_data["id"])  # 1234

// Decode in Node.js
const requestBody = Buffer.from(transaction.request.body, 'base64').toString('utf-8');
const requestData = JSON.parse(requestBody);
console.log(requestData.name);  // "John Doe"

const responseBody = Buffer.from(transaction.response.body, 'base64').toString('utf-8');
const responseData = JSON.parse(responseBody);
console.log(responseData.id);  // 1234

Capture Levels

Qtap provides four capture levels that control what data appears in the HttpTransaction object:

Level

Description

Includes

Use Case

none

No capture

Nothing

Disabled capture (can use with rules for selective capture)

summary

Basic info

metadata, request.method/url/status, duration_ms

High-level traffic monitoring

details

With headers

Everything in summary + request.headers + response.headers

Debugging without exposing bodies

full

Complete

Everything including request.body + response.body

Deep troubleshooting, audit trails

Example Configuration

stacks:
  my_stack:
    plugins:
      - type: http_capture
        config:
          level: summary        # Default: basic info only
          format: json
          rules:
            - name: "Full capture for errors"
              expr: http.res.status >= 400
              level: full       # Capture everything for errors

            - name: "Headers for specific API"
              expr: http.req.host == "api.example.com"
              level: headers    # Capture headers but not bodies

Key Relationships

Identifier Hierarchy

Connection (TCP session)
  ↓ connection_id
  ├─ Connection Event (1 per TCP connection)
  ├─ ArtifactRecord Event (N per HTTP transaction, links to ObjectStore)
  └─ Issue Event (0-N per connection, if errors detected)

Each HTTP transaction links via:
  ↓ request_id
  └─ HttpTransaction object in ObjectStore (plus any header/body artifacts)

connectionId: Links all events for a single TCP connection

One TCP connection can carry multiple HTTP requests (HTTP keep-alive)
Useful for correlating requests from the same socket

endpointId: The destination domain (e.g., "api.example.com")

NOT the source - this is the external service being called
Useful for grouping requests by destination

requestId: Unique per HTTP transaction

One-to-one mapping to HttpTransaction object
Primary key for correlating EventStore → ObjectStore

Source vs Destination

Critical distinction:

Field

Represents

Contains

Captured Via

Source

Your application

Process, container, pod, user details

eBPF syscall hooks

Destination

External service

IP address, port, resolved hostname

Network layer

Why source has more detail:

Qtap uses eBPF to hook into the source process's system calls
Full process context (exe path, user, container ID) available via kernel
Destination is just an IP:port visible from network packets

Direction Values

The direction field indicates traffic flow relative to Qtap's capture point:

Value

Description

Example

egress

All outbound traffic

Any request leaving your application

egress-external

Outbound to external networks

API calls to public internet

egress-internal

Outbound to internal networks

Service-to-service calls within cluster

ingress

Inbound traffic

Requests received by your server

all

Bidirectional

Capture both ingress and egress

Configure in qtap.yaml:

tap:
  direction: egress-external    # Only capture outbound external calls

Protocol Detection

L7 Protocol Values

The l7Protocol field indicates whether Qtap successfully parsed the traffic as HTTP:

Value

Meaning

What You Get

http1

HTTP/1.1 successfully parsed

✅ Connection Event + HttpTransaction + ArtifactRecord

http2

HTTP/2 successfully parsed

✅ Connection Event + HttpTransaction + ArtifactRecord

other

Non-HTTP protocol or parsing failed

⚠️ Connection Event only (no HTTP parsing)

Understanding `l7Protocol: "other"`

When you see l7Protocol: "other", Qtap detected a TCP connection but could not parse it as HTTP. This is expected and normal for non-HTTP protocols:

Common Protocols That Show as "other":

Database connections (MySQL port 3306, PostgreSQL port 5432, MongoDB port 27017, Redis port 6379)
SSH connections (port 22)
SMTP/email traffic (ports 25, 465, 587)
Custom binary protocols
gRPC with binary protobuf payloads (HTTP/2 framing detected, but content is binary)
DNS over HTTPS/TLS (HTTPS connection, but not standard HTTP request/response pattern)

What You Still Get:

✅ Connection Event with full metadata
✅ Process Attribution: Exact executable path (source.exe)
✅ Byte Counts: bytesSent and bytesReceived
✅ TLS Detection: tlsVersion and tlsProbeTypesDetected (when applicable)
✅ Endpoint: Destination IP/domain (endpointId)

What You Don't Get:

❌ HttpTransaction object (no request/response parsing)
❌ ArtifactRecord events
❌ Request headers, bodies, or application-level data

Troubleshooting HTTP Traffic Showing as "other":

If you expect HTTP traffic but see l7Protocol: "other":

Check tlsProbeTypesDetected - Is TLS detected? If empty, qtap may not have hooked the TLS library
Check gotProtocolEvent: true - Was protocol detection attempted?
Enable --bpf-trace for syscall-level debugging
Verify the application uses a supported TLS library (OpenSSL, Go TLS, Java SSL, Node TLS)

TLS Detection

is_tls: true - HTTPS traffic detected
tls_version: 772 - TLS 1.3 (771 = TLS 1.2)
tls_sni: "api.example.com" - Server Name Indication from handshake
tls_probe_types_detected: ["openssl"] - OpenSSL library hooks attached

Supported TLS libraries:

openssl - OpenSSL library (all versions)
gotls - Go's native TLS implementation (Pro version)
javassl - Java SSL/TLS implementation (Pro version)
nodetls - Node.js TLS implementation (Pro version)

Complete Example: Correlating Events

Scenario

Your application makes a POST request that returns a 500 error. Here's how the data flows:

1. Connection Event (EventStore)

{
  "meta": {
    "connectionId": "conn_abc123",
    "endpointId": "api.example.com",
    "tlsProbeTypesDetected": ["openssl"]
  },
  "direction": "egress-external",
  "socketProtocol": "tcp",
  "l7Protocol": "http2",
  "source": {
    "address": {"family": "ipv4", "ip": "10.0.1.50", "port": 54321},
    "exe": "/usr/bin/myapp",
    "hostname": "app-server-01"
  },
  "destination": {
    "address": {"family": "ipv4", "ip": "93.184.216.34", "port": 443}
  },
  "bytesSent": 1024,
  "bytesReceived": 2048
}

2. Issue Event (EventStore)

{
  "connectionId": "conn_abc123",
  "endpointId": "api.example.com",
  "requestId": "req_xyz987",
  "timestamp": "2025-10-23T10:00:06Z",
  "direction": "egress-external",
  "error": "Server Errors",
  "url": "https://api.example.com/process",
  "path": "/process",
  "method": "POST",
  "status": 500,
  "triggerConditions": ["detect_errors:status"],
  "triggerReasons": ["response status code [500] matched [5xx]"]
}

3. ArtifactRecord Event (EventStore)

{
  "connectionId": "conn_abc123",
  "endpointId": "api.example.com",
  "requestId": "req_xyz987",
  "timestamp": "2025-10-23T10:00:06Z",
  "type": "http_transaction",
  "digest": "2fd4e1c67a2d28fced849ee1bb76e7391b93eb12",
  "url": "http://minio.local/qtap-captures/2fd4e1c67a2d28fced849ee1bb76e7391b93eb12",
  "summary": {
    "connection_id": "conn_abc123",
    "direction": "egress-external",
    "duration_ms": 5234,
    "process_exe": "/usr/bin/myapp",
    "request_host": "api.example.com",
    "request_method": "POST",
    "request_protocol": "http2",
    "request_scheme": "https",
    "response_status": 500
  }
}

4. HttpTransaction Object (ObjectStore, retrieved from S3)

{
  "metadata": {
    "connection_id": "conn_abc123",
    "process_exe": "/usr/bin/myapp"
  },
  "request": {
    "request_id": "req_xyz987",
    "method": "POST",
    "url": "https://api.example.com/process",
    "headers": {
      "Authorization": "Bearer eyJhbGci..."
    },
    "body": "eyJkYXRhIjogIi4uLiJ9"
  },
  "response": {
    "status": 500,
    "body": "eyJlcnJvciI6ICJJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IifQ=="
  }
}

Query Pattern

// 1. Find errors in EventStore (fast)
const issues = await eventStore.query({
  event_type: "issue",
  status: 503,
  timestamp: { gte: "2025-10-24T00:00:00Z" }
});

for (const issue of issues) {
  const artifacts = await eventStore.query({
    event_type: "artifact_record",
    request_id: issue.requestId,
    type: "http_transaction"
  });

  for (const artifact of artifacts) {
    const obj = await s3.getObject({
      bucket: "qtap-captures",
      key: artifact.digest
    });
    const transaction = JSON.parse(obj.Body.toString("utf-8"));
    // Analyze as needed…
  }
}

Best Practices

Data Storage

EventStore → Observability Platform
- Axiom, Datadog, Honeycomb, etc.
- Safe for third-party platforms (no sensitive data)
- Enables dashboards, alerts, and real-time monitoring
ObjectStore → Your S3
- MinIO, AWS S3, Google Cloud Storage
- Keep sensitive data in your infrastructure
- Set TTL policies (recommended: 30-90 days)

Querying Strategy

Query EventStore first (fast, indexed)
- Find relevant connections/requests by metadata
- Filter by status codes, endpoints, duration, etc.
Fetch from ObjectStore only when needed (slower, S3 API calls)
- Retrieve full payloads for specific incidents
- Analyze auth tokens, request bodies, error details
Use identifiers to correlate
- connection_id: Group all requests from same TCP session
- request_id: Link metadata to payload
- endpoint_id: Filter by destination domain

Security Considerations

Never log ObjectStore data to stdout in production
Set appropriate S3 bucket policies (private, encrypted)
Use IAM roles, not static credentials
Implement TTL/lifecycle policies to auto-delete old data
Audit access to S3 payloads (contains auth tokens)

Performance Optimization

Use capture levels strategically:
- summary by default (low overhead)
- full only for errors or specific endpoints (via rules)
Filter noisy traffic at capture time (process filters, direction settings)
Batch S3 uploads with Fluent Bit for cost efficiency

Troubleshooting

Missing Fields

Problem: Expected fields not present in event

Solutions:

Container fields only present when running in Docker/containerd
Pod fields only present when running in Kubernetes
user/user_id only available with sufficient permissions

`l7Protocol: "other"`

Problem: HTTP parsing failed, no HttpTransaction created

Diagnostics:

Check tlsProbeTypesDetected - Empty array means TLS library not hooked
Check gotProtocolEvent: false - Protocol detection didn't run
Enable --bpf-trace="mod:socket,exe.contains:yourapp" for syscall traces

Common causes:

Non-standard HTTP libraries
Custom TLS wrappers
Binary not using OpenSSL/BoringSSL

Empty Bodies

Problem: request.body or response.body is empty string

Check capture level:

Must be level: full to capture bodies
level: headers only captures headers
level: summary captures neither

Verify with rules:

plugins:
  - type: http_capture
    config:
      level: summary      # Default: no bodies
      rules:
        - name: "Full capture for /api"
          expr: http.req.path matches /^\/api/
          level: full     # Override to capture bodies

Additional Resources

Healthy Capture Patterns: appendix/healthy-capture-patterns.md - Examples of successful captures
BPF Trace Guide: appendix/bpf-trace.md - Syscall-level debugging
Configuration Examples: getting-started/qtap/configuration/configuration-examples.md
Traffic Processing with Plugins: getting-started/qtap/configuration/traffic-processing-with-plugins.md - Rulekit expressions and plugin configuration

Last updated 1 month ago

hashtagOverview

hashtagEventStore Schema

hashtagEvent Types

hashtagConnection Event

hashtagField Reference

hashtagArtifactRecord Event

hashtagExample (type: "http_transaction")

hashtagField Reference

hashtagIssue Event

hashtagObjectStore Schema

hashtagHttpTransaction Object

hashtagSchema

hashtagField Reference

hashtagDecoding Bodies

hashtagCapture Levels

hashtagExample Configuration

hashtagKey Relationships

hashtagIdentifier Hierarchy

hashtagSource vs Destination

hashtagDirection Values

hashtagProtocol Detection

hashtagL7 Protocol Values

hashtagUnderstanding l7Protocol: "other"

hashtagTLS Detection

hashtagComplete Example: Correlating Events

hashtagScenario

hashtag1. Connection Event (EventStore)

hashtag2. Issue Event (EventStore)

hashtag3. ArtifactRecord Event (EventStore)

hashtag4. HttpTransaction Object (ObjectStore, retrieved from S3)

hashtagQuery Pattern

hashtagBest Practices

hashtagData Storage

hashtagQuerying Strategy

hashtagSecurity Considerations

hashtagPerformance Optimization

hashtagTroubleshooting

hashtagMissing Fields

hashtagl7Protocol: "other"

hashtagEmpty Bodies

hashtagAdditional Resources

Overview

EventStore Schema

Event Types

Connection Event

Field Reference

ArtifactRecord Event

Example (`type: "http_transaction"`)

Field Reference

Issue Event

ObjectStore Schema

HttpTransaction Object

Schema

Field Reference

Decoding Bodies

Capture Levels

Example Configuration

Key Relationships

Identifier Hierarchy

Source vs Destination

Direction Values

Protocol Detection

L7 Protocol Values

Understanding `l7Protocol: "other"`

TLS Detection

Complete Example: Correlating Events

Scenario

1. Connection Event (EventStore)

2. Issue Event (EventStore)

3. ArtifactRecord Event (EventStore)

4. HttpTransaction Object (ObjectStore, retrieved from S3)

Query Pattern

Best Practices

Data Storage

Querying Strategy

Security Considerations

Performance Optimization

Troubleshooting

Missing Fields

`l7Protocol: "other"`

Empty Bodies

Additional Resources