Exporting HTTP Headers to OpenTelemetry

Qtap can promote specific HTTP headers into OpenTelemetry attributes, making them available for filtering, grouping, and alerting in your observability backend. This is useful for identity headers, custom metadata, or any request/response header you want to query on.

Why Promote Headers?

When qtap captures HTTP traffic, it records standard metadata like method, URL, status code, and duration. But sometimes you need more — the identity of the authenticated user, the originating IP, or a custom tracking header.

With header promotion, those values appear as first-class OTel attributes in your backend. Instead of digging through raw payloads, you can:

• Filter dashboards by authenticated user email

• Group metrics by upstream service or region

• Alert on specific header values (e.g., missing auth headers)

• Correlate requests across services using custom trace headers

How It Works

The report_usage plugin accepts a headers configuration that specifies which HTTP headers to extract. When qtap observes matching headers in captured traffic, it includes them as customHeaders on the reported event. In an OTel backend, these are flattened into dotted attributes.

Headers are prefixed by direction:

• req.<Header-Name> — from the HTTP request

• res.<Header-Name> — from the HTTP response

In your OTel backend, these appear as customHeaders.req.<Header-Name> and customHeaders.res.<Header-Name> due to struct flattening.

Configuration

Step 1: Configure an OTel Event Store

Add an otel event store to your qtap config. If you already have one from the OpenTelemetry Integration guide, you can use your existing configuration.

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

Step 2: Add the report_usage Plugin with Headers

Add report_usage to your stack with the headers list. Each entry is a header name to extract from observed traffic.

stacks:
  default:
    plugins:
      - type: report_usage
        config:
          headers:
            - Cf-Access-Authenticated-User-Email
            - X-Forwarded-For
            - X-Request-Id
            - Content-Type

Header promotion works with any event store — including stdout for local debugging. You don't need an OTel backend to test the feature.

Step 3: Configure the Tap

Point your tap at the stack containing the report_usage plugin:

tap:
  direction: egress
  ignore_loopback: true
  http:
    stack: default

Complete Example

Here is a full qtap.yaml that captures HTTP traffic and promotes identity headers to OpenTelemetry:

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

  object_stores:
    - type: stdout

stacks:
  default:
    plugins:
      - type: http_metrics
      - type: access_logs
        config:
          mode: summary
          format: console
      - type: report_usage
        config:
          headers:
            - Cf-Access-Authenticated-User-Email
            - X-Forwarded-For
            - X-Request-Id
            - Content-Type

tap:
  direction: egress
  ignore_loopback: true
  http:
    stack: default

What You Get in Your Backend

When qtap observes traffic containing the configured headers, they appear as dotted OTLP attributes on log records:

customHeaders.req.Cf-Access-Authenticated-User-Email: "[email protected]"
customHeaders.req.X-Forwarded-For: "203.0.113.50"
customHeaders.req.X-Request-Id: "abc-123-def"
customHeaders.res.Content-Type: "application/json"

These attributes are queryable in any OTel-compatible backend — filter dashboards, build alerts, or group by header value.

Datadog Integration

If you use Datadog as your observability backend, promoted headers appear as log attributes that you can use in Log Explorer, dashboards, and monitors.

Viewing Headers in Datadog

1. Open Log Explorer in Datadog

2. Search for @customHeaders.req.Cf-Access-Authenticated-User-Email:* to find all logs with that header

3. Click a log entry to see the full attribute breakdown under customHeaders

Creating Facets

To use promoted headers in dashboard widgets and monitors, create a facet:

1. In Log Explorer, expand a log entry containing the header

2. Click the gear icon next to @customHeaders.req.Cf-Access-Authenticated-User-Email

3. Select Create facet

4. Give it a display name like "Authenticated User"

Once faceted, you can:

• Group by authenticated user in dashboard widgets

• Filter logs to a specific user

• Create monitors that alert when a specific user hits error rates

Example Datadog Query

service:qtap @customHeaders.req.Cf-Access-Authenticated-User-Email:"[email protected]" @event.summary.response_status:>=500

This finds all 5xx errors for a specific authenticated user.

Common Headers to Promote

Identity & Authentication

Header	Description
Cf-Access-Authenticated-User-Email	Cloudflare Access authenticated user
X-Forwarded-For	Client IP from reverse proxies
X-Forwarded-User	Authenticated user from auth proxies
X-Real-Ip	Original client IP

Request Tracking

Header	Description
X-Request-Id	Unique request identifier
X-Correlation-Id	Cross-service correlation
X-Amzn-Trace-Id	AWS trace identifier
Traceparent	W3C trace context

Application Metadata

Header	Description
Content-Type	Response content type
X-Cache	CDN cache hit/miss
X-Ratelimit-Remaining	Rate limit status

Sensitive headers: Be careful promoting headers like Authorization, Cookie, or Set-Cookie. Their values appear in plaintext in your OTel backend. Only promote sensitive headers if you understand the security implications and your backend has appropriate access controls.

Important Notes

• Case-insensitive matching — Header names are matched case-insensitively. Cf-Access-Authenticated-User-Email and cf-access-authenticated-user-email both work.

• Only present headers are included — If a configured header is not present in the actual request or response, it is simply omitted from the OTel output. No empty attributes are created.

• Direction prefixing — Request headers get the req. prefix, response headers get res.. This lets you distinguish the same header name appearing in both directions.

• Works with any event store — While this guide focuses on OTel, header promotion works with any event store including stdout. Use stdout for local debugging to verify headers are being captured correctly.

• Works with any OTel backend — When using an OTel event store, promoted headers use standard OTLP attributes, so they work with Datadog, Grafana, Honeycomb, New Relic, and any other OTel-compatible backend.

Next Steps

• OpenTelemetry Integration — Full OTel setup guide with logs and traces

• Traffic Processing with Plugins — All available plugins and configuration

• Traffic Capture Settings — Control what traffic qtap observes