Sending Qtap Events to OpenTelemetry

This guide shows you how to send qtap network observability data to any OpenTelemetry-compatible backend using the OTLP (OpenTelemetry Protocol).

What Qtap Sends

Qtap can export network events as OpenTelemetry Logs with rich structured attributes, making it compatible with any log aggregation or observability platform that supports OTLP. See Storage Configuration for details.

Optionally, qtap can also send OpenTelemetry Traces for internal performance monitoring by setting standard OpenTelemetry environment variables. See Tracing Configuration for details.

Two Types of Log Events

1. Connection Events (event.type: connection)

Tracks TCP connection lifecycle with metadata:

• Source and destination IPs, ports

• Protocol detection (http1, http2, tcp, etc.)

• TLS version and inspection status

• Process information (exe path, user ID, hostname)

• Bytes sent/received

2. Artifact Record Events (event.type: artifact_record)

HTTP transaction summaries with links to full payload data:

• Method, URL, path, status code

• User agent and content type

• Duration in milliseconds

• Bytes sent and received

• Connection and request IDs for correlation

• Process information (exe path)

• Reference URL to full request/response data in object stores

• Digest for data integrity

• Tags: strategy, host, protocol, IP, binary name

Prerequisites

• Qtap installed and running (see Getting Started)

• An OpenTelemetry-compatible backend or collector

Quick Start: Docker with OTel Collector

This example sets up an OpenTelemetry Collector to receive qtap events and export them for visualization.

Step 1: Configure Qtap

Create qtap-config.yaml:

Using an existing OTel Collector? Update the endpoint value below to point to your collector's address (e.g., otel-collector.monitoring.svc.cluster.local:4317 for Kubernetes).

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"  # OTel Collector gRPC endpoint (change for existing collector)
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

  object_stores:
    - type: stdout  # Or configure S3 for sensitive data

stacks:
  observability:
    plugins:
      - type: http_metrics      # Prometheus metrics
      - type: http_capture      # HTTP transaction capture
        config:
          level: summary
          format: json

tap:
  direction: egress
  ignore_loopback: false
  audit_include_dns: false
  http:
    stack: observability

Step 2: Deploy OpenTelemetry Collector

Already have an OpenTelemetry Collector running? Skip the deployment below and jump to Step 3: Start Qtap (make sure you updated the endpoint in Step 1 to point to your existing collector).

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints logs to console
  debug:
    verbosity: detailed

  # Add your backend exporter here (e.g., otlphttp for many SaaS platforms)
  # otlphttp:
  #   endpoint: "https://your-backend.com/v1/logs"
  #   headers:
  #     api-key: "${API_KEY}"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # Add your backend exporter here

Create docker-compose.yaml:

services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Start the collector:

docker compose up -d

Step 3: Start Qtap

docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Step 4: Generate Traffic and Verify

# Generate test traffic
curl https://httpbin.org/get

# Check OTel Collector logs
docker logs otel-collector

# You should see qtap log events with HTTP transaction data

Kubernetes Deployment

Using OpenTelemetry Operator

Already have OpenTelemetry Operator installed? Skip step 1 and proceed directly to deploying the collector (step 2), or if you already have a collector, skip to step 3 and just update your qtap ConfigMap with the collector endpoint.

1. Install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

1. Deploy an OpenTelemetry Collector:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qtap-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 10s

    exporters:
      otlphttp:
        endpoint: "https://your-backend.com/v1/logs"
        headers:
          api-key: "${API_KEY}"

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]

1. Update qtap DaemonSet configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: qtap-config
data:
  qtap.yaml: |
    version: 2
    services:
      event_stores:
        - type: otel
          endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
          protocol: grpc
          service_name: "qtap"
          environment: "production"
      object_stores:
        - type: s3
          # S3 configuration...
    # ... rest of config

Backend-Specific Configurations

Datadog

Datadog supports OTLP ingestion directly:

event_stores:
  - type: otel
    endpoint: "https://http-intake.logs.datadoghq.com/v2/logs"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      DD-API-KEY:
        type: env
        value: DATADOG_API_KEY
    tls:
      enabled: true

Honeycomb

event_stores:
  - type: otel
    endpoint: "api.honeycomb.io:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      x-honeycomb-team:
        type: env
        value: HONEYCOMB_API_KEY
      x-honeycomb-dataset:
        type: text
        value: "qtap-events"
    tls:
      enabled: true

New Relic

event_stores:
  - type: otel
    endpoint: "otlp.nr-data.net:4317"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      api-key:
        type: env
        value: NEW_RELIC_LICENSE_KEY
    tls:
      enabled: true

Grafana Cloud

event_stores:
  - type: otel
    endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      authorization:
        type: env
        value: GRAFANA_CLOUD_API_KEY  # Format: "Basic base64(instanceID:apiKey)"
    tls:
      enabled: true

Elastic

event_stores:
  - type: otel
    endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      Authorization:
        type: env
        value: ELASTIC_APM_SECRET_TOKEN
    tls:
      enabled: true

Understanding the Log Structure

Connection Event Example

{
  "Timestamp": "2025-10-16T20:00:15.610697267Z",
  "SeverityText": "",
  "SeverityNumber": 9,
  "Body": "Connection: [egress-external via tcp] hostname → httpbin.org",
  "Attributes": {
    "event.type": "connection",
    "event": {
      "destination.address.ip": "3.230.216.40",
      "destination.address.port": 443,
      "direction": "egress-external",
      "l7Protocol": "http2",
      "socketProtocol": "tcp",
      "source.exe": "/usr/bin/curl",
      "source.hostname": "qpoint",
      "tlsVersion": 772,
      "meta.connectionId": "d3okth87p3quljkmp1p0",
      "meta.endpointId": "httpbin.org"
    }
  }
}

Artifact Record Event Example

{
  "Timestamp": "2025-10-16T20:16:43.907223121Z",
  "Body": "Artifact stored: http_transaction",
  "Attributes": {
    "event.type": "artifact_record",
    "event": {
      "type": "http_transaction",
      "digest": "8aee8eee5b3b6f44e66757f430414a9fea6363d5",
      "url": "stdout://8aee8eee5b3b6f44e66757f430414a9fea6363d5",
      "summary.request_host": "httpbin.org",
      "summary.request_method": "GET",
      "summary.request_protocol": "http2",
      "summary.request_scheme": "https",
      "summary.response_status": 200,
      "summary.response_content_type": "application/json",
      "summary.direction": "egress-external",
      "summary.duration_ms": 127,
      "summary.connection_id": "d3ol5ao7p3qurnil1040",
      "summary.process_exe": "/usr/bin/curl",
      "connectionId": "d3ol5ao7p3qurnil1040",
      "endpointId": "httpbin.org",
      "requestId": "d3ol5ao7p3qurnil107g"
    }
  }
}

Querying and Filtering

Filter by Event Type

Most observability platforms allow filtering on attributes:

# Show only HTTP transaction summaries
event.type = "artifact_record"

# Show only TCP connections
event.type = "connection"

Find Slow Requests

event.type = "artifact_record" AND event.summary.duration_ms > 1000

Find Errors

event.type = "artifact_record" AND event.summary.response_status >= 500

Track Specific Endpoints

event.type = "artifact_record" AND event.summary.request_host = "api.example.com"

Monitor HTTP/2 vs HTTP/1 Traffic

# HTTP/2 traffic
event.type = "connection" AND event.l7Protocol = "http2"

# HTTP/1 traffic
event.type = "connection" AND event.l7Protocol = "http1"

Troubleshooting

No Events Appearing

Check qtap logs:

docker logs qtap | grep -i "otel\|error"

Common issues:

1. Wrong endpoint - Verify the endpoint address and port

   • gRPC default: 4317

   • HTTP default: 4318

   • With --network=host: use localhost:4317

   • In Docker bridge network: use host.docker.internal:4317

2. TLS mismatch - If backend requires TLS, set tls.enabled: true

3. Authentication - Verify headers/API keys are correct

Connection Refused Errors

rpc error: code = Unavailable desc = name resolver error

Solutions:

• Check if OTel Collector is running: docker ps | grep otel

• Verify network connectivity

• Check firewall rules for port 4317/4318

"Unknown service" Errors

rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.logs.v1.LogsService

This means the OTel Collector doesn't have a logs pipeline configured. Add a logs pipeline to your collector config:

service:
  pipelines:
    logs:  # ← Must have this
      receivers: [otlp]
      processors: [batch]
      exporters: [your_exporter]

Debugging with stdout Protocol

For local debugging, use the stdout protocol to see OTLP data printed to qtap logs:

event_stores:
  - type: otel
    protocol: stdout         # Prints OTLP data to console
    service_name: "qtap"
    environment: "development"

Then check qtap logs:

docker logs qtap

Best Practices

1. Use TLS in production - Always enable tls.enabled: true for production deployments

2. Store sensitive data securely - Use S3-compatible object stores for full HTTP payloads containing sensitive information

3. Filter appropriately - Use qtap's filter configuration to avoid capturing unnecessary traffic and reduce costs

4. Set resource attributes - Use service_name and environment for better filtering in your observability platform

5. Monitor qtap itself - Set up alerts on qtap's Prometheus metrics to ensure it's healthy

6. Use batch processing - OTel Collector's batch processor reduces API calls and improves performance

Next Steps

• Configure S3 Object Storage for sensitive payload data

• Set up Prometheus Metrics for qtap health monitoring

• Configure Traffic Filters to reduce noise

Additional Resources

• Storage Configuration Reference

• OpenTelemetry Collector Documentation

• OTLP Specification