OpenTelemetry Integration

This guide shows you how to send qtap observability data to OpenTelemetry-compatible backends using OTLP (OpenTelemetry Protocol).

Understanding Qtap's OpenTelemetry Integration

Qtap exports two types of data as OpenTelemetry Logs via the OTLP protocol:

Data Type	Configuration	What It Contains	Use Case
Events	qtap.yaml event_stores	Connection metadata, HTTP transaction summaries (method, URL, status, duration, process attribution)	Analytics, monitoring, troubleshooting — know WHAT happened and WHICH process did it
Objects	qtap.yaml object_stores	Full request/response headers and bodies	Compliance, security investigation, debugging — see the actual payload content

Both can be sent through the same OTel collector endpoint, giving you a unified pipeline for all captured data — no separate S3 setup required.

Events vs Objects: Events are lightweight metadata about every connection and HTTP transaction. Objects are the full request/response content, captured selectively based on your plugin configuration. You can send both through OTel, or use OTel for events and S3 for objects — see Storage Configuration for all options.

Prerequisites

• Qtap installed and running (see Getting Started)

• An OpenTelemetry-compatible backend or collector

Quick Start (5 minutes)

Send both events and artifacts to any OpenTelemetry-compatible backend.

Step 1: Configure Qtap

Create qtap-config.yaml:

version: 2

services:
  event_stores:
    - type: otel  # Send logs to OpenTelemetry
      endpoint: "localhost:4317"  # OTel Collector gRPC endpoint
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false  # Set to true for production

  object_stores:
    - type: otel  # Send artifacts through OTel (or use S3/stdout)
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

stacks:
  default:
    plugins:
      - type: http_capture
        config:
          level: summary  # (none|summary|headers|full)
          format: json    # (json|text)

tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)
  http:
    stack: default

Step 2: Deploy OpenTelemetry Collector

Already have an OpenTelemetry Collector? Update the endpoint in Step 1 to point to your collector and skip to Step 3.

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints logs to console for testing
  debug:
    verbosity: detailed

  # Add your backend exporter here
  # otlphttp:
  #   endpoint: "https://your-backend.com/v1/logs"
  #   headers:
  #     api-key: "${API_KEY}"

service:
  pipelines:
    logs:  # Logs pipeline for qtap event_stores
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # Add your backend exporter

Create docker-compose.yaml:

services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Start the collector:

docker compose up -d

Step 3: Start Qtap

docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Note: No additional environment variables are needed — logs are configured entirely through qtap.yaml.

Step 4: Generate Traffic and Verify

# Generate test traffic
curl https://httpbin.org/get

# Check OTel Collector logs
docker logs otel-collector

You should see two types of log events:

1. Connection Event:

LogRecord
Body: Connection: [egress-external via tcp] hostname → httpbin.org
Attributes:
  event.type: connection
  event.l7Protocol: http2
  event.tlsVersion: 772
  event.meta.connectionId: d3sggv07p3qrvfkj173g
  event.source.exe: /usr/bin/curl

2. Artifact Record Event:

LogRecord
Body: Artifact stored: http_transaction
Attributes:
  event.type: artifact_record
  event.summary.request_method: GET
  event.summary.request_host: httpbin.org
  event.summary.response_status: 200
  event.summary.duration_ms: 2023
  event.summary.connection_id: d3sggv07p3qrvfkj173g

3. Object (Artifact Data):

When using OTel object stores, the full request/response content is also sent as log records:

LogRecord
Body: Artifact: http_request (application/json, 1247 bytes)
Attributes:
  artifact.type: http_request
  artifact.content_type: application/json
  artifact.digest: sha256:a1b2c3d4...
  artifact.size_bytes: 1247
  artifact.data: <raw bytes — full request/response content>
  connection.id: d3sggv07p3qrvfkj173g
  connection.endpoint_id: httpbin.org
  connection.request_id: d3sggv07p3qrvfkj174g

Objects include the complete HTTP headers and bodies as captured by the http_capture plugin. The connection.id links objects back to their corresponding connection and artifact record events.

Success! Qtap is sending events and artifacts to OpenTelemetry. Use connection.id to correlate connection metadata, HTTP transaction summaries, and full request/response content.

Understanding the Data

What Logs Contain

Connection Events (event.type: connection):

• TCP connection metadata

• Protocol detection (http1, http2, tcp)

• TLS version and inspection status

• Bytes sent/received

• Process information (exe path, user ID)

Artifact Record Events (event.type: artifact_record):

• HTTP method, URL, path, status code

• User agent and content type

• Duration in milliseconds

• Process information

• Link to stored artifact (OTel or S3 URL depending on object store type)

What Objects Contain

When using OTel object stores (object_stores type: otel), artifact data is sent as additional log records:

• artifact.type — http_request or http_response

• artifact.content_type — MIME type (e.g., application/json)

• artifact.data — The raw bytes (full headers and body)

• artifact.size_bytes — Size of the artifact

• artifact.digest — Content hash for deduplication

• connection.id — Links back to the connection and artifact record events

• artifact.summary.* — Request/response metadata (method, host, status, etc.)

Events vs Objects: Events (event_stores) are lightweight metadata sent for every connection and HTTP transaction. Objects (object_stores) are the full payload content, captured based on your http_capture plugin level setting (summary, details, or full).

Example log query:

event.type = "artifact_record" AND event.summary.response_status >= 500

Kubernetes Deployment

Using OpenTelemetry Operator

Already have OpenTelemetry Operator installed? Skip to step 2.

1. Install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

1. Deploy an OpenTelemetry Collector:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qtap-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 10s

    exporters:
      otlphttp:
        endpoint: "https://your-backend.com/v1/logs"
        headers:
          api-key: "${API_KEY}"

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]

1. Deploy qtap using Helm:

First, create your qtap configuration file qtap-config.yaml:

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

  object_stores:
    - type: otel
      otel_endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: json

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack

Install qtap with Helm:

# Add Qpoint Helm repository
helm repo add qpoint https://helm.qpoint.io/
helm repo update

# Install qtap with config
helm install qtap qpoint/qtap \
  -n qpoint \
  --create-namespace \
  --set-file config=./qtap-config.yaml \
  --set logLevel=warn

Verify the deployment:

kubectl get pods -n qpoint
kubectl logs -n qpoint -l app.kubernetes.io/name=qtap --tail=50

Backend-Specific Configurations

SaaS Platforms

Datadog

Datadog supports OTLP ingestion directly:

event_stores:
  - type: otel
    endpoint: "https://http-intake.logs.datadoghq.com/v2/logs"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      DD-API-KEY:
        type: env
        value: DATADOG_API_KEY
    tls:
      enabled: true

Honeycomb

event_stores:
  - type: otel
    endpoint: "api.honeycomb.io:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      x-honeycomb-team:
        type: env
        value: HONEYCOMB_API_KEY
      x-honeycomb-dataset:
        type: text
        value: "qtap-logs"
    tls:
      enabled: true

New Relic

event_stores:
  - type: otel
    endpoint: "otlp.nr-data.net:4317"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      api-key:
        type: env
        value: NEW_RELIC_LICENSE_KEY
    tls:
      enabled: true

Grafana Cloud

event_stores:
  - type: otel
    endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      authorization:
        type: env
        value: GRAFANA_CLOUD_API_KEY  # Format: "Basic base64(instanceID:apiKey)"
    tls:
      enabled: true

Elastic

event_stores:
  - type: otel
    endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      Authorization:
        type: env
        value: ELASTIC_APM_SECRET_TOKEN
    tls:
      enabled: true

Self-Hosted

For self-hosted backends (Jaeger, Zipkin, Grafana Tempo, etc.), point qtap to your OpenTelemetry Collector, then configure the collector to export to your backend:

# Qtap config - same for all self-hosted backends
event_stores:
  - type: otel
    endpoint: "otel-collector.monitoring.svc.cluster.local:4317"
    protocol: grpc

Then configure your OTel Collector with the appropriate exporter for your backend.

Querying and Filtering

Filter by Event Type

# Show only HTTP transaction summaries
event.type = "artifact_record"

# Show only TCP connections
event.type = "connection"

Find Slow Requests

event.type = "artifact_record" AND event.summary.duration_ms > 1000

Find Errors

# HTTP errors
event.type = "artifact_record" AND event.summary.response_status >= 500

Track Specific Endpoints

# Specific host
event.type = "artifact_record" AND event.summary.request_host = "api.example.com"

# Specific path pattern
event.type = "artifact_record" AND event.summary.request_path LIKE "/api/users/%"

Monitor HTTP/2 vs HTTP/1 Traffic

# HTTP/2 connections (logs)
event.type = "connection" AND event.l7Protocol = "http2"

# HTTP/1 connections (logs)
event.type = "connection" AND event.l7Protocol = "http1"

Troubleshooting

No Logs Appearing

Check qtap logs for OTel connection:

docker logs qtap | grep -i "otel\|error"

Common issues:

1. Wrong endpoint - Verify the endpoint address and port

   • gRPC default: 4317

   • HTTP default: 4318

   • With --network=host: use localhost:4317

2. TLS mismatch - If backend requires TLS, set tls.enabled: true

3. Authentication - Verify headers/API keys are correct

4. Collector not running - Check docker ps | grep otel

Connection Refused Errors

rpc error: code = Unavailable desc = name resolver error

Solutions:

• Check if OTel Collector is running: docker ps | grep otel

• Verify network connectivity between qtap and collector

• Check firewall rules for port 4317/4318

Debugging with stdout Protocol

For local debugging, use the stdout protocol to see OTLP data printed to qtap logs:

event_stores:
  - type: otel
    protocol: stdout  # Prints OTLP data to console
    service_name: "qtap"
    environment: "development"

Then check qtap logs:

docker logs qtap

Best Practices

1. Use TLS in production - Always enable tls.enabled: true for production

2. Store artifacts securely - Use OTel object stores for unified pipelines, or S3 for separate storage

3. Filter appropriately - Use qtap's filters to avoid capturing unnecessary traffic

4. Set resource attributes - Use service_name and environment for filtering

5. Monitor qtap itself - Set up alerts on qtap's Prometheus metrics

6. Use batch processing - OTel Collector's batch processor reduces API calls

7. Use connection.id for correlation - Link connection events with artifact records using connection.id

Configuration Quick Reference

qtap.yaml:

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc

Next Steps

• Storage Configuration — OTel and S3 object storage options

• Self-Hosted ClickStack Observability — Full setup with OTel events + objects

• Set up Prometheus Metrics for qtap health monitoring

• Configure Traffic Filters to reduce noise

Additional Resources

• Storage Configuration Reference

• OpenTelemetry Collector Documentation

• OTLP Specification