OpenTelemetry Integration

This guide shows you how to send qtap observability data to OpenTelemetry-compatible backends using OTLP (OpenTelemetry Protocol).

Understanding Qtap's OpenTelemetry Integration

Qtap exports two types of data as OpenTelemetry Logs via the OTLP protocol:

Data Type

Configuration

What It Contains

Use Case

Events

qtap.yaml event_stores

Connection metadata, HTTP transaction summaries (method, URL, status, duration, process attribution)

Analytics, monitoring, troubleshooting — know WHAT happened and WHICH process did it

Objects

qtap.yaml object_stores

Full request/response headers and bodies

Compliance, security investigation, debugging — see the actual payload content

Both can be sent through the same OTel collector endpoint, giving you a unified pipeline for all captured data — no separate S3 setup required.

Events vs Objects: Events are lightweight metadata about every connection and HTTP transaction. Objects are the full request/response content, captured selectively based on your plugin configuration. You can send both through OTel, or use OTel for events and S3 for objects — see Storage Configuration for all options.

Prerequisites

Qtap installed and running (see Getting Started)
An OpenTelemetry-compatible backend or collector

Quick Start (5 minutes)

Send both events and artifacts to any OpenTelemetry-compatible backend.

Step 1: Configure Qtap

Create qtap-config.yaml:

version: 2

services:
  event_stores:
    - type: otel  # Send logs to OpenTelemetry
      endpoint: "localhost:4317"  # OTel Collector gRPC endpoint
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false  # Set to true for production

  object_stores:
    - type: otel  # Send artifacts through OTel (or use S3/stdout)
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

stacks:
  default:
    plugins:
      - type: http_capture
        config:
          level: summary  # (none|summary|headers|full)
          format: json    # (json|text)

tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)
  http:
    stack: default

Step 2: Deploy OpenTelemetry Collector

Already have an OpenTelemetry Collector? Update the endpoint in Step 1 to point to your collector and skip to Step 3.

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints logs to console for testing
  debug:
    verbosity: detailed

  # Add your backend exporter here
  # otlphttp:
  #   endpoint: "https://your-backend.com/v1/logs"
  #   headers:
  #     api-key: "${API_KEY}"

service:
  pipelines:
    logs:  # Logs pipeline for qtap event_stores
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # Add your backend exporter

Create docker-compose.yaml:

services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Start the collector:

docker compose up -d

Step 3: Start Qtap

docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Note: No additional environment variables are needed — logs are configured entirely through qtap.yaml.

Step 4: Generate Traffic and Verify

# Generate test traffic
curl https://httpbin.org/get

# Check OTel Collector logs
docker logs otel-collector

You should see two types of log events:

1. Connection Event:

LogRecord
Body: Connection: [egress-external via tcp] hostname → httpbin.org
Attributes:
  event.type: connection
  event.l7Protocol: http2
  event.tlsVersion: 772
  event.meta.connectionId: d3sggv07p3qrvfkj173g
  event.source.exe: /usr/bin/curl

2. Artifact Record Event:

LogRecord
Body: Artifact stored: http_transaction
Attributes:
  event.type: artifact_record
  event.summary.request_method: GET
  event.summary.request_host: httpbin.org
  event.summary.response_status: 200
  event.summary.duration_ms: 2023
  event.summary.connection_id: d3sggv07p3qrvfkj173g

3. Object (Artifact Data):

When using OTel object stores, the full request/response content is also sent as log records:

LogRecord
Body: Artifact: http_request (application/json, 1247 bytes)
Attributes:
  artifact.type: http_request
  artifact.content_type: application/json
  artifact.digest: sha256:a1b2c3d4...
  artifact.size_bytes: 1247
  artifact.data: <raw bytes — full request/response content>
  connection.id: d3sggv07p3qrvfkj173g
  connection.endpoint_id: httpbin.org
  connection.request_id: d3sggv07p3qrvfkj174g

Objects include the complete HTTP headers and bodies as captured by the http_capture plugin. The connection.id links objects back to their corresponding connection and artifact record events.

Success! Qtap is sending events and artifacts to OpenTelemetry. Use connection.id to correlate connection metadata, HTTP transaction summaries, and full request/response content.

Understanding the Data

What Logs Contain

Connection Events (event.type: connection):

TCP connection metadata
Protocol detection (http1, http2, tcp)
TLS version and inspection status
Bytes sent/received
Process information (exe path, user ID)

Artifact Record Events (event.type: artifact_record):

HTTP method, URL, path, status code
User agent and content type
Duration in milliseconds
Process information
Link to stored artifact (OTel or S3 URL depending on object store type)

What Objects Contain

When using OTel object stores (object_stores type: otel), artifact data is sent as additional log records:

artifact.type — http_request or http_response
artifact.content_type — MIME type (e.g., application/json)
artifact.data — The raw bytes (full headers and body)
artifact.size_bytes — Size of the artifact
artifact.digest — Content hash for deduplication
connection.id — Links back to the connection and artifact record events
artifact.summary.* — Request/response metadata (method, host, status, etc.)

Events vs Objects: Events (event_stores) are lightweight metadata sent for every connection and HTTP transaction. Objects (object_stores) are the full payload content, captured based on your http_capture plugin level setting (summary, details, or full).

Example log query:

event.type = "artifact_record" AND event.summary.response_status >= 500

Kubernetes Deployment

Using OpenTelemetry Operator

Already have OpenTelemetry Operator installed? Skip to step 2.

Install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Deploy an OpenTelemetry Collector:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qtap-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 10s

    exporters:
      otlphttp:
        endpoint: "https://your-backend.com/v1/logs"
        headers:
          api-key: "${API_KEY}"

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]

Deploy qtap using Helm:

First, create your qtap configuration file qtap-config.yaml:

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

  object_stores:
    - type: otel
      otel_endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: json

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack

Install qtap with Helm:

# Add Qpoint Helm repository
helm repo add qpoint https://helm.qpoint.io/
helm repo update

# Install qtap with config
helm install qtap qpoint/qtap \
  -n qpoint \
  --create-namespace \
  --set-file config=./qtap-config.yaml \
  --set logLevel=warn

Verify the deployment:

kubectl get pods -n qpoint
kubectl logs -n qpoint -l app.kubernetes.io/name=qtap --tail=50

Backend-Specific Configurations

SaaS Platforms

Datadog

Datadog supports OTLP ingestion directly:

event_stores:
  - type: otel
    endpoint: "https://http-intake.logs.datadoghq.com/v2/logs"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      DD-API-KEY:
        type: env
        value: DATADOG_API_KEY
    tls:
      enabled: true

Honeycomb

event_stores:
  - type: otel
    endpoint: "api.honeycomb.io:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      x-honeycomb-team:
        type: env
        value: HONEYCOMB_API_KEY
      x-honeycomb-dataset:
        type: text
        value: "qtap-logs"
    tls:
      enabled: true

New Relic

event_stores:
  - type: otel
    endpoint: "otlp.nr-data.net:4317"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      api-key:
        type: env
        value: NEW_RELIC_LICENSE_KEY
    tls:
      enabled: true

Grafana Cloud

event_stores:
  - type: otel
    endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      authorization:
        type: env
        value: GRAFANA_CLOUD_API_KEY  # Format: "Basic base64(instanceID:apiKey)"
    tls:
      enabled: true

Elastic

event_stores:
  - type: otel
    endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      Authorization:
        type: env
        value: ELASTIC_APM_SECRET_TOKEN
    tls:
      enabled: true

Self-Hosted

For self-hosted backends (Jaeger, Zipkin, Grafana Tempo, etc.), point qtap to your OpenTelemetry Collector, then configure the collector to export to your backend:

# Qtap config - same for all self-hosted backends
event_stores:
  - type: otel
    endpoint: "otel-collector.monitoring.svc.cluster.local:4317"
    protocol: grpc

Then configure your OTel Collector with the appropriate exporter for your backend.

Querying and Filtering

Filter by Event Type

# Show only HTTP transaction summaries
event.type = "artifact_record"

# Show only TCP connections
event.type = "connection"

Find Slow Requests

event.type = "artifact_record" AND event.summary.duration_ms > 1000

Find Errors

# HTTP errors
event.type = "artifact_record" AND event.summary.response_status >= 500

Track Specific Endpoints

# Specific host
event.type = "artifact_record" AND event.summary.request_host = "api.example.com"

# Specific path pattern
event.type = "artifact_record" AND event.summary.request_path LIKE "/api/users/%"

Monitor HTTP/2 vs HTTP/1 Traffic

# HTTP/2 connections (logs)
event.type = "connection" AND event.l7Protocol = "http2"

# HTTP/1 connections (logs)
event.type = "connection" AND event.l7Protocol = "http1"

Troubleshooting

No Logs Appearing

Check qtap logs for OTel connection:

docker logs qtap | grep -i "otel\|error"

Common issues:

Wrong endpoint - Verify the endpoint address and port
- gRPC default: 4317
- HTTP default: 4318
- With --network=host: use localhost:4317
TLS mismatch - If backend requires TLS, set tls.enabled: true
Authentication - Verify headers/API keys are correct
Collector not running - Check docker ps | grep otel

Connection Refused Errors

rpc error: code = Unavailable desc = name resolver error

Solutions:

Check if OTel Collector is running: docker ps | grep otel
Verify network connectivity between qtap and collector
Check firewall rules for port 4317/4318

Debugging with stdout Protocol

For local debugging, use the stdout protocol to see OTLP data printed to qtap logs:

event_stores:
  - type: otel
    protocol: stdout  # Prints OTLP data to console
    service_name: "qtap"
    environment: "development"

Then check qtap logs:

docker logs qtap

Best Practices

Use TLS in production - Always enable tls.enabled: true for production
Store artifacts securely - Use OTel object stores for unified pipelines, or S3 for separate storage
Filter appropriately - Use qtap's filters to avoid capturing unnecessary traffic
Set resource attributes - Use service_name and environment for filtering
Monitor qtap itself - Set up alerts on qtap's Prometheus metrics
Use batch processing - OTel Collector's batch processor reduces API calls
Use connection.id for correlation - Link connection events with artifact records using connection.id

Configuration Quick Reference

qtap.yaml:

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc

Next Steps

Storage Configuration — OTel and S3 object storage options
Self-Hosted ClickStack Observability — Full setup with OTel events + objects
Set up Prometheus Metrics for qtap health monitoring
Configure Traffic Filters to reduce noise

Additional Resources

PreviousPrometheus + Grafana Monitoring NextExporting HTTP Headers to OpenTelemetry

Last updated 1 day ago

hashtagUnderstanding Qtap's OpenTelemetry Integration

hashtagPrerequisites

hashtagQuick Start (5 minutes)

hashtagStep 1: Configure Qtap

hashtagStep 2: Deploy OpenTelemetry Collector

hashtagStep 3: Start Qtap

hashtagStep 4: Generate Traffic and Verify

hashtagUnderstanding the Data

hashtagWhat Logs Contain

hashtagWhat Objects Contain

hashtagKubernetes Deployment

hashtagUsing OpenTelemetry Operator

hashtagBackend-Specific Configurations

hashtagSaaS Platforms

hashtagDatadog

hashtagHoneycomb

hashtagNew Relic

hashtagGrafana Cloud

hashtagElastic

hashtagSelf-Hosted

hashtagQuerying and Filtering

hashtagFilter by Event Type

hashtagFind Slow Requests

hashtagFind Errors

hashtagTrack Specific Endpoints

hashtagMonitor HTTP/2 vs HTTP/1 Traffic

hashtagTroubleshooting

hashtagNo Logs Appearing

hashtagConnection Refused Errors

hashtagDebugging with stdout Protocol

hashtagBest Practices

hashtagConfiguration Quick Reference

hashtagNext Steps

hashtagAdditional Resources

Understanding Qtap's OpenTelemetry Integration

Prerequisites

Quick Start (5 minutes)

Step 1: Configure Qtap

Step 2: Deploy OpenTelemetry Collector

Step 3: Start Qtap

Step 4: Generate Traffic and Verify

Understanding the Data

What Logs Contain

What Objects Contain

Kubernetes Deployment

Using OpenTelemetry Operator

Backend-Specific Configurations

SaaS Platforms

Datadog

Honeycomb

New Relic

Grafana Cloud

Elastic

Self-Hosted

Querying and Filtering

Filter by Event Type

Find Slow Requests

Find Errors

Track Specific Endpoints

Monitor HTTP/2 vs HTTP/1 Traffic

Troubleshooting

No Logs Appearing

Connection Refused Errors

Debugging with stdout Protocol

Best Practices

Configuration Quick Reference

Next Steps

Additional Resources