Integrating Qtap with Your Observability Stack

You already have an observability stack — Grafana + Loki, Datadog, Elastic, or similar. This guide shows how Qtap slots into what you already run, without provisioning new infrastructure. Qtap speaks OTLP for both events and artifacts. You can also use S3 for artifact storage if you prefer. If your stack can receive OTLP, you're set.

Choose your path:

• Simplest setup: Send both events and artifacts through OTel → Self-Hosted ClickStack Observability

• Full Grafana stack from scratch: → Self-Hosted Grafana Observability Stack

• Integrate with existing infrastructure: Keep reading below

How Qtap Fits In

┌──────────────────────────────────────────────────────────────────┐
│                        YOUR INFRASTRUCTURE                       │
│                                                                  │
│   ┌───────┐                                                      │
│   │ Qtap  │──── Events (OTLP) ──────► OTel Collector             │
│   │ Agent │                               │                      │
│   │       │                               ▼                      │
│   │       │                        Your Log Backend              │
│   │       │                     (Loki, Datadog, Elastic, …)      │
│   │       │                                                      │
│   │       │── Objects ──► OTel (same collector)                   │
│   │       │          OR ► S3 Storage (MinIO, AWS S3, GCS, …)     │
│   └───────┘                                                      │
└──────────────────────────────────────────────────────────────────┘

Qtap produces two data streams: events and objects. Both can flow through your OTel pipeline, or you can send objects to S3 storage separately. When using S3, the access_url in each artifact_record event links back to the stored object.

Two Data Streams

	Events	Objects
What	Lightweight metadata — method, URL, status, duration, process	HTTP transaction content — metadata at summary level; headers and bodies at full level
Sensitivity	Low — safe to send to any backend	Varies — summary objects contain only metadata; full objects may contain API keys, tokens, PII
Protocol	OTLP (gRPC or HTTP)	OTLP (gRPC or HTTP) or S3 API
Destination	Any OTLP-compatible backend via OTel Collector	Same OTel Collector or any S3-compatible storage
Volume	Every observed request	Every captured request (content varies by capture level)

Choose your object storage approach:

• OTel objects — Simplest. Events and artifacts flow through the same collector. No S3 credentials needed. Best for ClickStack, self-contained setups, or when you don't need separate object storage.

• S3 objects — Events go to OTel, artifacts go to S3. Each artifact_record event contains a clickable url linking to the stored object. Best when you need durable, queryable object storage or have existing S3 infrastructure.

Configuring Qtap

Option 1: Everything through OTel (simplest)

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

  object_stores:
    - type: otel
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: full
          format: json

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack

Both events and artifacts go to the same OTel endpoint. No S3 credentials, no bucket setup.

Option 2: OTel events + S3 objects

One complete qtap.yaml that connects to your existing OTel Collector and S3 storage:

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"           # Your OTel Collector gRPC endpoint
      protocol: grpc                       # (grpc|http|stdout)
      service_name: "qtap"                 # Service name in your log backend
      environment: "production"            # Environment tag for filtering
      tls:
        enabled: false                     # Set true for production / remote collectors

  object_stores:
    - id: s3
      type: s3
      endpoint: minio.internal:9000        # Your S3-compatible storage endpoint
      bucket: qpoint                       # Your bucket name
      region: us-east-1                    # Your region
      access_url: https://objects.internal.example.com/qpoint/{{DIGEST}}
      insecure: false                      # Set true for HTTP-only endpoints
      access_key:
        type: env
        value: S3_ACCESS_KEY               # Environment variable name
      secret_key:
        type: env
        value: S3_SECRET_KEY               # Environment variable name

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400 && http.res.status < 600

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary                   # (none|summary|headers|full)
          format: json                     # (json|text)
          rules:
            - name: "Full capture on errors"
              expr: is_error()
              level: full                  # Stores headers + bodies in S3

tap:
  direction: egress                        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true                    # (true|false)
  audit_include_dns: false                 # (true|false)
  http:
    stack: default_stack
  filters:
    groups:
      - qpoint                             # Don't capture Qtap's own traffic

Replace these values with yours:

Placeholder	What to set
endpoint (event_stores)	Your OTel Collector address (e.g., otel-collector:4317)
endpoint (object_stores)	Your S3 storage address (e.g., s3.amazonaws.com)
bucket	Your S3 bucket name
region	Your S3 region
access_url	URL template that your dashboard users can reach (see Connecting to Your S3 Storage)
S3_ACCESS_KEY / S3_SECRET_KEY	Environment variable names holding your S3 credentials

Connecting to Your OTel Collector

If you already have an OTel Collector running, point Qtap's event_stores endpoint at it — no other changes needed. Qtap sends OTLP logs on the standard gRPC port (4317).

If your collector doesn't yet forward to a log backend, add the relevant exporter. For example, to forward to Loki:

# Add to your existing OTel Collector config
exporters:
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]          # Add alongside your existing exporters

This works with any OTLP-compatible backend — replace the exporter with whatever your stack uses (Datadog, Elastic, Honeycomb, etc.). See Backend-Specific Configurations for detailed examples.

Connecting to Your S3 Storage

Qtap's S3 object store works with any S3-compatible storage. Set the endpoint to match your provider:

Provider	Endpoint Example	access_url Example
MinIO	minio.internal:9000	https://minio.internal/{{BUCKET}}/{{DIGEST}}
AWS S3	s3.amazonaws.com	https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
Google Cloud Storage	storage.googleapis.com	https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
Garage	garage.internal:3900	http://garage.internal:3902/{{BUCKET}}/{{DIGEST}}

The access_url Template

The access_url determines the clickable link embedded in every artifact_record event. It supports three template variables:

• {{DIGEST}} — SHA1 hash of the stored object (always required)

• {{BUCKET}} — The bucket name from your config

• {{ENDPOINT}} — The S3 endpoint from your config

The URL must be reachable by whoever is reading your logs. For internal dashboards, use an internal hostname. For shared access, consider:

• Presigned URLs — Generate time-limited signed URLs for private buckets

• Public bucket — Allow anonymous reads (development only)

• Reverse proxy — Place nginx or similar in front of your S3 for access control

What You'll See

Qtap stores an HTTP transaction object in S3 and emits an artifact_record event for every captured request. At summary level, the object contains only metadata. At headers or full level, it includes the complete request/response headers and bodies.

The artifact_record Event

Search your log backend for event_type="artifact_record" to find stored objects:

{
  "type": "http_transaction",
  "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c",
  "url": "https://objects.internal.example.com/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c",
  "summary": {
    "request_method": "GET",
    "request_host": "httpbin.org",
    "response_status": 500,
    "duration_ms": 580,
    "process_exe": "/usr/bin/curl",
    "direction": "egress-external"
  }
}

The Stored Object

Click the url to fetch the full HTTP transaction from your S3 storage:

{
  "metadata": {
    "process_exe": "/usr/bin/curl",
    "container_name": "relaxed_zhukovsky",
    "container_image": "curlimages/curl",
    "bytes_sent": 72,
    "bytes_received": 127,
    "connection_id": "d654d887p3qnd27vfhag",
    "endpoint_id": "httpbin.org"
  },
  "request": {
    "method": "GET",
    "url": "https://httpbin.org/status/500",
    "scheme": "https",
    "authority": "httpbin.org",
    "protocol": "http2",
    "user_agent": "curl/8.18.0",
    "headers": {
      ":authority": "httpbin.org",
      ":method": "GET",
      ":path": "/status/500",
      "Accept": "*/*",
      "User-Agent": "curl/8.18.0"
    }
  },
  "response": {
    "status": 500,
    "content_type": "text/html; charset=utf-8",
    "headers": {
      "Content-Type": "text/html; charset=utf-8",
      "Server": "gunicorn/19.9.0"
    }
  },
  "duration_ms": 767,
  "direction": "egress-external"
}

This works in any log UI — search for artifact_record events, click the URL, and you have the full request and response.

All capture levels generate artifact_record events and store objects in S3. The difference is content: summary objects contain only metadata (method, URL, status), while headers and full objects include the complete request/response headers and bodies.

Common Configurations

These snippets show just the services: block — the rest of the config (rulekit, stacks, tap) stays the same as the full example above.

ClickStack (all OTel)

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: otel
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

See Self-Hosted ClickStack Observability for the full walkthrough.

Loki + MinIO

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: minio.internal:9000
      bucket: qpoint
      region: us-east-1
      access_url: https://minio.internal/qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: MINIO_ACCESS_KEY
      secret_key:
        type: env
        value: MINIO_SECRET_KEY

Datadog + AWS S3

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      bucket: my-company-qpoint
      region: us-west-2
      access_url: https://s3.amazonaws.com/my-company-qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY

Elastic + Google Cloud Storage

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: storage.googleapis.com
      bucket: my-company-qpoint
      region: us-central1
      access_url: https://storage.cloud.google.com/my-company-qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: GCS_ACCESS_KEY
      secret_key:
        type: env
        value: GCS_SECRET_KEY

Verification Checklist

1. Check OTel Collector — Look for received log entries in your collector's output

2. Query your log backend — Search for service_name="qtap" events

3. List S3 objects — Verify objects appear in your bucket after generating traffic

4. Test object linking — Find an artifact_record event and open the URL to confirm the full HTTP transaction is accessible

Next Steps

• Self-Hosted ClickStack Observability — Simplest self-hosted setup (everything through OTel)

• Self-Hosted Grafana Observability Stack — Build the full Grafana stack from scratch

• Storage Configuration — OTel and S3 object storage options, credential management

• Traffic Processing with Plugins — Rulekit rules, capture levels, conditional logic

• Qpoint Data Schema Reference — Full event schema documentation