Integrating Qtap with Your Observability Stack

You already have an observability stack — Grafana + Loki, Datadog, Elastic, or similar — and S3-compatible object storage. This guide shows how Qtap slots into what you already run, without provisioning new infrastructure. Qtap speaks OTLP for events and S3 for payloads. If your stack can receive either, you're set.

Need to build the full stack from scratch? See the Self-Hosted Grafana Observability Stack guide instead.

How Qtap Fits In

┌──────────────────────────────────────────────────────────────────┐
│                        YOUR INFRASTRUCTURE                       │
│                                                                  │
│   ┌───────┐                                                      │
│   │ Qtap  │──── Events (OTLP) ──────► OTel Collector             │
│   │ Agent │                               │                      │
│   │       │                               ▼                      │
│   │       │                        Your Log Backend              │
│   │       │                     (Loki, Datadog, Elastic, …)      │
│   │       │                               ▲                      │
│   │       │                               │ artifact URLs        │
│   │       │                               ▼                      │
│   │       │── Objects (S3) ──────►  Your S3 Storage              │
│   │       │                      (MinIO, AWS S3, GCS, …)         │
│   └───────┘                                                      │
└──────────────────────────────────────────────────────────────────┘

Qtap produces two independent data streams. Events flow through your existing OTel pipeline. Objects go directly to your S3 storage. The access_url in each artifact_record event links the two together — click it to fetch the full HTTP transaction from S3.

Two Data Paths

	Events	Objects
What	Lightweight metadata — method, URL, status, duration, process	HTTP transaction objects — metadata at summary level; headers and bodies at full level
Sensitivity	Low — safe to send to any backend	Varies — summary objects contain only metadata; full objects may contain API keys, tokens, PII
Protocol	OTLP (gRPC or HTTP)	S3 API
Destination	Any OTLP-compatible backend via OTel Collector	Any S3-compatible object storage
Volume	Every observed request	Every captured request (content varies by capture level)

The link between them: When Qtap stores an object in S3, it emits an artifact_record event containing a url field built from your access_url template. The {{DIGEST}} placeholder is replaced with the SHA1 hash of the stored object, creating a direct link from your log UI to the full HTTP transaction.

Configuring Qtap

One complete qtap.yaml that connects to your existing OTel Collector and S3 storage:

version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"           # Your OTel Collector gRPC endpoint
      protocol: grpc                       # (grpc|http|stdout)
      service_name: "qtap"                 # Service name in your log backend
      environment: "production"            # Environment tag for filtering
      tls:
        enabled: false                     # Set true for production / remote collectors

  object_stores:
    - id: s3
      type: s3
      endpoint: minio.internal:9000        # Your S3-compatible storage endpoint
      bucket: qpoint                       # Your bucket name
      region: us-east-1                    # Your region
      access_url: https://objects.internal.example.com/qpoint/{{DIGEST}}
      insecure: false                      # Set true for HTTP-only endpoints
      access_key:
        type: env
        value: S3_ACCESS_KEY               # Environment variable name
      secret_key:
        type: env
        value: S3_SECRET_KEY               # Environment variable name

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400 && http.res.status < 600

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary                   # (none|summary|headers|full)
          format: json                     # (json|text)
          rules:
            - name: "Full capture on errors"
              expr: is_error()
              level: full                  # Stores headers + bodies in S3

tap:
  direction: egress                        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true                    # (true|false)
  audit_include_dns: false                 # (true|false)
  http:
    stack: default_stack
  filters:
    groups:
      - qpoint                             # Don't capture Qtap's own traffic

Replace these values with yours:

Placeholder	What to set
endpoint (event_stores)	Your OTel Collector address (e.g., otel-collector:4317)
endpoint (object_stores)	Your S3 storage address (e.g., s3.amazonaws.com)
bucket	Your S3 bucket name
region	Your S3 region
access_url	URL template that your dashboard users can reach (see Connecting to Your S3 Storage)
S3_ACCESS_KEY / S3_SECRET_KEY	Environment variable names holding your S3 credentials

Connecting to Your OTel Collector

If you already have an OTel Collector running, point Qtap's event_stores endpoint at it — no other changes needed. Qtap sends OTLP logs on the standard gRPC port (4317).

If your collector doesn't yet forward to a log backend, add the relevant exporter. For example, to forward to Loki:

# Add to your existing OTel Collector config
exporters:
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]          # Add alongside your existing exporters

This works with any OTLP-compatible backend — replace the exporter with whatever your stack uses (Datadog, Elastic, Honeycomb, etc.). See Backend-Specific Configurations for detailed examples.

Connecting to Your S3 Storage

Qtap's S3 object store works with any S3-compatible storage. Set the endpoint to match your provider:

Provider	Endpoint Example	access_url Example
MinIO	minio.internal:9000	https://minio.internal/{{BUCKET}}/{{DIGEST}}
AWS S3	s3.amazonaws.com	https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
Google Cloud Storage	storage.googleapis.com	https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
Garage	garage.internal:3900	http://garage.internal:3902/{{BUCKET}}/{{DIGEST}}

The access_url Template

The access_url determines the clickable link embedded in every artifact_record event. It supports three template variables:

• {{DIGEST}} — SHA1 hash of the stored object (always required)

• {{BUCKET}} — The bucket name from your config

• {{ENDPOINT}} — The S3 endpoint from your config

The URL must be reachable by whoever is reading your logs. For internal dashboards, use an internal hostname. For shared access, consider:

• Presigned URLs — Generate time-limited signed URLs for private buckets

• Public bucket — Allow anonymous reads (development only)

• Reverse proxy — Place nginx or similar in front of your S3 for access control

What You'll See

Qtap stores an HTTP transaction object in S3 and emits an artifact_record event for every captured request. At summary level, the object contains only metadata. At headers or full level, it includes the complete request/response headers and bodies.

The artifact_record Event

Search your log backend for event_type="artifact_record" to find stored objects:

{
  "type": "http_transaction",
  "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c",
  "url": "https://objects.internal.example.com/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c",
  "summary": {
    "request_method": "GET",
    "request_host": "httpbin.org",
    "response_status": 500,
    "duration_ms": 580,
    "process_exe": "/usr/bin/curl",
    "direction": "egress-external"
  }
}

The Stored Object

Click the url to fetch the full HTTP transaction from your S3 storage:

{
  "metadata": {
    "process_exe": "/usr/bin/curl",
    "container_name": "relaxed_zhukovsky",
    "container_image": "curlimages/curl",
    "bytes_sent": 72,
    "bytes_received": 127,
    "connection_id": "d654d887p3qnd27vfhag",
    "endpoint_id": "httpbin.org"
  },
  "request": {
    "method": "GET",
    "url": "https://httpbin.org/status/500",
    "scheme": "https",
    "authority": "httpbin.org",
    "protocol": "http2",
    "user_agent": "curl/8.18.0",
    "headers": {
      ":authority": "httpbin.org",
      ":method": "GET",
      ":path": "/status/500",
      "Accept": "*/*",
      "User-Agent": "curl/8.18.0"
    }
  },
  "response": {
    "status": 500,
    "content_type": "text/html; charset=utf-8",
    "headers": {
      "Content-Type": "text/html; charset=utf-8",
      "Server": "gunicorn/19.9.0"
    }
  },
  "duration_ms": 767,
  "direction": "egress-external"
}

This works in any log UI — search for artifact_record events, click the URL, and you have the full request and response.

All capture levels generate artifact_record events and store objects in S3. The difference is content: summary objects contain only metadata (method, URL, status), while headers and full objects include the complete request/response headers and bodies.

Common Configurations

These snippets show just the services: block — the rest of the config (rulekit, stacks, tap) stays the same as the full example above.

Loki + MinIO

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: minio.internal:9000
      bucket: qpoint
      region: us-east-1
      access_url: https://minio.internal/qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: MINIO_ACCESS_KEY
      secret_key:
        type: env
        value: MINIO_SECRET_KEY

Datadog + AWS S3

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      bucket: my-company-qpoint
      region: us-west-2
      access_url: https://s3.amazonaws.com/my-company-qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY

Elastic + Google Cloud Storage

services:
  event_stores:
    - type: otel
      endpoint: "otel-collector:4317"
      protocol: grpc
      service_name: "qtap"
      tls:
        enabled: false

  object_stores:
    - type: s3
      endpoint: storage.googleapis.com
      bucket: my-company-qpoint
      region: us-central1
      access_url: https://storage.cloud.google.com/my-company-qpoint/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: GCS_ACCESS_KEY
      secret_key:
        type: env
        value: GCS_SECRET_KEY

Verification Checklist

1. Check OTel Collector — Look for received log entries in your collector's output

2. Query your log backend — Search for service_name="qtap" events

3. List S3 objects — Verify objects appear in your bucket after generating traffic

4. Test object linking — Find an artifact_record event and open the URL to confirm the full HTTP transaction is accessible

Next Steps

• Self-Hosted Grafana Observability Stack — Build the full stack from scratch

• Storage Configuration — S3 parameters, credential management, backend examples

• Traffic Processing with Plugins — Rulekit rules, capture levels, conditional logic

• Qpoint Data Schema Reference — Full event schema documentation