Self-Hosted Grafana Observability Stack

This guide walks you through setting up a complete, self-hosted observability stack where Qtap events flow to Grafana via Loki and full HTTP payloads stay in your own S3-compatible object storage. Every byte of captured data — metadata and payloads — remains inside your infrastructure.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        YOUR INFRASTRUCTURE                       │
│                                                                  │
│   ┌───────┐                                                      │
│   │ Qtap  │──── Events (metadata) ───► OTel Collector (:4317)    │
│   │ Agent │                               │                      │
│   │       │                               ▼                      │
│   │       │                           Loki (:3100)               │
│   │       │                               │                      │
│   │       │                               ▼                      │
│   │       │                           Grafana (:3000)            │
│   │       │                               ▲                      │
│   │       │                               │ click artifact URL   │
│   │       │                               ▼                      │
│   │       │                       nginx proxy (:3904)            │
│   │       │                               │                      │
│   │       │                               ▼                      │
│   │       │── Objects (payloads) ──► Garage S3 (:3900)           │
│   └───────┘                                                      │
└──────────────────────────────────────────────────────────────────┘

Service	Role	Port
Qtap	eBPF agent — captures HTTP traffic at kernel level	Host network
OTel Collector	Receives OTLP logs from Qtap, forwards to Loki	4317 (gRPC)
Loki	Log aggregation and storage	3100
Grafana	Query, explore, and visualize events	3000
Garage	S3-compatible object storage for HTTP payloads	3900 (S3 API)
nginx	Proxy for anonymous read access to stored objects	3904

This guide covers events and object linking. For sending events to any OpenTelemetry-compatible backend, see the OpenTelemetry Integration guide.

Prerequisites

• Docker and Docker Compose

• Linux kernel 5.10+ with eBPF support

• aws CLI (for S3 verification) — install guide

Understanding the Two Data Paths

Qtap produces two distinct outputs. Understanding the split is key to this architecture:

	Events	Objects
What	Lightweight metadata — method, URL, status, duration, process	HTTP transaction objects — metadata at summary level; headers and bodies at full level
Sensitivity	Low — safe to send anywhere	Varies — summary objects contain only metadata; full objects may contain API keys, tokens, PII
Storage	Loki (via OTel Collector)	Garage S3 (your infrastructure)
Volume	Every observed request	Every captured request (content varies by capture level)

The link between them: Qtap's access_url template embeds a clickable URL into each artifact_record event. When you find an interesting event in Grafana, you click the URL to fetch the complete HTTP transaction from your S3 storage.

access_url: http://localhost:3904/qpoint/{{DIGEST}}
                                         ^^^^^^^^
                                         Replaced with SHA1 hash of the stored object

Configuration Files

Create a project directory and add these files:

mkdir -p grafana-stack/grafana/provisioning/datasources
cd grafana-stack

OTel Collector

cat > otel-collector.yaml << 'EOF'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp/loki:
    endpoint: http://localhost:3100/otlp
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]
EOF

The collector listens for gRPC on port 4317 (where Qtap sends events) and forwards them to Loki's OTLP endpoint. Since the collector runs with network_mode: host, it reaches Loki at localhost:3100.

Loki

cat > loki.yaml << 'EOF'
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h
  allow_structured_metadata: true
  max_query_series: 100000

compactor:
  working_directory: /loki/compactor
  delete_request_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
EOF

Key settings: allow_structured_metadata: true lets Loki store Qtap's structured attributes (method, status, host, etc.) as queryable fields. Retention is set to 7 days (168h).

Garage (S3-Compatible Object Storage)

cat > garage.toml << 'EOF'
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "c052485a056fabf3c0832d98f63b14d58036f5189683fc39da199f43fde3f15e"

[s3_api]
s3_region = "us-east-1"
api_bind_addr = "[::]:3900"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage.localhost"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "demo-admin-token-for-local-use-only"
EOF

Garage provides the S3 API on port 3900 (where Qtap writes payloads), a web endpoint on 3902 (for anonymous reads), and an admin API on 3903 (for bucket management).

Nginx Proxy

cat > nginx.conf << 'EOF'
server {
    listen 3904;

    location /qpoint/ {
        rewrite ^/qpoint/(.*)$ /$1 break;
        proxy_pass http://garage:3902;
        proxy_set_header Host qpoint.web.garage.localhost;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
EOF

The nginx proxy translates requests like http://localhost:3904/qpoint/<DIGEST> into Garage web requests with the correct virtual-host Host header. This avoids needing wildcard DNS for Garage's subdomain-based routing.

Grafana Datasource

cat > grafana/provisioning/datasources/datasources.yaml << 'EOF'
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
    isDefault: true
    editable: true
EOF

Qtap

cat > qtap.yaml << 'EOF'
version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"      # OTel Collector gRPC
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

  object_stores:
    - id: garage
      type: s3
      endpoint: localhost:3900        # Garage S3 API
      bucket: qpoint
      region: us-east-1
      access_url: http://localhost:3904/qpoint/{{DIGEST}}
      insecure: true
      access_key:
        type: env
        value: GARAGE_ACCESS_KEY
      secret_key:
        type: env
        value: GARAGE_SECRET_KEY

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400 && http.res.status < 600

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary             # (none|summary|headers|full)
          format: json               # (json|text)
          rules:
            - name: "Full capture on errors"
              expr: is_error()
              level: full            # Stores headers + bodies in S3

tap:
  direction: egress                  # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true              # (true|false)
  audit_include_dns: false           # (true|false)
  http:
    stack: default_stack
  filters:
    groups:
      - qpoint                       # Don't capture Qtap's own traffic
EOF

This configuration captures all egress HTTP traffic at summary level (metadata only — no headers or bodies), and automatically escalates to full capture for any 4xx or 5xx response. Full captures store complete request/response headers in Garage S3. Both levels emit artifact_record events with clickable URLs pointing to the stored objects.

Docker Compose

cat > docker-compose.yaml << 'EOF'
services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: always
    ports:
      - "3100:3100"
    volumes:
      - ./loki.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    restart: always
    network_mode: host
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector-config.yaml:ro
    command: ["--config=/etc/otel-collector-config.yaml"]

  garage:
    image: dxflrs/garage:v2.1.0
    container_name: garage
    restart: always
    ports:
      - "3900:3900"
      - "3902:3902"
      - "3903:3903"
    volumes:
      - ./garage.toml:/etc/garage.toml:ro

  garage-web:
    image: nginx:alpine
    container_name: garage-web
    restart: always
    ports:
      - "3904:3904"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - garage

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Editor

volumes:
  loki-data:
  grafana-data:
EOF

The OTel Collector uses network_mode: host so that Qtap (running on the host) can reach it at localhost:4317. Because it shares the host network stack, it also reaches Loki at localhost:3100.

Running the Stack

Step 1: Start the Services

docker compose up -d

Verify all containers are running:

docker compose ps

Step 2: Initialize Garage

Wait for Garage to be ready, then configure the cluster layout, create a bucket, and set up access credentials:

# Wait for Garage to start
sleep 5

# Get the node ID
NODE_ID=$(docker exec garage /garage status 2>/dev/null | grep -oE '[a-f0-9]{16}' | head -1)
echo "Node ID: $NODE_ID"

# Configure cluster layout
docker exec garage /garage layout assign -z dc1 -c 1G "$NODE_ID"
docker exec garage /garage layout apply --version 1

# Create the bucket
docker exec garage /garage bucket create qpoint

# Import S3 credentials
docker exec garage /garage key import \
  --yes \
  -n qtap-key \
  "GK0123456789abcdef01234567" \
  "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"

# Grant read/write access
docker exec garage /garage bucket allow \
  --read --write --owner qpoint --key qtap-key

# Enable anonymous web reads (for the nginx proxy)
docker exec garage /garage bucket website --allow qpoint

The credentials above are for local development only. For production, generate unique keys and manage them securely.

Step 3: Start Qtap

docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  -e GARAGE_ACCESS_KEY=GK0123456789abcdef01234567 \
  -e GARAGE_SECRET_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Step 4: Wait for Initialization

sleep 6

Qtap needs a few seconds to load eBPF programs and start capturing.

Step 5: Generate Test Traffic

Generate both a successful request and an error to see both data paths:

# Successful request — captured at summary level (metadata only, no headers/bodies)
docker run --rm curlimages/curl -s https://httpbin.org/get > /dev/null

# Error request — captured at full level (headers + bodies stored in S3)
docker run --rm curlimages/curl -s https://httpbin.org/status/500 > /dev/null

Step 6: Verify Data Is Flowing

Check OTel Collector is receiving data:

docker logs otel-collector 2>&1 | tail -5

You should see log export messages.

Check Loki has events:

curl -s "http://localhost:3100/loki/api/v1/query_range?query=%7Bservice_name%3D%22qtap%22%7D&limit=5" | python3 -m json.tool | head -20

Check S3 has objects (from the error request):

AWS_ACCESS_KEY_ID=GK0123456789abcdef01234567 \
AWS_SECRET_ACCESS_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
aws --endpoint-url http://localhost:3900 \
  --region us-east-1 \
  s3 ls s3://qpoint/

Exploring Data in Grafana

Open http://localhost:3000 in your browser (default credentials: admin / admin).

Viewing Events

1. Navigate to Explore (compass icon in the left sidebar)

2. Select Loki as the datasource

3. Enter a LogQL query:

{service_name="qtap"}

1. Click Run query

You should see Qtap events — both connection events (TCP connection metadata) and artifact_record events (HTTP transaction summaries, with full headers and bodies for requests matching capture rules).

Useful LogQL Queries

Query	Description
{service_name="qtap"}	All Qtap events
{service_name="qtap"} | json | event_type="artifact_record"	Only artifact records (objects stored in S3)
{service_name="qtap"} | json | response_status >= 400	Error responses
{service_name="qtap"} | json | request_host="httpbin.org"	Traffic to a specific host
{service_name="qtap"} | json | duration_ms > 1000	Slow requests (> 1s)
{service_name="qtap"} | json | process_exe="/usr/bin/curl"	Requests from curl
{service_name="qtap"} | json | direction="egress-external"	External egress traffic

Expanding Log Entries

Click on any log entry to expand it. You'll see structured attributes including:

• request_method, request_host, request_scheme

• response_status, duration_ms

• process_exe, direction

• For artifact records: digest, url, type

Object Linking — From Events to Full Payloads

This is the key capability of this stack: linking lightweight events in Grafana to complete HTTP transactions stored in your own S3.

How It Works

1. Qtap captures a request — all capture levels store an object in Garage S3, keyed by its SHA1 digest. In our config, errors get full capture (with headers), while other traffic gets summary (metadata only)

2. The HTTP transaction object is stored in Garage S3 as JSON

3. Qtap emits an artifact_record event to the OTel Collector, which includes a url field pointing to the stored object

4. The event flows through to Loki and appears in Grafana

5. You click the URL to view the complete HTTP transaction

Walkthrough

1. Find an error event in Grafana

In Explore, query for artifact records:

{service_name="qtap"} | json | event_type="artifact_record"

2. Expand the log entry

Click on an artifact record event. Look for these fields:

{
  "type": "http_transaction",
  "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c",
  "url": "http://localhost:3904/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c",
  "summary": {
    "request_method": "GET",
    "request_host": "httpbin.org",
    "response_status": 500,
    "duration_ms": 580,
    "process_exe": "/usr/bin/curl",
    "direction": "egress-external"
  }
}

3. Click the URL

The url field is a direct link to the stored object. Click it (or open it in a new tab) to see the complete HTTP transaction:

{
  "metadata": {
    "process_exe": "/usr/bin/curl",
    "container_name": "relaxed_zhukovsky",
    "container_image": "curlimages/curl",
    "bytes_sent": 72,
    "bytes_received": 127,
    "connection_id": "d654d887p3qnd27vfhag",
    "endpoint_id": "httpbin.org"
  },
  "request": {
    "method": "GET",
    "url": "https://httpbin.org/status/500",
    "scheme": "https",
    "authority": "httpbin.org",
    "protocol": "http2",
    "user_agent": "curl/8.18.0",
    "headers": {
      ":authority": "httpbin.org",
      ":method": "GET",
      ":path": "/status/500",
      "Accept": "*/*",
      "User-Agent": "curl/8.18.0"
    }
  },
  "response": {
    "status": 500,
    "content_type": "text/html; charset=utf-8",
    "headers": {
      "Content-Type": "text/html; charset=utf-8",
      "Server": "gunicorn/19.9.0"
    }
  },
  "duration_ms": 767,
  "direction": "egress-external"
}

This is the full HTTP transaction — headers and metadata — stored entirely in your infrastructure.

All capture levels generate artifact_record events and store objects in S3. The difference is content: summary objects contain only metadata (method, URL, status, duration), while headers and full objects include the complete request/response headers and bodies. The object linking walkthrough above is most useful for full captures where you can inspect the actual HTTP payload.

The access_url Template

The link between events and objects is configured in the Qtap object_stores section:

access_url: http://localhost:3904/qpoint/{{DIGEST}}

{{DIGEST}} is replaced with the SHA1 hash of the stored object. The resulting URL is embedded in every artifact_record event.

For production, replace localhost with the hostname or IP address that Grafana users can reach:

access_url: https://objects.internal.example.com/qpoint/{{DIGEST}}

Cleanup

docker compose down -v
docker rm -f qtap

Next Steps

• Storage Configuration — S3, MinIO, AWS, GCS options

• Traffic Processing with Plugins — Rulekit rules, capture levels

• Prometheus + Grafana Monitoring — Add metrics dashboards alongside logs

• OpenTelemetry Integration — Send events to any OTLP backend

• Qpoint Data Schema Reference — Full event schema documentation