Self-Hosted Grafana Observability Stack

This guide walks you through setting up a complete, self-hosted observability stack where Qtap events flow to Grafana via Loki and full HTTP payloads stay in your own S3-compatible object storage. Every byte of captured data — metadata and payloads — remains inside your infrastructure.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        YOUR INFRASTRUCTURE                       │
│                                                                  │
│   ┌───────┐                                                      │
│   │ Qtap  │──── Events (metadata) ───► OTel Collector (:4317)    │
│   │ Agent │                               │                      │
│   │       │                               ▼                      │
│   │       │                           Loki (:3100)               │
│   │       │                               │                      │
│   │       │                               ▼                      │
│   │       │                           Grafana (:3000)            │
│   │       │                               ▲                      │
│   │       │                               │ click artifact URL   │
│   │       │                               ▼                      │
│   │       │                       nginx proxy (:3904)            │
│   │       │                               │                      │
│   │       │                               ▼                      │
│   │       │── Objects (payloads) ──► Garage S3 (:3900)           │
│   └───────┘                                                      │
└──────────────────────────────────────────────────────────────────┘

Service

Role

Port

Qtap

eBPF agent — captures HTTP traffic at kernel level

Host network

OTel Collector

Receives OTLP logs from Qtap, forwards to Loki

4317 (gRPC)

Loki

Log aggregation and storage

3100

Grafana

Query, explore, and visualize events

3000

Garage

S3-compatible object storage for HTTP payloads

3900 (S3 API)

nginx

Proxy for anonymous read access to stored objects

3904

This guide covers events and object linking — not traces. For network timing traces, see the OpenTelemetry Integration guide.

Prerequisites

Docker and Docker Compose
Linux kernel 5.10+ with eBPF support
aws CLI (for S3 verification) — install guide

Understanding the Two Data Paths

Qtap produces two distinct outputs. Understanding the split is key to this architecture:

Events

Objects

What

Lightweight metadata — method, URL, status, duration, process

HTTP transaction objects — metadata at summary level; headers and bodies at full level

Sensitivity

Low — safe to send anywhere

Varies — summary objects contain only metadata; full objects may contain API keys, tokens, PII

Storage

Loki (via OTel Collector)

Garage S3 (your infrastructure)

Volume

Every observed request

Every captured request (content varies by capture level)

The link between them: Qtap's access_url template embeds a clickable URL into each artifact_record event. When you find an interesting event in Grafana, you click the URL to fetch the complete HTTP transaction from your S3 storage.

access_url: http://localhost:3904/qpoint/{{DIGEST}}
                                         ^^^^^^^^
                                         Replaced with SHA1 hash of the stored object

Configuration Files

Create a project directory and add these files:

mkdir -p grafana-stack/grafana/provisioning/datasources
cd grafana-stack

OTel Collector

cat > otel-collector.yaml << 'EOF'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp/loki:
    endpoint: http://localhost:3100/otlp
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]
EOF

The collector listens for gRPC on port 4317 (where Qtap sends events) and forwards them to Loki's OTLP endpoint. Since the collector runs with network_mode: host, it reaches Loki at localhost:3100.

Loki

cat > loki.yaml << 'EOF'
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h
  allow_structured_metadata: true
  max_query_series: 100000

compactor:
  working_directory: /loki/compactor
  delete_request_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
EOF

Key settings: allow_structured_metadata: true lets Loki store Qtap's structured attributes (method, status, host, etc.) as queryable fields. Retention is set to 7 days (168h).

Garage (S3-Compatible Object Storage)

cat > garage.toml << 'EOF'
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "c052485a056fabf3c0832d98f63b14d58036f5189683fc39da199f43fde3f15e"

[s3_api]
s3_region = "us-east-1"
api_bind_addr = "[::]:3900"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage.localhost"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "demo-admin-token-for-local-use-only"
EOF

Garage provides the S3 API on port 3900 (where Qtap writes payloads), a web endpoint on 3902 (for anonymous reads), and an admin API on 3903 (for bucket management).

Nginx Proxy

cat > nginx.conf << 'EOF'
server {
    listen 3904;

    location /qpoint/ {
        rewrite ^/qpoint/(.*)$ /$1 break;
        proxy_pass http://garage:3902;
        proxy_set_header Host qpoint.web.garage.localhost;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
EOF

The nginx proxy translates requests like http://localhost:3904/qpoint/<DIGEST> into Garage web requests with the correct virtual-host Host header. This avoids needing wildcard DNS for Garage's subdomain-based routing.

Grafana Datasource

cat > grafana/provisioning/datasources/datasources.yaml << 'EOF'
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
    isDefault: true
    editable: true
EOF

Qtap

cat > qtap.yaml << 'EOF'
version: 2

services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"      # OTel Collector gRPC
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

  object_stores:
    - id: garage
      type: s3
      endpoint: localhost:3900        # Garage S3 API
      bucket: qpoint
      region: us-east-1
      access_url: http://localhost:3904/qpoint/{{DIGEST}}
      insecure: true
      access_key:
        type: env
        value: GARAGE_ACCESS_KEY
      secret_key:
        type: env
        value: GARAGE_SECRET_KEY

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400 && http.res.status < 600

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary             # (none|summary|headers|full)
          format: json               # (json|text)
          rules:
            - name: "Full capture on errors"
              expr: is_error()
              level: full            # Stores headers + bodies in S3

tap:
  direction: egress                  # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true              # (true|false)
  audit_include_dns: false           # (true|false)
  http:
    stack: default_stack
  filters:
    groups:
      - qpoint                       # Don't capture Qtap's own traffic
EOF

This configuration captures all egress HTTP traffic at summary level (metadata only — no headers or bodies), and automatically escalates to full capture for any 4xx or 5xx response. Full captures store complete request/response headers in Garage S3. Both levels emit artifact_record events with clickable URLs pointing to the stored objects.

Docker Compose

cat > docker-compose.yaml << 'EOF'
services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: always
    ports:
      - "3100:3100"
    volumes:
      - ./loki.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    restart: always
    network_mode: host
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector-config.yaml:ro
    command: ["--config=/etc/otel-collector-config.yaml"]

  garage:
    image: dxflrs/garage:v2.1.0
    container_name: garage
    restart: always
    ports:
      - "3900:3900"
      - "3902:3902"
      - "3903:3903"
    volumes:
      - ./garage.toml:/etc/garage.toml:ro

  garage-web:
    image: nginx:alpine
    container_name: garage-web
    restart: always
    ports:
      - "3904:3904"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - garage

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Editor

volumes:
  loki-data:
  grafana-data:
EOF

The OTel Collector uses network_mode: host so that Qtap (running on the host) can reach it at localhost:4317. Because it shares the host network stack, it also reaches Loki at localhost:3100.

Running the Stack

Step 1: Start the Services

docker compose up -d

Verify all containers are running:

docker compose ps

Step 2: Initialize Garage

Wait for Garage to be ready, then configure the cluster layout, create a bucket, and set up access credentials:

# Wait for Garage to start
sleep 5

# Get the node ID
NODE_ID=$(docker exec garage /garage status 2>/dev/null | grep -oE '[a-f0-9]{16}' | head -1)
echo "Node ID: $NODE_ID"

# Configure cluster layout
docker exec garage /garage layout assign -z dc1 -c 1G "$NODE_ID"
docker exec garage /garage layout apply --version 1

# Create the bucket
docker exec garage /garage bucket create qpoint

# Import S3 credentials
docker exec garage /garage key import \
  --yes \
  -n qtap-key \
  "GK0123456789abcdef01234567" \
  "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"

# Grant read/write access
docker exec garage /garage bucket allow \
  --read --write --owner qpoint --key qtap-key

# Enable anonymous web reads (for the nginx proxy)
docker exec garage /garage bucket website --allow qpoint

The credentials above are for local development only. For production, generate unique keys and manage them securely.

Step 3: Start Qtap

docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  -e GARAGE_ACCESS_KEY=GK0123456789abcdef01234567 \
  -e GARAGE_SECRET_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Step 4: Wait for Initialization

sleep 6

Qtap needs a few seconds to load eBPF programs and start capturing.

Step 5: Generate Test Traffic

Generate both a successful request and an error to see both data paths:

# Successful request — captured at summary level (metadata only, no headers/bodies)
docker run --rm curlimages/curl -s https://httpbin.org/get > /dev/null

# Error request — captured at full level (headers + bodies stored in S3)
docker run --rm curlimages/curl -s https://httpbin.org/status/500 > /dev/null

Step 6: Verify Data Is Flowing

Check OTel Collector is receiving data:

docker logs otel-collector 2>&1 | tail -5

You should see log export messages.

Check Loki has events:

curl -s "http://localhost:3100/loki/api/v1/query_range?query=%7Bservice_name%3D%22qtap%22%7D&limit=5" | python3 -m json.tool | head -20

Check S3 has objects (from the error request):

AWS_ACCESS_KEY_ID=GK0123456789abcdef01234567 \
AWS_SECRET_ACCESS_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
aws --endpoint-url http://localhost:3900 \
  --region us-east-1 \
  s3 ls s3://qpoint/

Exploring Data in Grafana

Open http://localhost:3000 in your browser (default credentials: admin / admin).

Viewing Events

Navigate to Explore (compass icon in the left sidebar)
Select Loki as the datasource
Enter a LogQL query:

{service_name="qtap"}

Click Run query

You should see Qtap events — both connection events (TCP connection metadata) and artifact_record events (HTTP transaction summaries, with full headers and bodies for requests matching capture rules).

Useful LogQL Queries

Query

Description

{service_name="qtap"}

All Qtap events

{service_name="qtap"} | json | event_type="artifact_record"

Only artifact records (objects stored in S3)

{service_name="qtap"} | json | response_status >= 400

Error responses

{service_name="qtap"} | json | request_host="httpbin.org"

Traffic to a specific host

{service_name="qtap"} | json | duration_ms > 1000

Slow requests (> 1s)

{service_name="qtap"} | json | process_exe="/usr/bin/curl"

Requests from curl

{service_name="qtap"} | json | direction="egress-external"

External egress traffic

Expanding Log Entries

Click on any log entry to expand it. You'll see structured attributes including:

request_method, request_host, request_scheme
response_status, duration_ms
process_exe, direction
For artifact records: digest, url, type

Object Linking — From Events to Full Payloads

This is the key capability of this stack: linking lightweight events in Grafana to complete HTTP transactions stored in your own S3.

How It Works

Qtap captures a request — all capture levels store an object in Garage S3, keyed by its SHA1 digest. In our config, errors get full capture (with headers), while other traffic gets summary (metadata only)
The HTTP transaction object is stored in Garage S3 as JSON
Qtap emits an artifact_record event to the OTel Collector, which includes a url field pointing to the stored object
The event flows through to Loki and appears in Grafana
You click the URL to view the complete HTTP transaction

Walkthrough

1. Find an error event in Grafana

In Explore, query for artifact records:

{service_name="qtap"} | json | event_type="artifact_record"

2. Expand the log entry

Click on an artifact record event. Look for these fields:

{
  "type": "http_transaction",
  "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c",
  "url": "http://localhost:3904/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c",
  "summary": {
    "request_method": "GET",
    "request_host": "httpbin.org",
    "response_status": 500,
    "duration_ms": 580,
    "process_exe": "/usr/bin/curl",
    "direction": "egress-external"
  }
}

3. Click the URL

The url field is a direct link to the stored object. Click it (or open it in a new tab) to see the complete HTTP transaction:

{
  "metadata": {
    "process_exe": "/usr/bin/curl",
    "container_name": "relaxed_zhukovsky",
    "container_image": "curlimages/curl",
    "bytes_sent": 72,
    "bytes_received": 127,
    "connection_id": "d654d887p3qnd27vfhag",
    "endpoint_id": "httpbin.org"
  },
  "request": {
    "method": "GET",
    "url": "https://httpbin.org/status/500",
    "scheme": "https",
    "authority": "httpbin.org",
    "protocol": "http2",
    "user_agent": "curl/8.18.0",
    "headers": {
      ":authority": "httpbin.org",
      ":method": "GET",
      ":path": "/status/500",
      "Accept": "*/*",
      "User-Agent": "curl/8.18.0"
    }
  },
  "response": {
    "status": 500,
    "content_type": "text/html; charset=utf-8",
    "headers": {
      "Content-Type": "text/html; charset=utf-8",
      "Server": "gunicorn/19.9.0"
    }
  },
  "duration_ms": 767,
  "direction": "egress-external"
}

This is the full HTTP transaction — headers and metadata — stored entirely in your infrastructure.

All capture levels generate artifact_record events and store objects in S3. The difference is content: summary objects contain only metadata (method, URL, status, duration), while headers and full objects include the complete request/response headers and bodies. The object linking walkthrough above is most useful for full captures where you can inspect the actual HTTP payload.

The `access_url` Template

The link between events and objects is configured in the Qtap object_stores section:

access_url: http://localhost:3904/qpoint/{{DIGEST}}

{{DIGEST}} is replaced with the SHA1 hash of the stored object. The resulting URL is embedded in every artifact_record event.

For production, replace localhost with the hostname or IP address that Grafana users can reach:

access_url: https://objects.internal.example.com/qpoint/{{DIGEST}}

Cleanup

docker compose down -v
docker rm -f qtap

Next Steps

Storage Configuration — S3, MinIO, AWS, GCS options
Traffic Processing with Plugins — Rulekit rules, capture levels
Prometheus + Grafana Monitoring — Add metrics dashboards alongside logs
OpenTelemetry Integration — Add traces for network timing
Qpoint Data Schema Reference — Full event schema documentation

PreviousOpenTelemetry Integration NextIntegrating Qtap with Your Observability Stack

Last updated 23 days ago

hashtagArchitecture

hashtagPrerequisites

hashtagUnderstanding the Two Data Paths

hashtagConfiguration Files

hashtagOTel Collector

hashtagLoki

hashtagGarage (S3-Compatible Object Storage)

hashtagNginx Proxy

hashtagGrafana Datasource

hashtagQtap

hashtagDocker Compose

hashtagRunning the Stack

hashtagStep 1: Start the Services

hashtagStep 2: Initialize Garage

hashtagStep 3: Start Qtap

hashtagStep 4: Wait for Initialization

hashtagStep 5: Generate Test Traffic

hashtagStep 6: Verify Data Is Flowing

hashtagExploring Data in Grafana

hashtagViewing Events

hashtagUseful LogQL Queries

hashtagExpanding Log Entries

hashtagObject Linking — From Events to Full Payloads

hashtagHow It Works

hashtagWalkthrough

hashtagThe access_url Template

hashtagCleanup

hashtagNext Steps

Architecture

Prerequisites

Understanding the Two Data Paths

Configuration Files

OTel Collector

Loki

Garage (S3-Compatible Object Storage)

Nginx Proxy

Grafana Datasource

Qtap

Docker Compose

Running the Stack

Step 1: Start the Services

Step 2: Initialize Garage

Step 3: Start Qtap

Step 4: Wait for Initialization

Step 5: Generate Test Traffic

Step 6: Verify Data Is Flowing

Exploring Data in Grafana

Viewing Events

Useful LogQL Queries

Expanding Log Entries

Object Linking — From Events to Full Payloads

How It Works

Walkthrough

The `access_url` Template

Cleanup

Next Steps