Traefik Traffic Capture

This guide shows you how to use Qtap to capture HTTP traffic flowing through Traefik, a modern cloud-native reverse proxy and load balancer. You'll learn how to observe both incoming client requests and outgoing upstream connections in a dynamic, label-based configuration environment.

What You'll Learn

• Capture Traefik ingress traffic (client requests)

• Capture Traefik egress traffic (backend service requests)

• Monitor both sides of a reverse proxy simultaneously

• Use Traefik's label-based configuration with Qtap

• Leverage Traefik's automatic service discovery

• Handle dynamic backend routing

• Set up Traefik + Qtap in Docker for testing

• Deploy production-ready configurations

Use Cases

Why capture Traefik traffic?

• Dynamic Service Discovery: Monitor auto-discovered services in Docker/Kubernetes

• API Gateway Monitoring: Track all API calls through your edge proxy

• Container Traffic Visibility: See communication between microservices

• Load Balancer Analytics: Understand traffic distribution across backends

• Automatic HTTPS Inspection: See inside TLS traffic without certificate management

• Debugging Service Routing: Verify Traefik routes traffic correctly

• Performance Analysis: Measure latency at each routing hop

---

Prerequisites

• Linux system with kernel 5.10+ and eBPF support

• Docker installed (for this guide's examples)

• Root/sudo access

• Basic understanding of Traefik and Docker labels

---

Part 1: Traefik with Multiple Backends

Traefik is unique because it configures routes via Docker labels instead of config files. Let's set up Traefik with multiple backend services.

Step 1: Create Project Directory

mkdir traefik-qtap-demo
cd traefik-qtap-demo

Step 2: Create Traefik Configuration

Create traefik.yaml:

# Traefik static configuration
api:
  dashboard: true
  insecure: true  # For testing only - dashboard on :8080

entryPoints:
  web:
    address: ":80"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false  # Only expose services with traefik.enable=true

log:
  level: INFO
  format: common

Step 3: Create Backend Services

We'll create two simple backend services to demonstrate routing.

Create backend-service.py:

#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import sys
import os

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "path": self.path,
            "message": f"Hello from {service_name}!"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def do_POST(self):
        content_length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(content_length).decode() if content_length > 0 else ""

        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "method": "POST",
            "received": body,
            "message": f"POST received by {service_name}"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def log_message(self, format, *args):
        # Suppress default logging
        pass

if __name__ == '__main__':
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8000
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}")
    server.serve_forever()

Step 4: Create Qtap Configuration

Create qtap.yaml:

version: 2

# Storage Configuration
services:
  # Connection metadata (anonymized)
  event_stores:
    - type: stdout

  # HTTP request/response data (sensitive)
  object_stores:
    - type: stdout

# Processing Stack
stacks:
  traefik_capture:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|headers|full) - Capture everything
          format: text     # (json|text) - Human-readable format

# Traffic Capture Settings
tap:
  direction: all           # (egress|ingress|all) - Capture BOTH directions
  ignore_loopback: false   # (true|false) - Capture localhost (traefik uses loopback)
  audit_include_dns: false # (true|false) - Skip DNS for cleaner output

  http:
    stack: traefik_capture # Use our traefik processing stack

  # Optional: Filter out noise
  filters:
    groups:
      - qpoint             # Don't capture qtap's own traffic

Step 5: Create Docker Compose Setup

Create docker-compose.yaml:

version: '3.8'

services:
  # Traefik reverse proxy
  traefik:
    image: traefik:v3.0
    container_name: traefik-demo
    ports:
      - "8083:80"      # HTTP entrypoint
      - "8084:8080"    # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yaml:/etc/traefik/traefik.yaml:ro
    networks:
      - demo-network

  # Backend Service A
  service-a:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py", "8000"]
    container_name: service-a
    environment:
      - SERVICE_NAME=service-a
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.routers.service-a.entrypoints=web"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"
      - "traefik.http.middlewares.service-a-stripprefix.stripprefix.prefixes=/api/service-a"
      - "traefik.http.routers.service-a.middlewares=service-a-stripprefix"
    networks:
      - demo-network

  # Backend Service B
  service-b:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py", "8000"]
    container_name: service-b
    environment:
      - SERVICE_NAME=service-b
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-b.rule=PathPrefix(`/api/service-b`)"
      - "traefik.http.routers.service-b.entrypoints=web"
      - "traefik.http.services.service-b.loadbalancer.server.port=8000"
      - "traefik.http.middlewares.service-b-stripprefix.stripprefix.prefixes=/api/service-b"
      - "traefik.http.routers.service-b.middlewares=service-b-stripprefix"
    networks:
      - demo-network

  # HTTPBin mock service (internal upstream)
  httpbin:
    image: kennethreitz/httpbin
    container_name: httpbin
    expose:
      - "80"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.httpbin.rule=PathPrefix(`/api/httpbin`)"
      - "traefik.http.routers.httpbin.entrypoints=web"
      - "traefik.http.services.httpbin.loadbalancer.server.port=80"
      - "traefik.http.middlewares.httpbin-stripprefix.stripprefix.prefixes=/api/httpbin"
      - "traefik.http.routers.httpbin.middlewares=httpbin-stripprefix"
    networks:
      - demo-network

# httpbin is bundled locally so you can exercise the full flow without relying on external connectivity. Traefik forwards `/api/httpbin/*`
# to the container on port 80, and you still see the canonical httpbin responses in your curls and Qtap captures.

  # Qtap agent
  qtap:
    image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0
    container_name: qtap-traefik
    privileged: true
    user: "0:0"
    cap_add:
      - CAP_BPF
      - CAP_SYS_ADMIN
    pid: host
    network_mode: host
    volumes:
      - /sys:/sys
      - /var/run/docker.sock:/var/run/docker.sock
      - ./qtap.yaml:/app/config/qtap.yaml
    environment:
      - TINI_SUBREAPER=1
    ulimits:
      memlock: -1
    command:
      - --log-level=info
      - --log-encoding=console
      - --config=/app/config/qtap.yaml

networks:
  demo-network:
    driver: bridge

Key Traefik Concepts:

• Labels: Configure routing via Docker labels (not config files)

• Routers: Define how to match incoming requests (PathPrefix, Host, etc.)

• Services: Define backend servers (load balancer targets)

• Middlewares: Transform requests (strip prefixes, add headers, etc.)

• Automatic Discovery: Traefik watches Docker for new containers

---

Part 2: Running and Testing

Step 1: Start the Services

# Start all services
docker compose up -d

# Wait for Qtap to initialize (CRITICAL!)
sleep 6

# Check Traefik dashboard (optional)
# Open http://localhost:8084 in browser

Step 2: Generate Test Traffic

# Test 1: Route to Service A (INGRESS + EGRESS)
curl http://localhost:8083/api/service-a/

# Test 2: Route to Service B
curl http://localhost:8083/api/service-b/

# Test 3: POST to Service A
curl -X POST http://localhost:8083/api/service-a/users \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice", "role": "admin"}'

# Test 4: Route to the internal httpbin service
curl http://localhost:8083/api/httpbin/get

# Test 5: POST to httpbin via Traefik
curl -X POST http://localhost:8083/api/httpbin/post \
  -H "Content-Type: application/json" \
  -d '{"test": "data"}'

# Test 6: Generate load to see routing distribution
for i in {1..10}; do
  curl -s http://localhost:8083/api/service-a/
  curl -s http://localhost:8083/api/service-b/
done

Step 3: View Captured Traffic

# View Qtap logs
docker logs qtap-traefik

# Filter for traefik process
docker logs qtap-traefik 2>&1 | grep -A 30 "traefik"

# Count transactions
docker logs qtap-traefik 2>&1 | grep -c "HTTP Transaction"

What you should see:

=== HTTP Transaction ===
Source Process: traefik (PID: 789, Container: traefik-demo)
Direction: INGRESS ← (client to traefik)
Method: POST
URL: http://localhost:8083/api/service-a/users
Status: 200 OK
Duration: 8ms

--- Request Headers ---
Host: localhost:8083
User-Agent: curl/7.81.0
Content-Type: application/json

--- Request Body ---
{"name": "Alice", "role": "admin"}

--- Response Headers ---
Content-Type: application/json

--- Response Body ---
{"service":"service-a","method":"POST","received":"{\"name\": \"Alice\", \"role\": \"admin\"}","message":"POST received by service-a"}
========================

=== HTTP Transaction ===
Source Process: traefik (PID: 789, Container: traefik-demo)
Direction: EGRESS → (traefik to backend)
Method: POST
URL: http://service-a:8000/users
Status: 200 OK
Duration: 5ms

--- Request Headers ---
X-Forwarded-For: 172.18.0.1
X-Forwarded-Proto: http

--- Request Body ---
{"name": "Alice", "role": "admin"}
========================

Key indicators:

• ✅ "exe" contains traefik - Process identified

• ✅ Direction: INGRESS - Client → Traefik

• ✅ Direction: EGRESS - Traefik → Backend service

• ✅ Two transactions per proxied request

• ✅ Path transformation visible (prefix stripped)

• ✅ Headers added by Traefik (X-Forwarded-*)

---

Part 3: Advanced Configurations

Configuration 1: Capture Only Specific Services

Use Rulekit to capture only traffic to specific backend services:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: is_service_a
      expr: http.req.path matches /^\/api\/service-a\//
    - name: is_external
      expr: http.req.path matches /^\/api\/httpbin\//
    - name: is_error
      expr: http.res.status >= 400

stacks:
  selective_capture:
    plugins:
      - type: http_capture
        config:
          level: none        # Don't capture by default
          format: json
          rules:
            # Capture all traffic to service-a
            - name: "Service A traffic"
              expr: is_service_a()
              level: full

            # Capture only errors from external services
            - name: "External errors"
              expr: is_external() && is_error()
              level: full

            # Capture slow requests anywhere
            - name: "Slow requests"
              expr: http.res.duration_ms > 1000
              level: headers  # Headers only

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: selective_capture

Configuration 2: Monitor Service Discovery

Capture traffic as Traefik discovers and routes to new services:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: has_service_header
      expr: http.req.headers.x-service-name != ""

stacks:
  discovery_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Just metadata for service analytics
          format: json

tap:
  direction: egress          # Focus on traefik→backend traffic
  ignore_loopback: false
  http:
    stack: discovery_monitoring

This captures metadata about which backends Traefik routes to, useful for understanding service discovery behavior.

Configuration 3: API Gateway with Rate Limiting Detection

Monitor API gateway patterns and detect potential rate limiting:

version: 2

services:
  event_stores:
    - type: stdout
  # Sensitive data to S3 (never leaves your infrastructure)
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      region: us-east-1
      bucket: my-company-Traefik-traffic
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY
      insecure: false

rulekit:
  macros:
    - name: is_rate_limited
      expr: http.res.status == 429
    - name: is_auth_failure
      expr: http.res.status == 401 || http.res.status == 403
    - name: is_api_path
      expr: http.req.path matches /^\/api\//

stacks:
  api_gateway:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture rate limiting events
            - name: "Rate limited"
              expr: is_rate_limited()
              level: full

            # Capture authentication failures
            - name: "Auth failures"
              expr: is_auth_failure()
              level: full

            # Capture all API errors
            - name: "API errors"
              expr: is_api_path() && http.res.status >= 400
              level: headers

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: api_gateway

Configuration 4: Production Setup with S3

version: 2

services:
  event_stores:
    - type: stdout

  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      region: us-east-1
      bucket: my-company-traefik-traffic
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY
      insecure: false

stacks:
  production_capture:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Only capture errors
            - name: "Production errors"
              expr: http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: production_capture

---

Part 4: Real-World Use Cases

Use Case 1: Microservices Mesh Monitoring

Monitor all service-to-service communication through Traefik:

docker-compose.yaml (add more services):

  service-c:
    build: ...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-c.rule=PathPrefix(`/api/service-c`)"
      # ... more labels

  service-d:
    build: ...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-d.rule=PathPrefix(`/api/service-d`)"
      # ... more labels

qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      region: us-east-1
      bucket: my-company-Traefik-traffic
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY
      insecure: false

rulekit:
  macros:
    - name: is_internal_service
      expr: http.req.path matches /^\/api\/service-[a-z]\//

stacks:
  mesh_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Metadata for analytics
          format: json
          rules:
            # Capture all internal service calls
            - name: "Service mesh traffic"
              expr: is_internal_service()
              level: summary

            # But capture errors in full
            - name: "Service mesh errors"
              expr: is_internal_service() && http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: mesh_monitoring

Use Case 2: Canary Deployment Monitoring

Monitor traffic distribution during canary deployments:

docker-compose.yaml:

  service-a-v1:
    # ... existing service-a config
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"
      - "traefik.http.services.service-a.loadbalancer.sticky.cookie=true"

  service-a-v2:
    # ... same as v1 but with SERVICE_NAME=service-a-v2
    environment:
      - SERVICE_NAME=service-a-v2
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"

Traefik will load balance between v1 and v2. Qtap captures which version served each request.

qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  canary_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Capture metadata to see distribution
          format: json

tap:
  direction: egress          # Focus on traefik→backend to see which version
  ignore_loopback: false
  http:
    stack: canary_monitoring

Analyze logs to see v1 vs v2 traffic distribution.

Use Case 3: Multi-Tenant API Gateway

Route different tenants to different backends:

docker-compose.yaml:

  tenant-a-backend:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.tenant-a.rule=Host(`tenant-a.example.com`)"

  tenant-b-backend:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.tenant-b.rule=Host(`tenant-b.example.com`)"

qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: tenant_a
      expr: http.req.headers.host matches /^tenant-a\./
    - name: tenant_b
      expr: http.req.headers.host matches /^tenant-b\./

stacks:
  multi_tenant:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture all tenant A traffic for audit
            - name: "Tenant A traffic"
              expr: tenant_a()
              level: full

            # Capture only errors for tenant B
            - name: "Tenant B errors"
              expr: tenant_b() && http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: multi_tenant

---

Understanding Traefik + Qtap

Dual Capture for Dynamic Routing

When Traefik routes a request, Qtap captures two transactions:

Transaction 1: INGRESS (Client → Traefik)

Source Process: traefik
Direction: INGRESS ←
URL: http://localhost:8083/api/service-a/users

Transaction 2: EGRESS (Traefik → Backend)

Source Process: traefik
Direction: EGRESS →
URL: http://service-a:8000/users

Notice:

• Path transformation: /api/service-a/users → /users (middleware stripped prefix)

• Container resolution: service-a:8000 (Docker DNS)

• Headers added: X-Forwarded-*

Traefik-Specific Features

Process Identification:

• Look for exe containing traefik

• Typically /usr/local/bin/traefik

Label-Based Configuration:

• Unlike NGINX/Caddy, routing is defined via container labels

• Qtap sees the result of routing decisions

• Changes to labels automatically discovered (no restart needed)

Automatic Service Discovery:

• Traefik watches Docker events

• New containers auto-routed

• Qtap captures new service traffic immediately

---

Troubleshooting

Not Seeing Traefik Traffic?

Check 1: Is Traefik routing correctly?

# Check Traefik dashboard
curl http://localhost:8084/api/rawdata

# Or check Traefik logs
docker logs traefik-demo

Check 2: Are services registered with Traefik?

# Verify labels are correct
docker inspect service-a | grep -A 10 Labels

Check 3: Is Qtap running before requests?

docker logs qtap-traefik | head -20

Check 4: Is ignore_loopback correct?

tap:
  ignore_loopback: false  # MUST be false for Docker networking

Seeing "l7Protocol": "other"?

• Wait longer after starting Qtap (6+ seconds)

• Check if Traefik is using HTTP/3 (not yet supported by Qtap)

• Verify traffic is actually HTTP

Labels Not Working?

Common label mistakes:

# ❌ Wrong - missing quotes
- traefik.http.routers.myapp.rule=PathPrefix(/api)

# ✅ Correct - quoted value
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

# ❌ Wrong - missing enable
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

# ✅ Correct - enable=true required
- "traefik.enable=true"
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

Too Much Traffic?

Apply conditional capture:

stacks:
  reduced:
    plugins:
      - type: http_capture
        config:
          level: none
          rules:
            - name: "Errors only"
              expr: http.res.status >= 400
              level: full

---

Performance Considerations

Traefik + Qtap Performance

• CPU: ~1-3% overhead for typical traffic

• Memory: ~50-200MB for Qtap

• Latency: Zero additional latency (passive observation)

Best practices for high-traffic Traefik:

1. Use level: summary for high volume

2. Apply rules to capture selectively

3. Filter health checks and monitoring endpoints

4. Send to S3 with batching (Fluent Bit)

5. Set TTL policies on storage

Scaling Recommendations

Traffic Volume	Recommended Level	Notes
< 100 req/sec	full	Capture everything
100-1000 req/sec	details	Headers only
1000-10000 req/sec	summary	Metadata only
> 10000 req/sec	conditional	Errors/slow requests only

---

Traefik vs NGINX/Caddy

Configuration:

• Traefik: Docker labels, dynamic discovery

• NGINX: Static config files

• Caddy: Static config files (but simpler)

Use Cases:

• Traefik: Containerized apps, Kubernetes, dynamic environments

• NGINX: Traditional deployments, high performance

• Caddy: Simplicity, automatic HTTPS

Qtap Compatibility:

• All three work perfectly with Qtap

• Traefik's dynamic routing is fully observable

• Same capture quality across all proxies

---

Next Steps

Learn More About Qtap:

• Traffic Capture Settings - Complete configuration

• Traffic Processing with Plugins - All plugin options

• Complete Guide - Progressive tutorial

Production Deployment:

• Storage Configuration - S3 setup guide

• Capturing All HTTP Traffic with Fluent Bit - Batching for scale

• Kubernetes Manifest - Deploy in K8s

Related Guides:

• Capturing NGINX Traffic - Traditional reverse proxy

• Capturing Caddy Traffic - Modern web server

• Ingress Traffic Capture with Python - Application servers

Alternative: Cloud Management:

• Qplane - Manage Qtap with visual dashboards

---

Cleanup

# Stop all services
docker compose down

# Remove images
docker compose down --rmi local

# Clean up files
rm backend-service.py traefik.yaml qtap.yaml docker-compose.yaml

---

This guide uses validated configurations. All examples are tested and guaranteed to work with Traefik and Qtap.