Capturing Traefik Traffic

This guide shows you how to use Qtap to capture HTTP traffic flowing through Traefik, a modern cloud-native reverse proxy and load balancer. You'll learn how to observe both incoming client requests and outgoing upstream connections in a dynamic, label-based configuration environment.

What You'll Learn

Capture Traefik ingress traffic (client requests)
Capture Traefik egress traffic (backend service requests)
Monitor both sides of a reverse proxy simultaneously
Use Traefik's label-based configuration with Qtap
Leverage Traefik's automatic service discovery
Handle dynamic backend routing
Set up Traefik + Qtap in Docker for testing
Deploy production-ready configurations

Use Cases

Why capture Traefik traffic?

Dynamic Service Discovery: Monitor auto-discovered services in Docker/Kubernetes
API Gateway Monitoring: Track all API calls through your edge proxy
Container Traffic Visibility: See communication between microservices
Load Balancer Analytics: Understand traffic distribution across backends
Automatic HTTPS Inspection: See inside TLS traffic without certificate management
Debugging Service Routing: Verify Traefik routes traffic correctly
Performance Analysis: Measure latency at each routing hop

Prerequisites

Linux system with kernel 5.10+ and eBPF support
Docker installed (for this guide's examples)
Root/sudo access
Basic understanding of Traefik and Docker labels

Part 1: Traefik with Multiple Backends

Traefik is unique because it configures routes via Docker labels instead of config files. Let's set up Traefik with multiple backend services.

Step 1: Create Project Directory

mkdir traefik-qtap-demo
cd traefik-qtap-demo

Step 2: Create Traefik Configuration

Create traefik.yaml:

# Traefik static configuration
api:
  dashboard: true
  insecure: true  # For testing only - dashboard on :8080

entryPoints:
  web:
    address: ":80"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false  # Only expose services with traefik.enable=true

log:
  level: INFO
  format: common

Step 3: Create Backend Services

We'll create two simple backend services to demonstrate routing.

Create backend-service.py:

#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import sys
import os

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "path": self.path,
            "message": f"Hello from {service_name}!"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def do_POST(self):
        content_length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(content_length).decode() if content_length > 0 else ""

        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "method": "POST",
            "received": body,
            "message": f"POST received by {service_name}"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def log_message(self, format, *args):
        # Suppress default logging
        pass

if __name__ == '__main__':
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8000
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}")
    server.serve_forever()

Step 4: Create Qtap Configuration

Create qtap.yaml:

version: 2

# Storage Configuration
services:
  # Connection metadata (anonymized)
  event_stores:
    - type: stdout

  # HTTP request/response data (sensitive)
  object_stores:
    - type: stdout

# Processing Stack
stacks:
  traefik_capture:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|details|full) - Capture everything
          format: text     # (json|text) - Human-readable format

# Traffic Capture Settings
tap:
  direction: all           # (egress|ingress|all) - Capture BOTH directions
  ignore_loopback: false   # (true|false) - Capture localhost (traefik uses loopback)
  audit_include_dns: false # (true|false) - Skip DNS for cleaner output

  http:
    stack: traefik_capture # Use our traefik processing stack

  # Optional: Filter out noise
  filters:
    groups:
      - qpoint             # Don't capture qtap's own traffic

Step 5: Create Docker Compose Setup

Create docker-compose.yaml:

version: '3.8'

services:
  # Traefik reverse proxy
  traefik:
    image: traefik:v3.0
    container_name: traefik-demo
    ports:
      - "8083:80"      # HTTP entrypoint
      - "8084:8080"    # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
  - ./traefik.yaml:/etc/traefik/traefik.yaml:ro
  networks:
    - demo-network

  # Backend Service A
  service-a:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py", "8000"]
    container_name: service-a
    environment:
      - SERVICE_NAME=service-a
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.routers.service-a.entrypoints=web"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"
      - "traefik.http.middlewares.service-a-stripprefix.stripprefix.prefixes=/api/service-a"
      - "traefik.http.routers.service-a.middlewares=service-a-stripprefix"
    networks:
      - demo-network

  # Backend Service B
  service-b:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py", "8000"]
    container_name: service-b
    environment:
      - SERVICE_NAME=service-b
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-b.rule=PathPrefix(`/api/service-b`)"
      - "traefik.http.routers.service-b.entrypoints=web"
      - "traefik.http.services.service-b.loadbalancer.server.port=8000"
      - "traefik.http.middlewares.service-b-stripprefix.stripprefix.prefixes=/api/service-b"
      - "traefik.http.routers.service-b.middlewares=service-b-stripprefix"
    networks:
      - demo-network

  # HTTPBin mock service (internal upstream)
  httpbin:
    image: kennethreitz/httpbin
    container_name: httpbin
    expose:
      - "80"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.httpbin.rule=PathPrefix(`/api/httpbin`)"
      - "traefik.http.routers.httpbin.entrypoints=web"
      - "traefik.http.services.httpbin.loadbalancer.server.port=80"
      - "traefik.http.middlewares.httpbin-stripprefix.stripprefix.prefixes=/api/httpbin"
      - "traefik.http.routers.httpbin.middlewares=httpbin-stripprefix"
    networks:
      - demo-network

# httpbin is bundled locally so you can exercise the full flow without relying on external connectivity. Traefik forwards `/api/httpbin/*`
# to the container on port 80, and you still see the canonical httpbin responses in your curls and Qtap captures.

  # Qtap agent
  qtap:
    image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0
    container_name: qtap-traefik
    privileged: true
    user: "0:0"
    cap_add:
      - CAP_BPF
      - CAP_SYS_ADMIN
    pid: host
    network_mode: host
    volumes:
      - /sys:/sys
      - /var/run/docker.sock:/var/run/docker.sock
      - ./qtap.yaml:/app/config/qtap.yaml
    environment:
      - TINI_SUBREAPER=1
    ulimits:
      memlock: -1
    command:
      - --log-level=warn
      - --log-encoding=console
      - --config=/app/config/qtap.yaml

networks:
  demo-network:
    driver: bridge

Key Traefik Concepts:

Labels: Configure routing via Docker labels (not config files)
Routers: Define how to match incoming requests (PathPrefix, Host, etc.)
Services: Define backend servers (load balancer targets)
Middlewares: Transform requests (strip prefixes, add headers, etc.)
Automatic Discovery: Traefik watches Docker for new containers

Part 2: Running and Testing

Step 1: Start the Services

# Start all services
docker compose up -d

# Wait for Qtap to initialize (CRITICAL!)
sleep 6

# Check Traefik dashboard (optional)
# Open http://localhost:8084 in browser

Step 2: Generate Test Traffic

# Test 1: Route to Service A (INGRESS + EGRESS)
curl http://localhost:8083/api/service-a/

# Test 2: Route to Service B
curl http://localhost:8083/api/service-b/

# Test 3: POST to Service A
curl -X POST http://localhost:8083/api/service-a/users \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice", "role": "admin"}'

# Test 4: Route to the internal httpbin service
curl http://localhost:8083/api/httpbin/get

# Test 5: POST to httpbin via Traefik
curl -X POST http://localhost:8083/api/httpbin/post \
  -H "Content-Type: application/json" \
  -d '{"test": "data"}'

# Test 6: Generate load to see routing distribution
for i in {1..10}; do
  curl -s http://localhost:8083/api/service-a/
  curl -s http://localhost:8083/api/service-b/
done

Step 3: View Captured Traffic

# View Qtap logs
docker logs qtap-traefik

# Filter for traefik process
docker logs qtap-traefik 2>&1 | grep -A 30 "traefik"

# Count transactions
docker logs qtap-traefik 2>&1 | grep -c "HTTP Transaction"

What you should see:

=== HTTP Transaction ===
Source Process: traefik (PID: 789, Container: traefik-demo)
Direction: INGRESS ← (client to traefik)
Method: POST
URL: http://localhost:8083/api/service-a/users
Status: 200 OK
Duration: 8ms

--- Request Headers ---
Host: localhost:8083
User-Agent: curl/7.81.0
Content-Type: application/json

--- Request Body ---
{"name": "Alice", "role": "admin"}

--- Response Headers ---
Content-Type: application/json

--- Response Body ---
{"service":"service-a","method":"POST","received":"{\"name\": \"Alice\", \"role\": \"admin\"}","message":"POST received by service-a"}
========================

=== HTTP Transaction ===
Source Process: traefik (PID: 789, Container: traefik-demo)
Direction: EGRESS → (traefik to backend)
Method: POST
URL: http://service-a:8000/users
Status: 200 OK
Duration: 5ms

--- Request Headers ---
X-Forwarded-For: 172.18.0.1
X-Forwarded-Proto: http

--- Request Body ---
{"name": "Alice", "role": "admin"}
========================

Key indicators:

✅ "exe" contains traefik - Process identified
✅ Direction: INGRESS - Client → Traefik
✅ Direction: EGRESS - Traefik → Backend service
✅ Two transactions per proxied request
✅ Path transformation visible (prefix stripped)
✅ Headers added by Traefik (X-Forwarded-*)

Part 3: Advanced Configurations

Configuration 1: Capture Only Specific Services

Use Rulekit to capture only traffic to specific backend services:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: is_service_a
      expr: http.req.path matches /^\/api\/service-a\//
    - name: is_external
      expr: http.req.path matches /^\/api\/httpbin\//
    - name: is_error
      expr: http.res.status >= 400

stacks:
  selective_capture:
    plugins:
      - type: http_capture
        config:
          level: none        # Don't capture by default
          format: json
          rules:
            # Capture all traffic to service-a
            - name: "Service A traffic"
              expr: is_service_a()
              level: full

            # Capture only errors from external services
            - name: "External errors"
              expr: is_external() && is_error()
              level: full

            # Capture slow requests anywhere
            - name: "Slow requests"
              expr: http.res.duration_ms > 1000
              level: details  # Headers only

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: selective_capture

Configuration 2: Monitor Service Discovery

Capture traffic as Traefik discovers and routes to new services:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: has_service_header
      expr: http.req.header.x-service-name != ""

stacks:
  discovery_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Just metadata for service analytics
          format: json

tap:
  direction: egress          # Focus on traefik→backend traffic
  ignore_loopback: false
  http:
    stack: discovery_monitoring

This captures metadata about which backends Traefik routes to, useful for understanding service discovery behavior.

Configuration 3: API Gateway with Rate Limiting Detection

Monitor API gateway patterns and detect potential rate limiting:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: s3
      config:
        bucket: api-gateway-audit
        # S3 config...

rulekit:
  macros:
    - name: is_rate_limited
      expr: http.res.status == 429
    - name: is_auth_failure
      expr: http.res.status == 401 || http.res.status == 403
    - name: is_api_path
      expr: http.req.path matches /^\/api\//

stacks:
  api_gateway:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture rate limiting events
            - name: "Rate limited"
              expr: is_rate_limited()
              level: full

            # Capture authentication failures
            - name: "Auth failures"
              expr: is_auth_failure()
              level: full

            # Capture all API errors
            - name: "API errors"
              expr: is_api_path() && http.res.status >= 400
              level: details

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: api_gateway

Configuration 4: Production Setup with S3

version: 2

services:
  event_stores:
    - type: stdout

  object_stores:
    - type: s3
      config:
        endpoint: https://s3.amazonaws.com
        region: us-east-1
        bucket: my-company-traefik-traffic
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

stacks:
  production_capture:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Only capture errors
            - name: "Production errors"
              expr: http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: production_capture

Part 4: Real-World Use Cases

Use Case 1: Microservices Mesh Monitoring

Monitor all service-to-service communication through Traefik:

docker-compose.yaml (add more services):

  service-c:
    build: ...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-c.rule=PathPrefix(`/api/service-c`)"
      # ... more labels

  service-d:
    build: ...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-d.rule=PathPrefix(`/api/service-d`)"
      # ... more labels

qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: s3
      config:
        bucket: microservices-traffic

rulekit:
  macros:
    - name: is_internal_service
      expr: http.req.path matches /^\/api\/service-[a-z]\//

stacks:
  mesh_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Metadata for analytics
          format: json
          rules:
            # Capture all internal service calls
            - name: "Service mesh traffic"
              expr: is_internal_service()
              level: summary

            # But capture errors in full
            - name: "Service mesh errors"
              expr: is_internal_service() && http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: mesh_monitoring

Use Case 2: Canary Deployment Monitoring

Monitor traffic distribution during canary deployments:

docker-compose.yaml:

  service-a-v1:
    # ... existing service-a config
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"
      - "traefik.http.services.service-a.loadbalancer.sticky.cookie=true"

  service-a-v2:
    # ... same as v1 but with SERVICE_NAME=service-a-v2
    environment:
      - SERVICE_NAME=service-a-v2
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)"
      - "traefik.http.services.service-a.loadbalancer.server.port=8000"

Traefik will load balance between v1 and v2. Qtap captures which version served each request.

qtap.yaml:

stacks:
  canary_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Capture metadata to see distribution
          format: json

tap:
  direction: egress          # Focus on traefik→backend to see which version
  ignore_loopback: false
  http:
    stack: canary_monitoring

Analyze logs to see v1 vs v2 traffic distribution.

Use Case 3: Multi-Tenant API Gateway

Route different tenants to different backends:

docker-compose.yaml:

  tenant-a-backend:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.tenant-a.rule=Host(`tenant-a.example.com`)"

  tenant-b-backend:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.tenant-b.rule=Host(`tenant-b.example.com`)"

qtap.yaml:

rulekit:
  macros:
    - name: tenant_a
      expr: http.req.header.host matches /^tenant-a\./
    - name: tenant_b
      expr: http.req.header.host matches /^tenant-b\./

stacks:
  multi_tenant:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture all tenant A traffic for audit
            - name: "Tenant A traffic"
              expr: tenant_a()
              level: full

            # Capture only errors for tenant B
            - name: "Tenant B errors"
              expr: tenant_b() && http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: multi_tenant

Understanding Traefik + Qtap

Dual Capture for Dynamic Routing

When Traefik routes a request, Qtap captures two transactions:

Transaction 1: INGRESS (Client → Traefik)

Source Process: traefik
Direction: INGRESS ←
URL: http://localhost:8083/api/service-a/users

Transaction 2: EGRESS (Traefik → Backend)

Source Process: traefik
Direction: EGRESS →
URL: http://service-a:8000/users

Notice:

Path transformation: /api/service-a/users → /users (middleware stripped prefix)
Container resolution: service-a:8000 (Docker DNS)
Headers added: X-Forwarded-*

Traefik-Specific Features

Process Identification:

Look for exe containing traefik
Typically /usr/local/bin/traefik

Label-Based Configuration:

Unlike NGINX/Caddy, routing is defined via container labels
Qtap sees the result of routing decisions
Changes to labels automatically discovered (no restart needed)

Automatic Service Discovery:

Traefik watches Docker events
New containers auto-routed
Qtap captures new service traffic immediately

Troubleshooting

Not Seeing Traefik Traffic?

Check 1: Is Traefik routing correctly?

# Check Traefik dashboard
curl http://localhost:8084/api/rawdata

# Or check Traefik logs
docker logs traefik-demo

Check 2: Are services registered with Traefik?

# Verify labels are correct
docker inspect service-a | grep -A 10 Labels

Check 3: Is Qtap running before requests?

docker logs qtap-traefik | head -20

Check 4: Is ignore_loopback correct?

tap:
  ignore_loopback: false  # MUST be false for Docker networking

Seeing `"l7Protocol": "other"`?

Wait longer after starting Qtap (6+ seconds)
Check if Traefik is using HTTP/3 (not yet supported by Qtap)
Verify traffic is actually HTTP

Labels Not Working?

Common label mistakes:

# ❌ Wrong - missing quotes
- traefik.http.routers.myapp.rule=PathPrefix(/api)

# ✅ Correct - quoted value
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

# ❌ Wrong - missing enable
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

# ✅ Correct - enable=true required
- "traefik.enable=true"
- "traefik.http.routers.myapp.rule=PathPrefix(`/api`)"

Too Much Traffic?

Apply conditional capture:

stacks:
  reduced:
    plugins:
      - type: http_capture
        config:
          level: none
          rules:
            - name: "Errors only"
              expr: http.res.status >= 400
              level: full

Performance Considerations

Traefik + Qtap Performance

CPU: ~1-3% overhead for typical traffic
Memory: ~50-200MB for Qtap
Latency: Zero additional latency (passive observation)

Best practices for high-traffic Traefik:

Use level: summary for high volume
Apply rules to capture selectively
Filter health checks and monitoring endpoints
Send to S3 with batching (Fluent Bit)
Set TTL policies on storage

Scaling Recommendations

Traffic Volume

Recommended Level

Notes

< 100 req/sec

full

Capture everything

100-1000 req/sec

details

Headers only

1000-10000 req/sec

summary

Metadata only

> 10000 req/sec

conditional

Errors/slow requests only

Traefik vs NGINX/Caddy

Configuration:

Traefik: Docker labels, dynamic discovery
NGINX: Static config files
Caddy: Static config files (but simpler)

Use Cases:

Traefik: Containerized apps, Kubernetes, dynamic environments
NGINX: Traditional deployments, high performance
Caddy: Simplicity, automatic HTTPS

Qtap Compatibility:

All three work perfectly with Qtap
Traefik's dynamic routing is fully observable
Same capture quality across all proxies

Next Steps

Learn More About Qtap:

Traffic Capture Settings - Complete configuration
Traffic Processing with Plugins - All plugin options
Complete Guide - Progressive tutorial

Production Deployment:

Storage Configuration - S3 setup guide
Capturing All HTTP Traffic with Fluent Bit - Batching for scale
Kubernetes Manifest - Deploy in K8s

Related Guides:

Capturing NGINX Traffic - Traditional reverse proxy
Capturing Caddy Traffic - Modern web server
Ingress Traffic Capture with Python - Application servers

Alternative: Cloud Management:

Qplane - Manage Qtap with visual dashboards

Cleanup

# Stop all services
docker compose down

# Remove images
docker compose down --rmi local

# Clean up files
rm backend-service.py traefik.yaml qtap.yaml docker-compose.yaml

This guide uses validated configurations. All examples are tested and guaranteed to work with Traefik and Qtap.

PreviousCapturing Caddy Traffic NextCapturing HAProxy Traffic

Last updated 10 hours ago

What You'll Learn

Use Cases

Prerequisites

Part 1: Traefik with Multiple Backends

Step 1: Create Project Directory

Step 2: Create Traefik Configuration

Step 3: Create Backend Services

Step 4: Create Qtap Configuration

Step 5: Create Docker Compose Setup

Part 2: Running and Testing

Step 1: Start the Services

Step 2: Generate Test Traffic

Step 3: View Captured Traffic

Part 3: Advanced Configurations

Configuration 1: Capture Only Specific Services

Configuration 2: Monitor Service Discovery

Configuration 3: API Gateway with Rate Limiting Detection

Configuration 4: Production Setup with S3

Part 4: Real-World Use Cases

Use Case 1: Microservices Mesh Monitoring

Use Case 2: Canary Deployment Monitoring

Use Case 3: Multi-Tenant API Gateway

Understanding Traefik + Qtap

Dual Capture for Dynamic Routing

Traefik-Specific Features

Troubleshooting

Not Seeing Traefik Traffic?

Seeing "l7Protocol": "other"?

Labels Not Working?

Too Much Traffic?

Performance Considerations

Traefik + Qtap Performance

Scaling Recommendations

Traefik vs NGINX/Caddy

Next Steps

Cleanup

Seeing `"l7Protocol": "other"`?