Capturing Envoy Traffic

This guide shows you how to use Qtap to capture HTTP traffic flowing through Envoy, the modern cloud-native proxy designed for service meshes. You'll learn how to observe both incoming and outgoing connections in a microservices architecture.

What You'll Learn

Capture Envoy ingress traffic (incoming requests)
Capture Envoy egress traffic (upstream service requests)
Monitor service mesh traffic patterns
Configure Envoy listeners, routes, and clusters
Apply conditional capture rules
Set up Envoy + Qtap in Docker for testing
Deploy production-ready configurations

Use Cases

Why capture Envoy traffic?

Service Mesh Visibility: Monitor all service-to-service communication
Istio/Consul Integration: Observe traffic in service mesh deployments
gRPC Traffic Inspection: See gRPC calls between microservices
Debugging Routing: Verify Envoy routes traffic correctly
Observability: Deep insights into request/response patterns
Security Auditing: Monitor for suspicious traffic patterns
Performance Analysis: Measure latency and identify bottlenecks

Prerequisites

Linux system with kernel 5.10+ and eBPF support
Docker installed (for this guide's examples)
Root/sudo access
Basic understanding of Envoy configuration concepts

Part 1: Basic Envoy Proxy Setup

Envoy uses YAML configuration with a specific structure: listeners, routes, and clusters.

Step 1: Create Project Directory

mkdir envoy-qtap-demo
cd envoy-qtap-demo

Step 2: Create Envoy Configuration

Create envoy.yaml:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              # Route /api/* to API service
              - match:
                  prefix: "/api/"
                route:
                  prefix_rewrite: "/"
                  cluster: api_service
              # Route /web/* to web service
              - match:
                  prefix: "/web/"
                route:
                  prefix_rewrite: "/"
                  cluster: web_service
              # Default route to httpbin
              - match:
                  prefix: "/"
                route:
                  cluster: httpbin_service

  clusters:
  - name: api_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: api_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-api
                port_value: 8001

  - name: web_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: web_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-web
                port_value: 8002

  - name: httpbin_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: httpbin_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: httpbin.org
                port_value: 80

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

Step 3: Create Backend Service

Create backend-service.py:

#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import os

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "path": self.path,
            "message": f"Hello from {service_name}!"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def do_POST(self):
        content_length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(content_length).decode() if content_length > 0 else ""

        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "method": "POST",
            "received": body
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    port = int(os.getenv('PORT', '8000'))
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}")
    server.serve_forever()

Step 4: Create Qtap Configuration

Create qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  envoy_capture:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|details|full)
          format: text     # (json|text)

tap:
  direction: all           # (egress|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)

  http:
    stack: envoy_capture

  filters:
    groups:
      - qpoint

Step 5: Create Docker Compose Setup

Create docker-compose.yaml:

version: '3.8'

services:
  # Envoy proxy
  envoy:
    image: envoyproxy/envoy:v1.31-latest
    container_name: envoy-demo
    ports:
      - "8087:10000"    # HTTP listener
      - "8088:9901"     # Admin interface
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
    command: ["-c", "/etc/envoy/envoy.yaml"]
    networks:
      - demo-network

  # Backend API service
  backend-api:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py"]
    container_name: backend-api
    environment:
      - SERVICE_NAME=backend-api
      - PORT=8001
    networks:
      - demo-network

  # Backend web service
  backend-web:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py"]
    container_name: backend-web
    environment:
      - SERVICE_NAME=backend-web
      - PORT=8002
    networks:
      - demo-network

  # Qtap agent
  qtap:
    image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0
    container_name: qtap-envoy
    privileged: true
    user: "0:0"
    cap_add:
      - CAP_BPF
      - CAP_SYS_ADMIN
    pid: host
    network_mode: host
    volumes:
      - /sys:/sys
      - /var/run/docker.sock:/var/run/docker.sock
      - ./qtap.yaml:/app/config/qtap.yaml
    environment:
      - TINI_SUBREAPER=1
    ulimits:
      memlock: -1
    command:
      - --log-level=warn
      - --log-encoding=console
      - --config=/app/config/qtap.yaml

networks:
  demo-network:
    driver: bridge

Key Envoy Concepts:

Listeners: Accept incoming connections
Routes: Map requests to clusters
Clusters: Groups of upstream endpoints
Filters: Process requests (router, logging, etc.)

Part 2: Running and Testing

Step 1: Start the Services

# Start all services
docker compose up -d

# Wait for Qtap to initialize (CRITICAL!)
sleep 6

# Check Envoy admin interface (optional)
curl http://localhost:8088/stats

Step 2: Generate Test Traffic

# Test 1: Route to API service
curl http://localhost:8087/api/users

# Test 2: Route to web service
curl http://localhost:8087/web/index

# Test 3: Default route (httpbin)
curl http://localhost:8087/get

# Test 4: POST to API service
curl -X POST http://localhost:8087/api/create \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice"}'

# Test 5: Multiple requests to see routing
for i in {1..5}; do
  curl -s http://localhost:8087/api/data | jq .
  curl -s http://localhost:8087/web/page | jq .
done

Step 3: View Captured Traffic

# View Qtap logs
docker logs qtap-envoy

# Filter for envoy process
docker logs qtap-envoy 2>&1 | grep -A 30 "envoy"

# Check directions
docker logs qtap-envoy 2>&1 | grep "Direction:"

What you should see:

=== HTTP Transaction ===
Source Process: envoy (PID: 123, Container: envoy-demo)
Direction: INGRESS ← (client to envoy)
Method: POST
URL: http://localhost:8087/api/create
Status: 200 OK

--- Request Headers ---
Host: localhost:8087
Content-Type: application/json

--- Request Body ---
{"name": "Alice"}
========================

=== HTTP Transaction ===
Source Process: envoy (PID: 123, Container: envoy-demo)
Direction: EGRESS → (envoy to backend)
Method: POST
URL: http://backend-api:8001/create
Status: 200 OK
========================

Key indicators:

✅ "exe" contains envoy
✅ Direction: INGRESS - Client → Envoy
✅ Direction: EGRESS - Envoy → Backend
✅ Path transformation visible
✅ Cluster routing working

Part 3: Advanced Configurations

Configuration 1: Service Mesh Monitoring

Capture service-to-service traffic:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  mesh_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Metadata for analytics
          format: json

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: mesh_monitoring

Configuration 2: Selective Capture by Service

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: is_api_service
      expr: http.req.path matches /^\/api\//
    - name: is_error
      expr: http.res.status >= 400

stacks:
  selective:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture API traffic
            - name: "API traffic"
              expr: is_api_service()
              level: full

            # Capture errors
            - name: "Errors"
              expr: is_error()
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: selective

Configuration 3: Production with S3

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: s3
      config:
        endpoint: https://s3.amazonaws.com
        region: us-east-1
        bucket: my-company-envoy-traffic
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

stacks:
  production:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            - name: "Errors only"
              expr: http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: production

Real-World Use Cases

Use Case 1: Istio Service Mesh

Monitor Envoy sidecars in Istio:

# Qtap deployed as DaemonSet captures all Envoy traffic
tap:
  direction: all
  http:
    stack: service_mesh_capture

# Analyze service-to-service communication
# See which services talk to each other
# Measure latency between services

Use Case 2: API Gateway

Envoy as edge proxy:

rulekit:
  macros:
    - name: is_external
      expr: http.req.header.x-forwarded-for != ""

stacks:
  api_gateway:
    plugins:
      - type: http_capture
        config:
          level: none
          rules:
            # Capture external traffic
            - name: "External requests"
              expr: is_external()
              level: full

Troubleshooting

Not Seeing Envoy Traffic?

Check 1: Is Envoy running?

docker logs envoy-demo
curl http://localhost:8088/stats

Check 2: Qtap running before traffic?

docker logs qtap-envoy | head -20

Check 3: Envoy routing correctly?

# Check Envoy config
curl http://localhost:8088/config_dump

Seeing `"l7Protocol": "other"`?

Envoy might be using HTTP/2 or HTTP/3
Wait longer after starting Qtap
Check if TLS is involved

Performance Considerations

Envoy + Qtap:

CPU: ~1-3% overhead
Memory: ~50-200MB
Latency: Zero additional (passive)

Best practices:

Use summary for high volume
Apply conditional rules
Filter admin traffic
Send to S3 with batching

Next Steps

Learn More:

Related Guides:

Cleanup

docker compose down
docker compose down --rmi local
rm backend-service.py envoy.yaml qtap.yaml docker-compose.yaml

This guide uses validated configurations for Envoy and Qtap.

PreviousCapturing HAProxy Traffic NextHTTPS Header Capture Without Proxies

Last updated 16 hours ago