Envoy Traffic Capture

This guide shows you how to use Qtap to capture HTTP traffic flowing through Envoy, the modern cloud-native proxy designed for service meshes. You'll learn how to observe both incoming and outgoing connections in a microservices architecture.

What You'll Learn

• Capture Envoy ingress traffic (incoming requests)

• Capture Envoy egress traffic (upstream service requests)

• Monitor service mesh traffic patterns

• Configure Envoy listeners, routes, and clusters

• Apply conditional capture rules

• Set up Envoy + Qtap in Docker for testing

• Deploy production-ready configurations

Use Cases

Why capture Envoy traffic?

• Service Mesh Visibility: Monitor all service-to-service communication

• Istio/Consul Integration: Observe traffic in service mesh deployments

• gRPC Traffic Inspection: See gRPC calls between microservices

• Debugging Routing: Verify Envoy routes traffic correctly

• Observability: Deep insights into request/response patterns

• Security Auditing: Monitor for suspicious traffic patterns

• Performance Analysis: Measure latency and identify bottlenecks

---

Prerequisites

• Linux system with kernel 5.10+ and eBPF support

• Docker installed (for this guide's examples)

• Root/sudo access

• Basic understanding of Envoy configuration concepts

---

Part 1: Basic Envoy Proxy Setup

Envoy uses YAML configuration with a specific structure: listeners, routes, and clusters.

Step 1: Create Project Directory

mkdir envoy-qtap-demo
cd envoy-qtap-demo

Step 2: Create Envoy Configuration

Create envoy.yaml:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              # Route /api/* to API service
              - match:
                  prefix: "/api/"
                route:
                  prefix_rewrite: "/"
                  cluster: api_service
              # Route /web/* to web service
              - match:
                  prefix: "/web/"
                route:
                  prefix_rewrite: "/"
                  cluster: web_service
              # Default route to httpbin
              - match:
                  prefix: "/"
                route:
                  cluster: httpbin_service

  clusters:
  - name: api_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: api_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-api
                port_value: 8001

  - name: web_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: web_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-web
                port_value: 8002

  - name: httpbin_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: httpbin_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: httpbin.org
                port_value: 80

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

Step 3: Create Backend Service

Create backend-service.py:

#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import os

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "path": self.path,
            "message": f"Hello from {service_name}!"
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def do_POST(self):
        content_length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(content_length).decode() if content_length > 0 else ""

        service_name = os.getenv('SERVICE_NAME', 'unknown')

        response = {
            "service": service_name,
            "method": "POST",
            "received": body
        }

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(response).encode())

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    port = int(os.getenv('PORT', '8000'))
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}")
    server.serve_forever()

Step 4: Create Qtap Configuration

Create qtap.yaml:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  envoy_capture:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|details|full)
          format: text     # (json|text)

tap:
  direction: all           # (egress|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)

  http:
    stack: envoy_capture

  filters:
    groups:
      - qpoint

Step 5: Create Docker Compose Setup

Create docker-compose.yaml:

version: '3.8'

services:
  # Envoy proxy
  envoy:
    image: envoyproxy/envoy:v1.31-latest
    container_name: envoy-demo
    ports:
      - "8087:10000"    # HTTP listener
      - "8088:9901"     # Admin interface
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
    command: ["-c", "/etc/envoy/envoy.yaml"]
    networks:
      - demo-network

  # Backend API service
  backend-api:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py"]
    container_name: backend-api
    environment:
      - SERVICE_NAME=backend-api
      - PORT=8001
    networks:
      - demo-network

  # Backend web service
  backend-web:
    build:
      context: .
      dockerfile_inline: |
        FROM python:3.11-slim
        WORKDIR /app
        COPY backend-service.py /app/
        RUN chmod +x /app/backend-service.py
        CMD ["python3", "/app/backend-service.py"]
    container_name: backend-web
    environment:
      - SERVICE_NAME=backend-web
      - PORT=8002
    networks:
      - demo-network

  # Qtap agent
  qtap:
    image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0
    container_name: qtap-envoy
    privileged: true
    user: "0:0"
    cap_add:
      - CAP_BPF
      - CAP_SYS_ADMIN
    pid: host
    network_mode: host
    volumes:
      - /sys:/sys
      - /var/run/docker.sock:/var/run/docker.sock
      - ./qtap.yaml:/app/config/qtap.yaml
    environment:
      - TINI_SUBREAPER=1
    ulimits:
      memlock: -1
    command:
      - --log-level=warn
      - --log-encoding=console
      - --config=/app/config/qtap.yaml

networks:
  demo-network:
    driver: bridge

Key Envoy Concepts:

• Listeners: Accept incoming connections

• Routes: Map requests to clusters

• Clusters: Groups of upstream endpoints

• Filters: Process requests (router, logging, etc.)

---

Part 2: Running and Testing

Step 1: Start the Services

# Start all services
docker compose up -d

# Wait for Qtap to initialize (CRITICAL!)
sleep 6

# Check Envoy admin interface (optional)
curl http://localhost:8088/stats

Step 2: Generate Test Traffic

# Test 1: Route to API service
curl http://localhost:8087/api/users

# Test 2: Route to web service
curl http://localhost:8087/web/index

# Test 3: Default route (httpbin)
curl http://localhost:8087/get

# Test 4: POST to API service
curl -X POST http://localhost:8087/api/create \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice"}'

# Test 5: Multiple requests to see routing
for i in {1..5}; do
  curl -s http://localhost:8087/api/data | jq .
  curl -s http://localhost:8087/web/page | jq .
done

Step 3: View Captured Traffic

# View Qtap logs
docker logs qtap-envoy

# Filter for envoy process
docker logs qtap-envoy 2>&1 | grep -A 30 "envoy"

# Check directions
docker logs qtap-envoy 2>&1 | grep "Direction:"

What you should see:

=== HTTP Transaction ===
Source Process: envoy (PID: 123, Container: envoy-demo)
Direction: INGRESS ← (client to envoy)
Method: POST
URL: http://localhost:8087/api/create
Status: 200 OK

--- Request Headers ---
Host: localhost:8087
Content-Type: application/json

--- Request Body ---
{"name": "Alice"}
========================

=== HTTP Transaction ===
Source Process: envoy (PID: 123, Container: envoy-demo)
Direction: EGRESS → (envoy to backend)
Method: POST
URL: http://backend-api:8001/create
Status: 200 OK
========================

Key indicators:

• ✅ "exe" contains envoy

• ✅ Direction: INGRESS - Client → Envoy

• ✅ Direction: EGRESS - Envoy → Backend

• ✅ Path transformation visible

• ✅ Cluster routing working

---

Part 3: Advanced Configurations

Configuration 1: Service Mesh Monitoring

Capture service-to-service traffic:

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  mesh_monitoring:
    plugins:
      - type: http_capture
        config:
          level: summary     # Metadata for analytics
          format: json

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: mesh_monitoring

Configuration 2: Selective Capture by Service

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: is_api_service
      expr: http.req.path matches /^\/api\//
    - name: is_error
      expr: http.res.status >= 400

stacks:
  selective:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            # Capture API traffic
            - name: "API traffic"
              expr: is_api_service()
              level: full

            # Capture errors
            - name: "Errors"
              expr: is_error()
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: selective

Configuration 3: Production with S3

version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com
      region: us-east-1
      bucket: my-company-envoy-traffic
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY
      insecure: false

stacks:
  production:
    plugins:
      - type: http_capture
        config:
          level: none
          format: json
          rules:
            - name: "Errors only"
              expr: http.res.status >= 400
              level: full

tap:
  direction: all
  ignore_loopback: false
  http:
    stack: production

---

Real-World Use Cases

Use Case 1: Istio Service Mesh

Monitor Envoy sidecars in Istio:

# Qtap deployed as DaemonSet captures all Envoy traffic
tap:
  direction: all
  http:
    stack: service_mesh_capture

# Analyze service-to-service communication
# See which services talk to each other
# Measure latency between services

Use Case 2: API Gateway

Envoy as edge proxy:

rulekit:
  macros:
    - name: is_external
      expr: http.req.headers.x-forwarded-for != ""

stacks:
  api_gateway:
    plugins:
      - type: http_capture
        config:
          level: none
          rules:
            # Capture external traffic
            - name: "External requests"
              expr: is_external()
              level: full

---

Troubleshooting

Not Seeing Envoy Traffic?

Check 1: Is Envoy running?

docker logs envoy-demo
curl http://localhost:8088/stats

Check 2: Qtap running before traffic?

docker logs qtap-envoy | head -20

Check 3: Envoy routing correctly?

# Check Envoy config
curl http://localhost:8088/config_dump

Seeing "l7Protocol": "other"?

• Envoy might be using HTTP/2 or HTTP/3

• Wait longer after starting Qtap

• Check if TLS is involved

---

Performance Considerations

Envoy + Qtap:

• CPU: ~1-3% overhead

• Memory: ~50-200MB

• Latency: Zero additional (passive)

Best practices:

1. Use summary for high volume

2. Apply conditional rules

3. Filter admin traffic

4. Send to S3 with batching

---

Next Steps

Learn More:

• Traffic Capture Settings

• Storage Configuration

Related Guides:

• Capturing NGINX Traffic

• Capturing Traefik Traffic

• Capturing HAProxy Traffic

---

Cleanup

docker compose down
docker compose down --rmi local
rm backend-service.py envoy.yaml qtap.yaml docker-compose.yaml

---

This guide uses validated configurations for Envoy and Qtap.