Sending All Traffic to S3

Capture all HTTP/HTTPS traffic and store it in S3-compatible storage for persistent, secure access.

Who This Is For

Use this guide if you want to:

• Store all captured traffic persistently in S3

• Keep sensitive HTTP data within your own infrastructure

• Set up a simple, no-filtering capture pipeline

• Get started with S3 storage before adding rules later

Prerequisites:

• S3-compatible storage ready (AWS S3, MinIO, Google Cloud Storage)

• S3 credentials (access key and secret key)

• S3 bucket created

Time to complete: 10 minutes

---

Configuration

Save this as qtap-s3.yaml:

version: 2

# Storage Configuration
services:
  # Connection metadata to console (for visibility)
  event_stores:
    - type: stdout

  # HTTP payloads to S3 (where sensitive data lives)
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com          # Your S3 endpoint
      bucket: my-qtap-data                # Your bucket name
      region: us-east-1                   # Your region
      access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
      insecure: false                     # Use HTTPS
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

# Processing Stack
stacks:
  capture_all:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|headers|full) - Capture everything
          format: json     # (json|text) - Structured for storage

# Traffic Capture Settings
tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true    # (true|false) - Skip localhost
  audit_include_dns: false # (true|false) - Skip DNS queries
  http:
    stack: capture_all

Running Qtap

Set Your S3 Credentials

export S3_ACCESS_KEY=your_access_key
export S3_SECRET_KEY=your_secret_key

Start Qtap with Docker

docker run -d \
  --name qtap-s3 \
  --user 0:0 \
  --privileged \
  --cap-add CAP_BPF \
  --cap-add CAP_SYS_ADMIN \
  --pid=host \
  --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-s3.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  -e S3_ACCESS_KEY \
  -e S3_SECRET_KEY \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"

Or with Linux Binary

sudo S3_ACCESS_KEY=$S3_ACCESS_KEY S3_SECRET_KEY=$S3_SECRET_KEY \
  qtap --config=qtap-s3.yaml

Testing

Wait for qtap to initialize, then generate some traffic:

sleep 6
curl https://httpbin.org/get

Check the logs for confirmation:

docker logs qtap-s3 2>&1 | grep "httpbin"

Verify objects are in your S3 bucket:

aws s3 ls s3://my-qtap-data/

S3 Provider Examples

AWS S3

object_stores:
  - type: s3
    endpoint: s3.amazonaws.com
    bucket: my-qtap-data
    region: us-west-2
    access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: AWS_ACCESS_KEY_ID
    secret_key:
      type: env
      value: AWS_SECRET_ACCESS_KEY

MinIO (Self-Hosted)

object_stores:
  - type: s3
    endpoint: minio.internal:9000
    bucket: qtap-data
    region: us-east-1
    access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: MINIO_ACCESS_KEY
    secret_key:
      type: env
      value: MINIO_SECRET_KEY

Google Cloud Storage

object_stores:
  - type: s3
    endpoint: storage.googleapis.com
    bucket: my-qtap-data
    region: us-central1
    access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: GCS_ACCESS_KEY
    secret_key:
      type: env
      value: GCS_SECRET_KEY

Understanding the Output

With this configuration:

• Console output shows connection metadata (timestamps, endpoints, duration)

• S3 bucket stores full HTTP payloads (headers, bodies, sensitive data)

Each captured transaction creates an object in S3 with a unique digest identifier. The access_url template lets you construct URLs to retrieve stored objects.

Capture Levels

Adjust the level setting based on your needs:

Level	What's Captured	Use Case
none	Nothing	Use with rules for selective capture
summary	Method, URL, status, duration	Lightweight monitoring
headers	Summary + all headers	Header inspection
full	Headers + request/response bodies	Complete visibility

What's Next?

• Add filtering rules: Traffic Processing with Plugins

• Capture only errors: Set level: none as default, then add rules to capture http.res.status >= 400 at level: full

• Kubernetes deployment: Helm Chart with S3 secrets

• Full configuration reference: Storage Configuration