> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/guides/qtap-guides/getting-started/sending-all-traffic-to-s3.md).

# Sending All Traffic to S3

Capture all HTTP/HTTPS traffic and store it in S3-compatible storage for persistent, secure access.

## Who This Is For

**Use this guide if you want to:**

* Store all captured traffic persistently in S3
* Keep sensitive HTTP data within your own infrastructure
* Set up a simple, no-filtering capture pipeline
* Get started with S3 storage before adding rules later

**Prerequisites:**

* S3-compatible storage ready (AWS S3, MinIO, Google Cloud Storage)
* S3 credentials (access key and secret key)
* S3 bucket created

**Time to complete:** 10 minutes

***

## Configuration

Save this as `qtap-s3.yaml`:

```yaml
version: 2

# Storage Configuration
services:
  # Connection metadata to console (for visibility)
  event_stores:
    - type: stdout

  # HTTP payloads to S3 (where sensitive data lives)
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com          # Your S3 endpoint
      bucket: my-qtap-data                # Your bucket name
      region: us-east-1                   # Your region
      access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
      insecure: false                     # Use HTTPS
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

# Processing Stack
stacks:
  capture_all:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|headers|full) - Capture everything
          format: json     # (json|text) - Structured for storage

# Traffic Capture Settings
tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true    # (true|false) - Skip localhost
  audit_include_dns: false # (true|false) - Skip DNS queries
  http:
    stack: capture_all
```

## Running Qtap

### Set Your S3 Credentials

```bash
export S3_ACCESS_KEY=your_access_key
export S3_SECRET_KEY=your_secret_key
```

### Start Qtap with Docker

```bash
docker run -d \
  --name qtap-s3 \
  --user 0:0 \
  --privileged \
  --cap-add CAP_BPF \
  --cap-add CAP_SYS_ADMIN \
  --pid=host \
  --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-s3.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  -e S3_ACCESS_KEY \
  -e S3_SECRET_KEY \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"
```

### Or with Linux Binary

```bash
sudo S3_ACCESS_KEY=$S3_ACCESS_KEY S3_SECRET_KEY=$S3_SECRET_KEY \
  qtap --config=qtap-s3.yaml
```

## Testing

Wait for qtap to initialize, then generate some traffic:

```bash
sleep 6
curl https://httpbin.org/get
```

Check the logs for confirmation:

```bash
docker logs qtap-s3 2>&1 | grep "httpbin"
```

Verify objects are in your S3 bucket:

```bash
aws s3 ls s3://my-qtap-data/
```

## S3 Provider Examples

### AWS S3

```yaml
object_stores:
  - type: s3
    endpoint: s3.amazonaws.com
    bucket: my-qtap-data
    region: us-west-2
    access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: AWS_ACCESS_KEY_ID
    secret_key:
      type: env
      value: AWS_SECRET_ACCESS_KEY
```

### MinIO (Self-Hosted)

```yaml
object_stores:
  - type: s3
    endpoint: minio.internal:9000
    bucket: qtap-data
    region: us-east-1
    access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: MINIO_ACCESS_KEY
    secret_key:
      type: env
      value: MINIO_SECRET_KEY
```

### Google Cloud Storage

```yaml
object_stores:
  - type: s3
    endpoint: storage.googleapis.com
    bucket: my-qtap-data
    region: us-central1
    access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: GCS_ACCESS_KEY
    secret_key:
      type: env
      value: GCS_SECRET_KEY
```

## Understanding the Output

With this configuration:

* **Console output** shows connection metadata (timestamps, endpoints, duration)
* **S3 bucket** stores full HTTP payloads (headers, bodies, sensitive data)

Each captured transaction creates an object in S3 with a unique digest identifier. The `access_url` template lets you construct URLs to retrieve stored objects.

## Capture Levels

Adjust the `level` setting based on your needs:

| Level     | What's Captured                   | Use Case                             |
| --------- | --------------------------------- | ------------------------------------ |
| `none`    | Nothing                           | Use with rules for selective capture |
| `summary` | Method, URL, status, duration     | Lightweight monitoring               |
| `headers` | Summary + all headers             | Header inspection                    |
| `full`    | Headers + request/response bodies | Complete visibility                  |

## What's Next?

* **Add filtering rules**: [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md)
* **Capture only errors**: Set `level: none` as default, then add rules to capture `http.res.status >= 400` at `level: full`
* **Kubernetes deployment**: [Helm Chart](/getting-started/qtap/installation/helm-chart.md) with S3 secrets
* **Full configuration reference**: [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md)