> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/guides/qtap-guides/getting-started/sending-all-traffic-to-s3.md).

# Sending All Traffic to S3

Capture all HTTP/HTTPS traffic and store it in S3-compatible storage for persistent, secure access.

## Who This Is For

**Use this guide if you want to:**

* Store all captured traffic persistently in S3
* Keep sensitive HTTP data within your own infrastructure
* Set up a simple, no-filtering capture pipeline
* Get started with S3 storage before adding rules later

**Prerequisites:**

* S3-compatible storage ready (AWS S3, MinIO, Google Cloud Storage)
* S3 credentials (access key and secret key)
* S3 bucket created

**Time to complete:** 10 minutes

***

## Configuration

Save this as `qtap-s3.yaml`:

```yaml
version: 2

# Storage Configuration
services:
  # Connection metadata to console (for visibility)
  event_stores:
    - type: stdout

  # HTTP payloads to S3 (where sensitive data lives)
  object_stores:
    - type: s3
      endpoint: s3.amazonaws.com          # Your S3 endpoint
      bucket: my-qtap-data                # Your bucket name
      region: us-east-1                   # Your region
      access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
      insecure: false                     # Use HTTPS
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

# Processing Stack
stacks:
  capture_all:
    plugins:
      - type: http_capture
        config:
          level: full      # (none|summary|headers|full) - Capture everything
          format: json     # (json|text) - Structured for storage

# Traffic Capture Settings
tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true    # (true|false) - Skip localhost
  audit_include_dns: false # (true|false) - Skip DNS queries
  http:
    stack: capture_all
```

## Running Qtap

### Set Your S3 Credentials

```bash
export S3_ACCESS_KEY=your_access_key
export S3_SECRET_KEY=your_secret_key
```

### Start Qtap with Docker

```bash
docker run -d \
  --name qtap-s3 \
  --user 0:0 \
  --privileged \
  --cap-add CAP_BPF \
  --cap-add CAP_SYS_ADMIN \
  --pid=host \
  --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-s3.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  -e S3_ACCESS_KEY \
  -e S3_SECRET_KEY \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"
```

### Or with Linux Binary

```bash
sudo S3_ACCESS_KEY=$S3_ACCESS_KEY S3_SECRET_KEY=$S3_SECRET_KEY \
  qtap --config=qtap-s3.yaml
```

## Testing

Wait for qtap to initialize, then generate some traffic:

```bash
sleep 6
curl https://httpbin.org/get
```

Check the logs for confirmation:

```bash
docker logs qtap-s3 2>&1 | grep "httpbin"
```

Verify objects are in your S3 bucket:

```bash
aws s3 ls s3://my-qtap-data/
```

## S3 Provider Examples

### AWS S3

```yaml
object_stores:
  - type: s3
    endpoint: s3.amazonaws.com
    bucket: my-qtap-data
    region: us-west-2
    access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: AWS_ACCESS_KEY_ID
    secret_key:
      type: env
      value: AWS_SECRET_ACCESS_KEY
```

### MinIO (Self-Hosted)

```yaml
object_stores:
  - type: s3
    endpoint: minio.internal:9000
    bucket: qtap-data
    region: us-east-1
    access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: MINIO_ACCESS_KEY
    secret_key:
      type: env
      value: MINIO_SECRET_KEY
```

### Google Cloud Storage

```yaml
object_stores:
  - type: s3
    endpoint: storage.googleapis.com
    bucket: my-qtap-data
    region: us-central1
    access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: GCS_ACCESS_KEY
    secret_key:
      type: env
      value: GCS_SECRET_KEY
```

## Understanding the Output

With this configuration:

* **Console output** shows connection metadata (timestamps, endpoints, duration)
* **S3 bucket** stores full HTTP payloads (headers, bodies, sensitive data)

Each captured transaction creates an object in S3 with a unique digest identifier. The `access_url` template lets you construct URLs to retrieve stored objects.

## Capture Levels

Adjust the `level` setting based on your needs:

| Level     | What's Captured                   | Use Case                             |
| --------- | --------------------------------- | ------------------------------------ |
| `none`    | Nothing                           | Use with rules for selective capture |
| `summary` | Method, URL, status, duration     | Lightweight monitoring               |
| `headers` | Summary + all headers             | Header inspection                    |
| `full`    | Headers + request/response bodies | Complete visibility                  |

## What's Next?

* **Add filtering rules**: [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md)
* **Capture only errors**: Set `level: none` as default, then add rules to capture `http.res.status >= 400` at `level: full`
* **Kubernetes deployment**: [Helm Chart](/getting-started/qtap/installation/helm-chart.md) with S3 secrets
* **Full configuration reference**: [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.qpoint.io/guides/qtap-guides/getting-started/sending-all-traffic-to-s3.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.