# Object Storage Configuration

Object storage is where Qpoint stores sensitive request and response payloads captured by your Qtap agents.

⚠️ **Before Production Deployment:** By default, new Qplane accounts are configured to use Qpoint Cloud object storage for testing and development. **You must configure your own object storage (S3, GCS, Azure Blob, or MinIO) before deploying Qtap agents in any production or sensitive environment** to ensure sensitive data stays within your infrastructure.

This guide explains when data is stored, how to configure your own object storage, and security best practices for production deployments.

## Understanding Qpoint's Data Architecture

Qpoint uses a "separation of data" architecture to preserve data sovereignty and give you complete control over sensitive information:

**Anonymized Metadata → Pulse (Qpoint Cloud)**

* Connection metadata: IPs, ports, timings, status codes
* Sent to Qpoint's cloud control plane (Pulse) for dashboards, metrics, and alerts
* Contains no sensitive payload data
* Required for Qplane dashboards to function

**Sensitive Payloads → Object Storage**

* Actual request/response headers and bodies
* By default, sent to Qpoint Cloud object storage (testing only)
* **For production:** Configure your own object storage so payloads are sent directly to YOUR infrastructure (S3, GCS, MinIO, etc.)
* Qplane UI fetches payloads on-demand directly to your browser
* Configuring your own storage preserves data sovereignty and regulatory compliance

## When Data is Stored in Object Storage

⚠️ **Critical:** Not all traffic is stored to object storage. Payloads are only captured and stored when specific plugins are configured to do so.

### Plugins That Write to Object Storage

**1. Detect Errors Plugin**

Captures full request/response details when HTTP errors occur.

**When it writes:**

* HTTP status codes: 404, 500, 502, 503, 504, and other configurable error codes
* Automatically enabled in Qplane with pre-configured error detection rules

**What it stores:**

* Complete request headers and body
* Complete response headers and body
* Timing information

**Learn more:** [Detect Errors Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md#detect-errors)

**2. HTTP Capture Plugin**

More flexible, rule-based capture for any traffic matching your criteria.

**When it writes:**

* Only when capture level is set to `details` or `full`
* Based on configurable rulekit expressions (domain, path, method, headers, etc.)

**Capture levels:**

* `none` - No object storage writes
* `summary` - No object storage writes
* `details` - Writes headers to object storage
* `full` - Writes headers AND bodies to object storage

**Example use case:** Capture all calls to `api.openai.com` to audit what prompts are being sent.

**Learn more:** [HTTP Capture Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/traffic-processing-with-plugins.md#http-capture)

**3. Scan Payloads & Data (QScan) Plugin**

Classifies sensitive data in traffic (PII, credentials, API keys, etc.).

**When it writes:**

* When **"Record matching metadata in object store"** is enabled
  * Stores metadata about detections: path in payload, detection score, data type found
* When **"Store value with metadata"** is checked for specific data types
  * Also stores the actual matched sensitive values (e.g., the email address, phone number, or credential found)

**What it detects:**

* PII (names, emails, phone numbers, addresses, SSNs, dates of birth)
* Credentials (passwords, API keys, tokens, usernames)
* Financial data (credit cards, bank accounts, IBAN codes)
* Location data (IP addresses, street addresses, cities, zip codes)
* Medical identifiers (medical license numbers)
* Custom patterns you define

**What gets stored in object storage:**

* **Metadata (when enabled):** Detection type, location in payload, confidence score
* **Values (per data type):** The actual sensitive data matched (if "Store value with metadata" is checked for that type)

**Learn more:** [QScan Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md#scan-payloads-data-qscan)

### Plugins That Do NOT Write to Object Storage

**Report Usage Plugin**

* Only sends anonymized metadata to Pulse
* No payloads stored anywhere

**Access Logs Plugin**

* Writes formatted logs to stdout only
* No object storage interaction

**HTTP Metrics Plugin**

* Exposes Prometheus metrics only
* No object storage interaction

## Before You Deploy: Configuration Requirements

⚠️ **WARNING: Configure Your Object Storage BEFORE Deploying in Sensitive Environments**

**Default Behavior:**

* Qpoint Cloud object storage is configured by default
* Designed for testing, preview, and proof-of-concept only
* **NOT suitable for production or sensitive data**

**Why This Matters:**

* **Default error detection is active immediately** - The pre-configured "Basic Reporting and Error Detection" stack starts capturing error payloads (400, 401, 403, 404, 429, 500, 502, 503, etc.) as soon as an agent deploys
* Once an agent starts capturing traffic, payloads are immediately sent to the configured object storage
* If you haven't configured your own storage, **sensitive data in error responses will be sent to Qpoint Cloud**
* Reconfiguring later doesn't retroactively move already-captured data
* See [What's Captured by Default](#whats-captured-by-default-in-new-accounts) above for the complete list of error codes that trigger captures

**Before deploying Qtap in any environment with sensitive data:**

1. ✅ Configure your own object storage (S3, GCS, Azure Blob, MinIO)
2. ✅ Verify the configuration in Snapshot YAML
3. ✅ Test with a non-sensitive agent deployment first
4. ✅ Only then deploy to production/sensitive environments

## What's Captured by Default in New Accounts

⚠️ **Important:** Every new Qplane account automatically includes a "Basic Reporting and Error Detection" stack with pre-configured error detection rules. This means **object storage writes begin immediately** when your agents detect errors.

**Default Error Detection Rules:**

Every new account comes with 6 pre-configured rules that automatically capture errors:

By default, **full request and response data** (headers AND bodies) is captured to object storage for these status codes:

| Error Type                | Status Codes                    | What's Captured         |
| ------------------------- | ------------------------------- | ----------------------- |
| **App Error**             | 500                             | Full request + response |
| **Infrastructure Outage** | 502, 503, 520-523, 525-526, 530 | Full request + response |
| **Client Error**          | 400                             | Full request + response |
| **Authentication Error**  | 401, 403, 407                   | Full request + response |
| **Rate Limited**          | 429                             | Full request + response |
| **Not Found**             | 404                             | Full request + response |

**Why This Matters:**

* ✅ **Benefit:** Immediate error visibility and debugging capability from day one
* ⚠️ **Risk:** If you haven't configured your own object storage, these payloads go to Qpoint Cloud (testing-only storage)
* 🔒 **Action Required:** Configure your own object storage **BEFORE** deploying agents in any environment with sensitive data

**Where Object Storage is NOT Used:**

The `report_usage` plugin (also in the default stack) only sends anonymized metadata to Pulse - it does **not** write to object storage.

## Supported Object Storage Providers

Qpoint supports any S3-compatible object storage:

| Provider                  | Type        | Deployment          | Best For                              |
| ------------------------- | ----------- | ------------------- | ------------------------------------- |
| **AWS S3**                | Cloud       | AWS-managed         | AWS customers, production at scale    |
| **Google Cloud Storage**  | Cloud       | GCP-managed         | GCP customers, multi-cloud            |
| **Azure Blob Storage**    | Cloud       | Azure-managed       | Azure customers                       |
| **MinIO**                 | Self-hosted | Your infrastructure | Air-gapped environments, full control |
| **Any S3-compatible API** | Varies      | Varies              | Custom requirements                   |

## Configuration Guide

### Step 1: Navigate to Object Stores

1. In Qplane, go to **Settings → Deploy → Services**
2. Find the **Object Stores** section
3. Click **"+ Add Object Store"**

### Step 2: Configure Your Provider

#### AWS S3 Configuration

**Prerequisites:**

* S3 bucket created (e.g., `my-company-qpoint-payloads`)
* IAM user with S3 write permissions and an access key pair (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`)

**Configuration parameters:**

```yaml
object_stores:
  - id: production_s3
    type: s3
    endpoint: s3.amazonaws.com
    bucket: my-company-qpoint-payloads
    region: us-west-2
    access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
    access_key:
      type: env
      value: AWS_ACCESS_KEY_ID
    secret_key:
      type: env
      value: AWS_SECRET_ACCESS_KEY
```

**Parameter explanations:**

* `endpoint` - S3 service endpoint (use `s3.amazonaws.com` for AWS)
* `bucket` - Your bucket name (must exist before deploying agents)
* `region` - AWS region where bucket is located
* `access_url` - Template for retrieving stored objects (Qplane uses this to fetch payloads for display)
* `access_key` / `secret_key` - Required for S3 authentication (IAM roles and ambient credential chains are not currently supported)

**IAM Policy Example:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::my-company-qpoint-payloads/*"
    }
  ]
}
```

**See also:** [AWS S3 Configuration Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/aws-s3.md)

#### Google Cloud Storage Configuration

**Configuration parameters:**

```yaml
object_stores:
  - id: production_gcs
    type: gcs
    endpoint: storage.googleapis.com
    bucket: my-company-qpoint-payloads
    region: us-west1
    access_url: https://storage.googleapis.com/{{BUCKET}}/{{DIGEST}}
    credentials:
      type: env
      value: GOOGLE_APPLICATION_CREDENTIALS
```

**See also:** [Google Cloud Storage Guide](/appendix/object-storage/google-cloud-storage.md)

#### MinIO Configuration (Self-Hosted)

**Configuration parameters:**

```yaml
object_stores:
  - id: internal_minio
    type: s3
    endpoint: minio.internal.company.com:9000
    bucket: qpoint-payloads
    region: us-east-1
    access_url: https://minio.internal.company.com:9000/{{BUCKET}}/{{DIGEST}}
    access_key:
      type: env
      value: MINIO_ACCESS_KEY
    secret_key:
      type: env
      value: MINIO_SECRET_KEY
```

**Notes:**

* MinIO uses the S3-compatible API (`type: s3`)
* Endpoint should include your MinIO host and port
* Region can be any value (MinIO doesn't enforce AWS regions)

**See also:** [MinIO Self-Hosted Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/minio.md)

### Step 3: Set Environment Variables (if using credentials)

If your configuration references environment variables for credentials, you must set these on the hosts running Qtap agents.

**Example for systemd:**

```bash
# /etc/systemd/system/qtap.service.d/override.conf
[Service]
Environment="AWS_ACCESS_KEY_ID=AKIA..."
Environment="AWS_SECRET_ACCESS_KEY=..."
```

**Example for Kubernetes:**

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: qpoint-object-storage
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <base64-encoded>
  AWS_SECRET_ACCESS_KEY: <base64-encoded>
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: qtap
spec:
  template:
    spec:
      containers:
      - name: qtap
        envFrom:
        - secretRef:
            name: qpoint-object-storage
```

**See also:** [Qtap Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) for local YAML configuration examples

## Verifying Your Configuration

### 1. Check Snapshot YAML

After adding your object storage:

1. Go to **Settings → Deploy → Snapshot**
2. Look for your object store in the `services.object_stores` section
3. Verify all parameters are correct (endpoint, bucket, region)

**Example of what you should see:**

```yaml
services:
  object_stores:
    - id: production_s3
      type: s3
      endpoint: s3.amazonaws.com
      bucket: my-company-qpoint-payloads
      region: us-west-2
      access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
```

### 2. Deploy a Test Agent

Before deploying to production:

1. Deploy Qtap in a non-sensitive test environment
2. Configure a plugin to capture traffic (e.g., HTTP Capture with `level: full`)
3. Generate test traffic that matches your capture rules
4. Check your object storage bucket for uploaded objects

**Expected behavior:**

* Objects appear in your bucket within seconds of captured requests
* Object keys are SHA256 digests (e.g., `abc123def456...`)
* Objects contain JSON-encoded request/response data

### 3. Monitor Agent Logs

Qtap logs object storage upload attempts:

```bash
# View Qtap logs (systemd)
journalctl -u qtap -f

# View Qtap logs (Kubernetes)
kubectl logs -n qpoint daemonset/qtap -f
```

**What to look for:**

✅ **Success:**

```
[INFO] Uploaded payload to s3://my-bucket/abc123...
```

❌ **Failure:**

```
[ERROR] Failed to upload to object storage: Access Denied
[ERROR] Failed to upload to object storage: No such bucket
```

**Common issues:**

* Incorrect credentials or IAM permissions
* Bucket doesn't exist
* Network connectivity to object storage endpoint
* Incorrect region configuration

## Related Documentation

* [Settings Overview](/getting-started/qplane/configuration/settings.md) - Complete settings reference
* [Stacks & Plugins](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md) - Learn about plugins that capture data
* [Traffic Processing with Plugins](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/traffic-processing-with-plugins.md) - HTTP Capture plugin details
* [Qtap Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) - Local YAML configuration examples
* [AWS S3 Configuration Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/aws-s3.md) - Provider-specific setup
* [Google Cloud Storage Guide](/appendix/object-storage/google-cloud-storage.md) - GCS-specific setup
* [MinIO Self-Hosted Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/minio.md) - Self-hosted MinIO setup


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.qpoint.io/getting-started/qplane/configuration/object-storage.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.