Storage Configuration

Understanding Qtap Storage Components

Qtap captures two distinct types of data, each with different storage requirements:

Events (Connection Metadata): Anonymized information about connections, including timestamps, endpoints, and performance metrics
Objects (Payload Content): Actual request and response data including headers and bodies, which may contain sensitive information

Each type has its own dedicated storage configuration in the services section of your qpoint.yaml file.

Event Stores

Event stores handle anonymized metadata about network connections. This data is generally not sensitive but is useful for analytics, troubleshooting, and monitoring.

Console Output (stdout)

The simplest option for development and debugging:

event_stores:
  - id: console_stdout
    type: stdout

This configuration sends all event data to the console where Qtap is running, making it immediately visible but not persistent.

Pulse Service (Coming Soon)

For self-hosted environments with a Pulse instance:

event_stores:
  - id: pulse
    type: pulse
    endpoint: http://pulse-service:8000
    token: 
      type: env
      value: PULSE_TOKEN

This connects to a Pulse service for advanced analytics and visualization.

Object Stores

Object stores contain the actual content of requests and responses, which often includes sensitive information. This data requires more careful handling and secure storage.

Console Output (stdout)

For development and debugging:

object_stores:
  - id: console_stdout
    type: stdout

Sends all object data to the console.

S3-Compatible Storage

For secure, persistent storage:

object_stores:
  - id: s3_store
    type: s3
    endpoint: storage.example.com:9000
    bucket: qpoint-objects
    region: us-east-1
    access_url: https://storage.example.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: S3_ACCESS_KEY
    secret_key:
      type: env
      value: S3_SECRET_KEY

This configuration:

Stores objects in an S3-compatible storage service
Uses HTTPS for secure transmission (insecure: false)
Retrieves credentials from environment variables
Provides a template URL for accessing stored objects

S3 Configuration Parameters

Parameter

Description

Example

endpoint

S3 server hostname and port

minio.example.com:9000

bucket

S3 bucket name

qpoint-objects

region

S3 region name

us-east-1

access_url

URL template for object access

https://storage.example.com/{{BUCKET}}/{{DIGEST}}

insecure

Allow HTTP instead of HTTPS

false (recommended)

access_key

S3 access key configuration

See credential management

secret_key

S3 secret key configuration

See credential management

URL Template Variables

The access_url parameter supports these template variables:

{{ENDPOINT}}: The S3 endpoint
{{BUCKET}}: The bucket name
{{DIGEST}}: The unique file identifier

Credential Management

For security, Qtap supports retrieving credentials from environment variables rather than storing them directly in the configuration file.

Environment Variable Configuration

access_key:
  type: env
  value: S3_ACCESS_KEY  # Name of the environment variable

When running Qtap, ensure these environment variables are set:

export S3_ACCESS_KEY=your_access_key
export S3_SECRET_KEY=your_secret_key

For Docker:

docker run \
  # Other parameters...
  -e S3_ACCESS_KEY=your_access_key \
  -e S3_SECRET_KEY=your_secret_key \
  # Rest of command...

For Kubernetes, use secrets:

kubectl create secret generic s3-credentials \
  --from-literal=access-key='YOUR_ACCESS_KEY' \
  --from-literal=secret-key='YOUR_SECRET_KEY' \
  -n qpoint

And reference them in your Helm values:

extraEnv:
  - name: S3_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: access-key
  - name: S3_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: secret-key