Documentation
  • Introduction
    • How It Works
    • Architecture & Data Flow
    • Why another Agent?
    • eBPF Concepts
    • Use Cases
  • Deployment
  • Qtap
    • Getting Started
    • System Requirements
    • Installation
      • Linux Binary
      • Docker Container
      • Helm Chart
      • Kubernetes Manifest
    • Configuration
      • Storage Configuration
      • Traffic Processing with Plugins
      • Traffic Capture Settings
      • Configuration Examples
  • Qplane
    • Getting Started
      • Create an Account
      • Install Qtap
      • Review your Dashboards
    • Installation
      • Linux Binary
      • Docker Container
      • Helm Chart
    • Configuration
  • Security & Compliance
  • License
  • Appendix
    • Qcontrol (Beta)
    • Java
    • Object Storage
      • Google Cloud Storage
    • S3 Credentials for Qtap using Kubernetes Secrets
  • FAQ
Powered by GitBook
On this page
  • Understanding Qtap Storage Components
  • Event Stores
  • Console Output (stdout)
  • Pulse Service (Coming Soon)
  • Object Stores
  • Console Output (stdout)
  • S3-Compatible Storage
  • Credential Management
  • Storage Configuration Examples
  • MinIO Configuration
  • AWS S3 Configuration
  • Google Cloud Storage Configuration
  • Security Best Practices
  • Complete Storage Configuration Example
  1. Qtap
  2. Configuration

Storage Configuration

Understanding Qtap Storage Components

Qtap captures two distinct types of data, each with different storage requirements:

  1. Events (Connection Metadata): Anonymized information about connections, including timestamps, endpoints, and performance metrics

  2. Objects (Payload Content): Actual request and response data including headers and bodies, which may contain sensitive information

Each type has its own dedicated storage configuration in the services section of your qpoint.yaml file.

Event Stores

Event stores handle anonymized metadata about network connections. This data is generally not sensitive but is useful for analytics, troubleshooting, and monitoring.

Console Output (stdout)

The simplest option for development and debugging:

event_stores:
  - id: console_stdout
    type: stdout

This configuration sends all event data to the console where Qtap is running, making it immediately visible but not persistent.

Pulse Service (Coming Soon)

For self-hosted environments with a Pulse instance:

event_stores:
  - id: pulse
    type: pulse
    endpoint: http://pulse-service:8000
    token: 
      type: env
      value: PULSE_TOKEN

This connects to a Pulse service for advanced analytics and visualization.

Object Stores

Object stores contain the actual content of requests and responses, which often includes sensitive information. This data requires more careful handling and secure storage.

Console Output (stdout)

For development and debugging:

object_stores:
  - id: console_stdout
    type: stdout

Sends all object data to the console.

S3-Compatible Storage

For secure, persistent storage:

object_stores:
  - id: s3_store
    type: s3
    endpoint: storage.example.com:9000
    bucket: qpoint-objects
    region: us-east-1
    access_url: https://storage.example.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: S3_ACCESS_KEY
    secret_key:
      type: env
      value: S3_SECRET_KEY

This configuration:

  • Stores objects in an S3-compatible storage service

  • Uses HTTPS for secure transmission (insecure: false)

  • Retrieves credentials from environment variables

  • Provides a template URL for accessing stored objects

S3 Configuration Parameters

Parameter
Description
Example

endpoint

S3 server hostname and port

minio.example.com:9000

bucket

S3 bucket name

qpoint-objects

region

S3 region name

us-east-1

access_url

URL template for object access

https://storage.example.com/{{BUCKET}}/{{DIGEST}}

insecure

Allow HTTP instead of HTTPS

false (recommended)

access_key

S3 access key configuration

See credential management

secret_key

S3 secret key configuration

See credential management

URL Template Variables

The access_url parameter supports these template variables:

  • {{ENDPOINT}}: The S3 endpoint

  • {{BUCKET}}: The bucket name

  • {{DIGEST}}: The unique file identifier

Credential Management

For security, Qtap supports retrieving credentials from environment variables rather than storing them directly in the configuration file.

Environment Variable Configuration

access_key:
  type: env
  value: S3_ACCESS_KEY  # Name of the environment variable

When running Qtap, ensure these environment variables are set:

export S3_ACCESS_KEY=your_access_key
export S3_SECRET_KEY=your_secret_key

For Docker:

docker run \
  # Other parameters...
  -e S3_ACCESS_KEY=your_access_key \
  -e S3_SECRET_KEY=your_secret_key \
  # Rest of command...

For Kubernetes, use secrets:

kubectl create secret generic s3-credentials \
  --from-literal=access-key='YOUR_ACCESS_KEY' \
  --from-literal=secret-key='YOUR_SECRET_KEY' \
  -n qpoint

And reference them in your Helm values:

extraEnv:
  - name: S3_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: access-key
  - name: S3_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: secret-key

Storage Configuration Examples

MinIO Configuration

object_stores:
  - id: minio
    type: s3
    endpoint: minio.example.com:9000
    bucket: qpoint
    region: us-east-1
    access_url: https://minio.example.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: MINIO_ACCESS_KEY
    secret_key:
      type: env
      value: MINIO_SECRET_KEY

AWS S3 Configuration

For AWS S3:

object_stores:
  - id: aws_s3
    type: s3
    endpoint: s3.amazonaws.com
    bucket: my-company-qpoint
    region: us-west-2
    access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: AWS_ACCESS_KEY_ID
    secret_key:
      type: env
      value: AWS_SECRET_ACCESS_KEY

Google Cloud Storage Configuration

For Google Cloud Storage:

object_stores:
  - id: gcs
    type: s3
    endpoint: storage.googleapis.com
    bucket: my-company-qpoint
    region: us-central1
    access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}
    insecure: false
    access_key:
      type: env
      value: GCS_ACCESS_KEY
    secret_key:
      type: env
      value: GCS_SECRET_KEY

Security Best Practices

When configuring storage, especially for production environments:

  1. Use HTTPS: Always set insecure: false to enforce encrypted connections

  2. Environment Variables: Never store credentials in the configuration file

  3. Bucket Policies: Restrict access to your storage bucket with appropriate IAM policies

  4. Encryption: Enable server-side encryption for stored objects

  5. Lifecycle Rules: Configure automatic deletion of old data to comply with retention policies

  6. Audit Logging: Enable access logging for your storage service

Complete Storage Configuration Example

version: 2

services:
  event_stores:
    - id: console_stdout
      type: stdout
  
  object_stores:
    - id: minio
      type: s3
      endpoint: minio.internal:9000
      bucket: qpoint-objects
      region: us-east-1
      access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

This configuration sends connection metadata to the console for easy monitoring while securely storing the actual request and response content in MinIO.

PreviousConfigurationNextTraffic Processing with Plugins

Last updated 24 days ago

is a popular self-hosted, S3-compatible object store:

MinIO