Storage Configuration
Last updated
Last updated
Qtap captures two distinct types of data, each with different storage requirements:
(Connection Metadata): Anonymized information about connections, including timestamps, endpoints, and performance metrics
(Payload Content): Actual request and response data including headers and bodies, which may contain sensitive information
Each type has its own dedicated storage configuration in the services
section of your qpoint.yaml
file.
Event stores handle anonymized metadata about network connections. This data is generally not sensitive but is useful for analytics, troubleshooting, and monitoring.
The simplest option for development and debugging:
This configuration sends all event data to the console where Qtap is running, making it immediately visible but not persistent.
For sending events to Axiom for analytics and monitoring:
This configuration sends event data to an Axiom dataset for advanced analytics and visualization.
Axiom Configuration Parameters
dataset
The name of the Axiom dataset to send events to
token
Axiom API token
For self-hosted environments with a Pulse instance:
This connects to a Pulse service for advanced analytics and visualization.
Object stores contain the actual content of requests and responses, which often includes sensitive information. This data requires more careful handling and secure storage.
For development and debugging:
Sends all object data to the console.
For secure, persistent storage:
This configuration:
Stores objects in an S3-compatible storage service
Uses HTTPS for secure transmission (insecure: false
)
Retrieves credentials from environment variables
Provides a template URL for accessing stored objects
S3 Configuration Parameters
endpoint
S3 server hostname and port
minio.example.com:9000
bucket
S3 bucket name
qpoint-objects
region
S3 region name
us-east-1
access_url
URL template for object access
https://storage.example.com/{{BUCKET}}/{{DIGEST}}
insecure
Allow HTTP instead of HTTPS
false
(recommended)
access_key
S3 access key configuration
See credential management
secret_key
S3 secret key configuration
See credential management
URL Template Variables
The access_url
parameter supports these template variables:
{{ENDPOINT}}
: The S3 endpoint
{{BUCKET}}
: The bucket name
{{DIGEST}}
: The unique file identifier
For security, Qtap supports retrieving credentials from environment variables or using direct text values.
Environment Variable Configuration
Direct Text Configuration
For S3 credentials, when running Qtap, ensure these environment variables are set:
For Docker:
For Kubernetes, use secrets:
And reference them in your Helm values:
For AWS S3:
For Google Cloud Storage:
When configuring storage, especially for production environments:
Use HTTPS: Always set insecure: false
to enforce encrypted connections
Environment Variables: Never store credentials in the configuration file
Bucket Policies: Restrict access to your storage bucket with appropriate IAM policies
Encryption: Enable server-side encryption for stored objects
Lifecycle Rules: Configure automatic deletion of old data to comply with retention policies
Audit Logging: Enable access logging for your storage service
This configuration sends connection metadata to the console for easy monitoring while securely storing the actual request and response content in MinIO.
is a popular self-hosted, S3-compatible object store: