# Storage Configuration ## Understanding Qtap Storage Components Qtap captures two distinct types of data, each with different storage requirements: 1. [**Events**](#event-stores) (Connection Metadata): Anonymized information about connections, including timestamps, endpoints, and performance metrics 2. [**Objects**](#object-stores) (Payload Content): Actual request and response data including headers and bodies, which may contain sensitive information Each type has its own dedicated storage configuration in the `services` section of your `qpoint.yaml` file. {% hint style="info" %} Objects (Payload Content) are selectively captured based on the plugin configuration. See [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md) {% endhint %} ```yaml services: event_stores: - id: console_stdout type: stdout object_stores: - id: console_stdout type: stdout ``` ## Event Stores Event stores handle anonymized metadata about network connections. This data is generally not sensitive but is useful for analytics, troubleshooting, and monitoring. ### **Console Output (stdout)** The simplest option for development and debugging: ```yaml event_stores: - id: console_stdout type: stdout ``` This configuration sends all event data to the console where Qtap is running, making it immediately visible but not persistent. ### **Axiom** For sending events to Axiom for analytics and monitoring: ```yaml event_stores: - type: axiom dataset: type: text value: qpoint-events token: type: env value: AXIOM_TOKEN ``` This configuration sends event data to an Axiom dataset for advanced analytics and visualization. **Axiom Configuration Parameters**
ParameterDescription
datasetThe name of the Axiom dataset to send events to
tokenAxiom API token
### **OpenTelemetry (OTLP)** Send events to any OpenTelemetry-compatible backend using the OTLP protocol. Qtap exports events as **OpenTelemetry Logs** with rich structured attributes. ```yaml event_stores: - type: otel endpoint: "localhost:4317" # OTLP endpoint (required for grpc/http, not needed for stdout) protocol: grpc # grpc, http, or stdout (required) service_name: "qtap" # Service name for resource attributes (optional) environment: "production" # Environment tag (optional) headers: # Custom headers (optional) api-key: type: env value: OTEL_API_KEY tls: enabled: false # Enable TLS (optional, default: false) ``` This configuration works with any OTLP-compatible backend including: * OpenTelemetry Collector * Datadog (OTLP ingestion) * Honeycomb (OTLP API) * New Relic (OTLP endpoint) * Grafana Cloud (OTLP) * Elastic (OTLP) **OpenTelemetry Configuration Parameters**
ParameterDescription
endpointOTLP endpoint address (e.g., "localhost:4317" for gRPC, "localhost:4318" for HTTP). Required for grpc and http protocols. Not needed for stdout protocol.
protocolProtocol to use: "grpc" (default port 4317), "http" (default port 4318), or "stdout" for debugging
service_nameService name added to resource attributes (default: "qtap")
environmentEnvironment name for filtering/grouping (e.g., "production", "staging")
headersCustom headers for authentication (e.g., API keys). Each header can use type: env to load from environment variables or type: text for direct values
tls.enabledEnable TLS for secure connections (default: false)
**Event Types Sent** Qtap sends two types of OpenTelemetry log events: 1. **Connection Events** (`event.type: connection`) - TCP connection lifecycle and metadata including TLS version, protocol detection, bytes transferred 2. **Artifact Records** (`event.type: artifact_record`) - HTTP transaction summaries with method, URL, status, duration, bytes, and links to full request/response data in object stores See the [OpenTelemetry Integration Guide](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry.md) for detailed setup instructions. ### **Qpoint Pulse Service (Coming Soon)** For self-hosted environments with a Pulse instance: ```yaml event_stores: - id: pulse type: pulse endpoint: http://pulse-service:8000 token: type: env value: PULSE_TOKEN ``` This connects to a Pulse service for advanced analytics and visualization. ## Object Stores Object stores contain the actual content of requests and responses, which often includes sensitive information. This data requires more careful handling and secure storage. ### **Console Output (stdout)** For development and debugging: ```yaml object_stores: - id: console_stdout type: stdout ``` Sends all object data to the console. ### **OpenTelemetry (OTLP)** Send captured artifacts (request/response headers and bodies) through your existing OpenTelemetry pipeline. Artifacts are exported as **OpenTelemetry Logs**, flowing through the same collector as your events. ```yaml object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false ``` This pairs naturally with the OTel event store — point both at the same collector endpoint for a unified pipeline: ```yaml services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false ``` **OpenTelemetry Object Store Configuration Parameters**
ParameterDescription
otel_endpointOTLP endpoint address (e.g., "localhost:4317" for gRPC, "localhost:4318" for HTTP)
protocolProtocol to use: "grpc" (default port 4317), "http" (default port 4318), or "stdout" for debugging
service_nameService name added to resource attributes (default: "qtap")
environmentEnvironment name for filtering/grouping (e.g., "production", "staging")
headersCustom headers for authentication. Each header can use type: env to load from environment variables or type: text for direct values
tls.enabledEnable TLS for secure connections (default: false)
See the [sample ClickStack configuration](https://github.com/qpoint-io/qtap/blob/main/examples/sample-otel-objectstore-clickstack.yaml) for a complete working example, and the [OTel Object Store release notes](/release-notes/otel-object-store.md) for more details. ### **S3-Compatible Storage** For secure, persistent storage: ```yaml object_stores: - id: s3_store type: s3 endpoint: storage.example.com:9000 bucket: qpoint-objects region: us-east-1 access_url: https://storage.example.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY ``` This configuration: * Stores objects in an S3-compatible storage service * Uses HTTPS for secure transmission (`insecure: false`) * Retrieves credentials from environment variables * Provides a template URL for accessing stored objects **S3 Configuration Parameters** | Parameter | Description | Example | | ------------ | ------------------------------ | --------------------------------------------------- | | `endpoint` | S3 server hostname and port | `minio.example.com:9000` | | `bucket` | S3 bucket name | `qpoint-objects` | | `region` | S3 region name | `us-east-1` | | `access_url` | URL template for object access | `https://storage.example.com/{{BUCKET}}/{{DIGEST}}` | | `insecure` | Allow HTTP instead of HTTPS | `false` (recommended) | | `access_key` | S3 access key configuration | See credential management | | `secret_key` | S3 secret key configuration | See credential management | **URL Template Variables** The `access_url` parameter supports these template variables: * `{{ENDPOINT}}`: The S3 endpoint * `{{BUCKET}}`: The bucket name * `{{DIGEST}}`: The unique file identifier ## Credential Management For security, Qtap supports retrieving credentials from environment variables or using direct text values. **Environment Variable Configuration** ```yaml token: type: env value: AXIOM_TOKEN # Name of the environment variable ``` **Direct Text Configuration** ```yaml token: type: text value: your_actual_token # Direct token value (not recommended for production) ``` For S3 credentials, when running Qtap, ensure these environment variables are set: ```bash export S3_ACCESS_KEY=your_access_key export S3_SECRET_KEY=your_secret_key ``` For Docker: ```bash docker run \ # Other parameters... -e S3_ACCESS_KEY=your_access_key \ -e S3_SECRET_KEY=your_secret_key \ # Rest of command... ``` For Kubernetes, use secrets: ```bash kubectl create secret generic s3-credentials \ --from-literal=access-key='YOUR_ACCESS_KEY' \ --from-literal=secret-key='YOUR_SECRET_KEY' \ -n qpoint ``` And reference them in your Helm values: ```yaml extraEnv: - name: S3_ACCESS_KEY valueFrom: secretKeyRef: name: s3-credentials key: access-key - name: S3_SECRET_KEY valueFrom: secretKeyRef: name: s3-credentials key: secret-key ``` ## Object Storage Configuration Examples ### MinIO Configuration [MinIO](https://min.io/) is a popular self-hosted, S3-compatible object store: ```yaml object_stores: - id: minio type: s3 endpoint: minio.example.com:9000 bucket: qpoint region: us-east-1 access_url: https://minio.example.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` ### AWS S3 Configuration For AWS S3: ```yaml object_stores: - id: aws_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` ### Google Cloud Storage Configuration For Google Cloud Storage: ```yaml object_stores: - id: gcs type: s3 endpoint: storage.googleapis.com bucket: my-company-qpoint region: us-central1 access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: GCS_ACCESS_KEY secret_key: type: env value: GCS_SECRET_KEY ``` ## Security Best Practices When configuring storage, especially for production environments: 1. **Use HTTPS**: Always set `insecure: false` to enforce encrypted connections 2. **Environment Variables**: Never store credentials in the configuration file 3. **Bucket Policies**: Restrict access to your storage bucket with appropriate IAM policies 4. **Encryption**: Enable server-side encryption for stored objects 5. **Lifecycle Rules**: Configure automatic deletion of old data to comply with retention policies 6. **Audit Logging**: Enable access logging for your storage service ## Complete Storage Configuration Example ```yaml version: 2 services: event_stores: - id: console_stdout type: stdout object_stores: - id: minio type: s3 endpoint: minio.internal:9000 bucket: qpoint-objects region: us-east-1 access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY ``` This configuration sends connection metadata to the console for easy monitoring while securely storing the actual request and response content in MinIO. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.qpoint.io/getting-started/qtap/configuration/storage-configuration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.