Self-Managed (YAML)

This page describes the architecture for self-managed deployments using Qtap (data plane) with local YAML configuration files.

Overview

In self-managed mode, Qtap (data plane) runs standalone with all configuration defined in YAML files. This architecture provides full control, supports air-gapped deployments, and enables infrastructure-as-code workflows while maintaining complete data sovereignty.

Data Flow

Events (Anonymized Metadata):

Qtap (Data Plane) → stdout/logs → Your Log Aggregation (Fluentd, Logstash, CloudWatch, etc.)

Objects (Sensitive Payloads):

Qtap (Data Plane) → Your S3-Compatible Object Store

Coming Soon: Self-hosted Pulse/ClickHouse infrastructure will be available in a future release, enabling centralized event storage and analytics within your own environment.

Components

Qtap (The Data Plane)

In self-managed mode, Qtap (data plane) operates independently without a control plane:

  • Configuration: All settings defined in qtap.yaml (or qpoint.yaml)

  • Event Output: Events written to stdout/logs

  • Object Storage: Payloads sent directly to your S3-compatible store

  • Updates: Manual agent version management

  • Deployment: Docker, Kubernetes, or Linux binary

Your Object Store (Required)

S3-compatible object storage for sensitive payload data:

  • Supported: AWS S3, Google Cloud Storage, MinIO, any S3-compatible API

  • Configuration: Defined in YAML services.object_stores section

  • Access: Direct access via S3 console, CLI, or your own tools

  • Security: Data never leaves your infrastructure

Your Log Aggregation (Recommended)

Collect and analyze event logs from Qtap stdout:

  • Options: Fluentd, Logstash, CloudWatch Logs, Cloud Logging, Splunk, etc.

  • Benefits: Centralized logging, search, alerting, and analysis

  • Configuration: Standard log collection from container stdout

Qscan (Optional)

For sensitive data classification:

  • Deployment: Docker container in your environment

  • Configuration: Define services.qscan in YAML

  • Function: Identifies PII, credentials, secrets in payloads

YAML Configuration

Basic Structure

version: 2

services:
  event_stores:
    - id: console
      type: stdout    # Events go to stdout/logs

  object_stores:
    - id: my_s3
      type: s3
      endpoint: s3.amazonaws.com
      bucket: qpoint-data
      region: us-east-1
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: text

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack

Configuration Management

Version Control:

  • Store YAML in Git for change tracking

  • Use branches for different environments

  • Code review configuration changes

Deployment:

  • Mount YAML into Docker container

  • Use ConfigMaps/Secrets in Kubernetes

  • Redeploy agents to apply changes

Documentation:

AWS Installation & Usage Workflow

Data Flow

Events: Qtap (Data Plane) → stdout → CloudWatch Logs (or your log aggregator)

Objects: Qtap (Data Plane) → Your S3 Bucket

Steps

  1. Host Your Services in AWS:

    • Create an S3 bucket in your AWS account to serve as your Object Store.

    • (Optional) Deploy the Qscan Docker container within your VPC for sensitive data classification.

  2. Install the Data Plane: Deploy the Qtap agent (data plane) onto your EC2 instances or EKS cluster. Ensure the environment variables for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally QSCAN_TOKEN are available to the agent process.

  3. Configure via YAML: Create a qtap.yaml file

    • Mount the YAML file into your Qtap container or provide via --config flag

    • All configuration changes require editing YAML and redeploying

  4. Collect Events: Set up log aggregation:

    • Events appear in stdout/logs where Qtap runs

    • Use existing infrastructure (Fluentd, Logstash, CloudWatch) to collect and analyze

    • No centralized dashboard available (coming in future)

  5. Access Payloads: Retrieve objects directly from your S3 bucket using the S3 console, CLI, or your own tools.

GCP Installation & Usage Workflow

Data Flow

Events: Qtap (Data Plane) → stdout → Cloud Logging (or your log aggregator)

Objects: Qtap (Data Plane) → Your GCS Bucket

Steps

  1. Host Your Services in GCP:

    • Create a Google Cloud Storage (GCS) bucket in your GCP project to serve as your Object Store.

    • (Optional) Deploy the Qscan Docker container within your VPC for sensitive data classification.

  2. Install the Data Plane: Deploy the Qtap agent (data plane) onto your Compute Engine VMs or GKE cluster. Ensure the environment variables for GCS_ACCESS_KEY, GCS_SECRET_KEY, and optionally QSCAN_TOKEN are available to the agent process.

  3. Configure via YAML: Create a qtap.yaml file

    • Mount the YAML file into your Qtap container or provide via --config flag

    • All configuration changes require editing YAML and redeploying

  4. Collect Events: Set up log aggregation

    • Events appear in stdout/logs where Qtap runs

    • Use existing infrastructure (Fluentd, Logstash, Cloud Logging) to collect and analyze

    • No centralized dashboard available (coming in future)

  5. Access Payloads: Retrieve objects directly from your GCS bucket using the GCS console, gsutil CLI, or your own tools.

Limitations & Roadmap

Current Limitations:

  • No centralized dashboard (use log aggregation tools)

  • No built-in alerting (use log-based alerts)

  • Manual configuration management (no auto-propagation)

  • Manual agent updates

Coming in Future Releases:

  • Self-hosted Pulse/ClickHouse infrastructure

  • Centralized event storage and analytics within your environment

  • Dashboard and alerting capabilities for air-gapped deployments

Get Started

Ready to deploy with YAML configuration?

Quick Start:

Configuration Guides:

Back to Overview:

PreviousCloud-Managed (Qplane)NextWhy another Agent?

Last updated