> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/readme/architecture-overview/architecture-qtap-yaml.md).

# Self-Managed (YAML)

This page describes the architecture for self-managed deployments using Qtap (data plane) with local YAML configuration files.

## Overview

In self-managed mode, Qtap sensors (data plane) run standalone with all configuration defined in YAML files. This architecture provides full control, supports air-gapped deployments, and enables infrastructure-as-code workflows while maintaining complete data sovereignty.

## Data Flow

**Events (Anonymized Metadata):**

```
Qtap (Data Plane) → stdout/logs → Your Log Aggregation (Fluentd, Logstash, CloudWatch, etc.)
```

**Objects (Sensitive Payloads):**

```
Qtap (Data Plane) → Your S3-Compatible Object Store
```

{% hint style="info" %}
**Enterprise Only:** Self-hosted Pulse/ClickHouse infrastructure is available for enterprise customers, enabling centralized event storage and analytics within your own environment. Contact us to find out more.
{% endhint %}

## Components

### Qtap (The Data Plane)

In self-managed mode, Qtap (data plane) operates independently without a control plane:

* **Configuration:** All settings defined in `qtap.yaml` (or `qpoint.yaml`)
* **Event Output:** Events written to stdout/logs
* **Object Storage:** Payloads sent directly to your S3-compatible store
* **Updates:** Manual agent version management
* **Deployment:** Docker, Kubernetes, or Linux binary

### Your Object Store (Required)

S3-compatible object storage for sensitive payload data:

* **Supported:** AWS S3, Google Cloud Storage, MinIO, any S3-compatible API
* **Configuration:** Defined in YAML `services.object_stores` section
* **Access:** Direct access via S3 console, CLI, or your own tools
* **Security:** Data never leaves your infrastructure

### Your Log Aggregation (Recommended)

Collect and analyze event logs from Qtap stdout:

* **Options:** Fluentd, Logstash, CloudWatch Logs, Cloud Logging, Splunk, etc.
* **Benefits:** Centralized logging, search, alerting, and analysis
* **Configuration:** Standard log collection from container stdout

### Qscan (Optional)

For sensitive data classification:

* **Deployment:** Docker container in your environment
* **Configuration:** Define `services.qscan` in YAML
* **Function:** Identifies PII, credentials, secrets in payloads

## YAML Configuration

### Basic Structure

```yaml
version: 2

services:
  event_stores:
    - id: console
      type: stdout    # Events go to stdout/logs

  object_stores:
    - id: my_s3
      type: s3
      endpoint: s3.amazonaws.com
      bucket: qpoint-data
      region: us-east-1
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: text

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack
```

### Configuration Management

**Version Control:**

* Store YAML in Git for change tracking
* Use branches for different environments
* Code review configuration changes

**Deployment:**

* Mount YAML into Docker container
* Use ConfigMaps/Secrets in Kubernetes
* Redeploy agents to apply changes

**Documentation:**

* [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md)
* [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md)
* [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings.md)

## AWS Installation & Usage Workflow

### Data Flow

**Events:** Qtap (Data Plane) → stdout → CloudWatch Logs (or your log aggregator)

**Objects:** Qtap (Data Plane) → Your S3 Bucket

### Steps

1. **Host Your Services in AWS:**
   * Create an S3 bucket in your AWS account to serve as your **Object Store**.
   * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification.
2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your EC2 instances or EKS cluster. Ensure the environment variables for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process.
3. **Configure via YAML:** Create a `qtap.yaml` file
   * Mount the YAML file into your Qtap container or provide via `--config` flag
   * All configuration changes require editing YAML and redeploying
4. **Collect Events:** Set up log aggregation:
   * Events appear in stdout/logs where Qtap runs
   * Use existing infrastructure (Fluentd, Logstash, CloudWatch) to collect and analyze
   * No centralized dashboard available (coming in future)
5. **Access Payloads:** Retrieve objects directly from your S3 bucket using the S3 console, CLI, or your own tools.

## GCP Installation & Usage Workflow

### Data Flow

**Events:** Qtap (Data Plane) → stdout → Cloud Logging (or your log aggregator)

**Objects:** Qtap (Data Plane) → Your GCS Bucket

### Steps

1. **Host Your Services in GCP:**
   * Create a Google Cloud Storage (GCS) bucket in your GCP project to serve as your **Object Store**.
   * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification.
2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your Compute Engine VMs or GKE cluster. Ensure the environment variables for `GCS_ACCESS_KEY`, `GCS_SECRET_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process.
3. **Configure via YAML:** Create a `qtap.yaml` file
   * Mount the YAML file into your Qtap container or provide via `--config` flag
   * All configuration changes require editing YAML and redeploying
4. **Collect Events:** Set up log aggregation
   * Events appear in stdout/logs where Qtap runs
   * Use existing infrastructure (Fluentd, Logstash, Cloud Logging) to collect and analyze
   * No centralized dashboard available (coming in future)
5. **Access Payloads:** Retrieve objects directly from your GCS bucket using the GCS console, gsutil CLI, or your own tools.

***

## Limitations & Roadmap

**Current Limitations:**

* No centralized dashboard (use log aggregation tools)
* No built-in alerting (use log-based alerts)
* Manual configuration management (no auto-propagation)
* Manual agent updates

**Coming in Future Releases:**

* Self-hosted Pulse/ClickHouse infrastructure
* Centralized event storage and analytics within your environment
* Dashboard and alerting capabilities for air-gapped deployments

## Get Started

Ready to deploy with YAML configuration?

**Quick Start:**

* [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide.md) - Progressive tutorial (4 levels, 50 minutes)
* [Starter Configuration - Stdout Only](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only.md) - Simplest possible setup

**Configuration Guides:**

* [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) - Complete S3 setup and event stores
* [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md) - Configure stacks and plugins
* [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings.md) - Define what traffic to capture

**Back to Overview:**

* [Architecture Overview](/readme/architecture-overview.md) - Compare deployment modes