> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/readme/architecture-overview/architecture-qtap-yaml.md).

# Self-Managed (YAML)

This page describes the architecture for self-managed deployments using Qtap (data plane) with local YAML configuration files.

## Overview

In self-managed mode, Qtap sensors (data plane) run standalone with all configuration defined in YAML files. This architecture provides full control, supports air-gapped deployments, and enables infrastructure-as-code workflows while maintaining complete data sovereignty.

## Data Flow

**Events (Anonymized Metadata):**

```
Qtap (Data Plane) → stdout/logs → Your Log Aggregation (Fluentd, Logstash, CloudWatch, etc.)
```

**Objects (Sensitive Payloads):**

```
Qtap (Data Plane) → Your S3-Compatible Object Store
```

{% hint style="info" %}
**Enterprise Only:** Self-hosted Pulse/ClickHouse infrastructure is available for enterprise customers, enabling centralized event storage and analytics within your own environment. Contact us to find out more.
{% endhint %}

## Components

### Qtap (The Data Plane)

In self-managed mode, Qtap (data plane) operates independently without a control plane:

* **Configuration:** All settings defined in `qtap.yaml` (or `qpoint.yaml`)
* **Event Output:** Events written to stdout/logs
* **Object Storage:** Payloads sent directly to your S3-compatible store
* **Updates:** Manual agent version management
* **Deployment:** Docker, Kubernetes, or Linux binary

### Your Object Store (Required)

S3-compatible object storage for sensitive payload data:

* **Supported:** AWS S3, Google Cloud Storage, MinIO, any S3-compatible API
* **Configuration:** Defined in YAML `services.object_stores` section
* **Access:** Direct access via S3 console, CLI, or your own tools
* **Security:** Data never leaves your infrastructure

### Your Log Aggregation (Recommended)

Collect and analyze event logs from Qtap stdout:

* **Options:** Fluentd, Logstash, CloudWatch Logs, Cloud Logging, Splunk, etc.
* **Benefits:** Centralized logging, search, alerting, and analysis
* **Configuration:** Standard log collection from container stdout

### Qscan (Optional)

For sensitive data classification:

* **Deployment:** Docker container in your environment
* **Configuration:** Define `services.qscan` in YAML
* **Function:** Identifies PII, credentials, secrets in payloads

## YAML Configuration

### Basic Structure

```yaml
version: 2

services:
  event_stores:
    - id: console
      type: stdout    # Events go to stdout/logs

  object_stores:
    - id: my_s3
      type: s3
      endpoint: s3.amazonaws.com
      bucket: qpoint-data
      region: us-east-1
      access_key:
        type: env
        value: AWS_ACCESS_KEY_ID
      secret_key:
        type: env
        value: AWS_SECRET_ACCESS_KEY

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: text

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack
```

### Configuration Management

**Version Control:**

* Store YAML in Git for change tracking
* Use branches for different environments
* Code review configuration changes

**Deployment:**

* Mount YAML into Docker container
* Use ConfigMaps/Secrets in Kubernetes
* Redeploy agents to apply changes

**Documentation:**

* [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md)
* [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md)
* [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings.md)

## AWS Installation & Usage Workflow

### Data Flow

**Events:** Qtap (Data Plane) → stdout → CloudWatch Logs (or your log aggregator)

**Objects:** Qtap (Data Plane) → Your S3 Bucket

### Steps

1. **Host Your Services in AWS:**
   * Create an S3 bucket in your AWS account to serve as your **Object Store**.
   * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification.
2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your EC2 instances or EKS cluster. Ensure the environment variables for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process.
3. **Configure via YAML:** Create a `qtap.yaml` file
   * Mount the YAML file into your Qtap container or provide via `--config` flag
   * All configuration changes require editing YAML and redeploying
4. **Collect Events:** Set up log aggregation:
   * Events appear in stdout/logs where Qtap runs
   * Use existing infrastructure (Fluentd, Logstash, CloudWatch) to collect and analyze
   * No centralized dashboard available (coming in future)
5. **Access Payloads:** Retrieve objects directly from your S3 bucket using the S3 console, CLI, or your own tools.

## GCP Installation & Usage Workflow

### Data Flow

**Events:** Qtap (Data Plane) → stdout → Cloud Logging (or your log aggregator)

**Objects:** Qtap (Data Plane) → Your GCS Bucket

### Steps

1. **Host Your Services in GCP:**
   * Create a Google Cloud Storage (GCS) bucket in your GCP project to serve as your **Object Store**.
   * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification.
2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your Compute Engine VMs or GKE cluster. Ensure the environment variables for `GCS_ACCESS_KEY`, `GCS_SECRET_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process.
3. **Configure via YAML:** Create a `qtap.yaml` file
   * Mount the YAML file into your Qtap container or provide via `--config` flag
   * All configuration changes require editing YAML and redeploying
4. **Collect Events:** Set up log aggregation
   * Events appear in stdout/logs where Qtap runs
   * Use existing infrastructure (Fluentd, Logstash, Cloud Logging) to collect and analyze
   * No centralized dashboard available (coming in future)
5. **Access Payloads:** Retrieve objects directly from your GCS bucket using the GCS console, gsutil CLI, or your own tools.

***

## Limitations & Roadmap

**Current Limitations:**

* No centralized dashboard (use log aggregation tools)
* No built-in alerting (use log-based alerts)
* Manual configuration management (no auto-propagation)
* Manual agent updates

**Coming in Future Releases:**

* Self-hosted Pulse/ClickHouse infrastructure
* Centralized event storage and analytics within your environment
* Dashboard and alerting capabilities for air-gapped deployments

## Get Started

Ready to deploy with YAML configuration?

**Quick Start:**

* [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide.md) - Progressive tutorial (4 levels, 50 minutes)
* [Starter Configuration - Stdout Only](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only.md) - Simplest possible setup

**Configuration Guides:**

* [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) - Complete S3 setup and event stores
* [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins.md) - Configure stacks and plugins
* [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings.md) - Define what traffic to capture

**Back to Overview:**

* [Architecture Overview](/readme/architecture-overview.md) - Compare deployment modes


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.qpoint.io/readme/architecture-overview/architecture-qtap-yaml.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.