# Introduction Modern enterprises face a critical visibility challenge: their applications depend on an ever-expanding web of service connections - both across team boundaries within an organization (east-west) and to external third-party services (north-south). As organizations scale, they accumulate hundreds of undocumented dependencies, each introducing potential: * Security risks from unmonitored data flows * Reliability issues from unpredictable dependencies * Operational blind spots complicating troubleshooting * Compliance gaps in sensitive data handling * Hidden costs from inefficient service usage #### Traditional Monitoring Falls Short Existing tools operate far from where connections originate, creating critical visibility gaps: * **No Process Context**: They can't identify which specific processes are making calls * **Missing Identity**: They lose essential context about service identity * **Encryption Blindness**: They only see traffic after encryption, if at all * **Limited Payload Insight**: They lack visibility into the actual data being transmitted This leads to a fundamental problem: *organizations don't truly know what's flowing between their services or why*. ### A Process-Aware Approach Qpoint takes a fundamentally different approach by operating directly at the source of each connection. Using lightweight eBPF sensors, we provide unprecedented visibility into all service traffic where it matters most – at its origin. #### Process-Level Intelligence Traditional monitoring solutions often struggle to provide detailed context about service interactions. Qpoint addresses this limitation by offering comprehensive process attribution for every connection, whether to internal or external services. Our solution identifies exactly which processes initiate connections while maintaining complete service context and identity. This extends to detailed container and pod-level information, providing a complete understanding of all service communication patterns. ### Visibility at the Source By operating where connections originate, Qpoint can: * See and analyze traffic before encryption occurs * Provide detailed payload visibility without certificate management * Maintain service identity throughout the connection lifecycle * Offer this deep visibility without any application changes ### Key Capabilities #### 1. Comprehensive Service Connection Discovery * Automatic discovery of all service dependencies (both internal and external) * Real-time inventory of connections across distributed environments * Identification of undocumented integrations and unknown endpoints #### 2. Pre-Encryption Traffic Visibility * Observes requests/responses before TLS encryption—no certificates or proxies needed * Captures headers, payload metadata, and errors for deep troubleshooting * Provides insights without breaking encryption or security models #### 3. Process-Level Attribution * Links each connection to the specific application, service, or container * Pinpoints which process is calling which service in real time * Maps dependencies across team and organizational boundaries #### 4. Reliability, Cost & Usage Analytics * Monitors SLA compliance, error rates, and performance metrics across all services * Tracks API calls by service/team to optimize usage and manage costs * Identifies performance bottlenecks and reliability issues #### 5. Security & Compliance Enforcement * Detects sensitive data in traffic to prevent unauthorized disclosures * Delivers full audit trails for internal and external connections * Enables zero-trust policies for service-to-service communication #### 6. Zero-Impact Deployment * Lightweight eBPF sensor on Linux (no kernel modifications, minimal overhead) * No architectural changes, no proxy re-routing, no code instrumentation required * Seamless integration with existing monitoring and security tools {% hint style="info" %} **Why eBPF is safe**: Every eBPF program is verified by the Linux kernel before execution, ensuring it cannot crash your systems or access unauthorized memory. eBPF is fundamentally safer than traditional kernel modules and is trusted in production by major financial institutions and all three major cloud providers. [Learn more about eBPF safety](/readme/ebpf-safety-and-trust). {% endhint %} ### Why This Matters This process-level visibility fundamentally transforms how organizations understand and manage their service dependencies: * **Development Teams**: See exactly how your services interact with other teams' services and external APIs * **Platform Teams**: Map and understand the true connection landscape across your entire organization * **Security Teams**: Track which applications are communicating with what, both internally and externally * **Operations Teams**: Attribute performance issues and costs to specific services and processes * **Compliance Teams**: Trace sensitive data flows to their source across all service boundaries ### Qpoint's Unique Position Qpoint serves as a vital complement to existing infrastructure by adding a critical layer of process-aware visibility that was previously unattainable. By operating at the connection source, we occupy a unique position where we can: * See the actual data before encryption * Identify the specific process making each call * Maintain full service context across all boundaries * Provide this without requiring certificate management or application modifications ### Not Just Another Security or Monitoring Tool Qpoint isn't: * A replacement for your perimeter firewall * A traditional network monitoring solution * An APM tool focused solely on application performance * A service mesh requiring extensive architecture changes Instead, Qpoint provides surgical visibility at the source of your traffic, enabling teams to understand all service interactions with unprecedented clarity and context - whether those connections cross team boundaries within your organization or extend to external third-party services. ### Technical Differentiators * **No-Code Integration**: Eliminates the need to instrument or modify applications * **Process Awareness**: Goes beyond IP-level monitoring to attribute every service call to its originating process * **Pre-Encryption Access**: Maintains full TLS security while enabling deep packet visibility * **Boundary-Crossing Visibility**: Tracks connections across all organizational and external boundaries * **Low Overhead**: Kernel-level eBPF technology ensures minimal performance impact *** ## Getting Started Ready to gain visibility into your service connections? Choose the path that matches your situation: ### Choose Your Path **First time exploring Qpoint?** Start with the concepts to understand how eBPF-based visibility works: * [How It Works](/readme/how-it-works) - Technical deep dive into eBPF and TLS visibility (10 min read) * [Architecture Overview](/readme/architecture-overview) - Understand the complete system architecture (5 min read) * [Qplane vs Qtap: Choosing Your Deployment](/readme/choosing-your-deployment) - Decide between cloud-managed and self-managed Then choose your deployment path below based on your needs. *** **Need to debug a production issue RIGHT NOW?** [**Production Debugging with HTTPS Visibility →**](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) 30-second to 5-minute setup for urgent troubleshooting. Get immediate visibility without permanent installation. **Use this when you need to:** * Debug authentication failures (401/403 errors) immediately * Investigate API integration issues in production * See inside HTTPS traffic without certificates or proxies * No permanent installation or configuration required *** **Want a managed solution with centralized dashboards?** [**POC Kick Off Guide (Qplane) →**](/guides/qplane-guides/poc-kick-off-guide) Get from zero to insights in 10 minutes with Qplane's cloud control plane. **Choose Qplane if you want:** * Centralized dashboard for multi-environment visibility * Automatic configuration propagation across agents * Team collaboration and role-based access * Managed control plane (no infrastructure to maintain) *** **Need full control with self-managed configuration?** [**Complete Guide: From Hello World to Production (Qtap) →**](/guides/qtap-guides/getting-started/getting-started-complete-guide) Progressive 4-level tutorial (50 minutes total) for standalone Qtap deployment with YAML configuration. **Choose Qtap if you want:** * Full control via version-controlled YAML files * Air-gapped or isolated environments * Data sovereignty (all data stays in your infrastructure) * Maximum flexibility and customization **Alternative Qtap quick starts:** * [5-Minute Quickstart](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) - Fast preview with stdout logging * [All Qtap Guides](https://github.com/qpoint-io/documentation/blob/main/guides/getting-started/qtap-guides.md) - Browse all tutorials and integration guides *** ### Not Sure Which Path to Choose? | Your Requirements | Recommended Path | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | "I need visibility NOW for a production issue" | [Production Debugging](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) (30 seconds) | | "I want to try Qpoint with minimal setup" | [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) (Qplane, 10 min) | | "I need data to stay in my infrastructure" | [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) (Qtap, 50 min) | | "I'm evaluating for enterprise deployment" | Start with [Choosing Your Deployment](/readme/choosing-your-deployment), then [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) | | "I want to understand the technology first" | [How It Works](/readme/how-it-works) → [Architecture](/readme/architecture-overview) → Choose path above | *** ### What's the Difference Between Qplane and Qtap? **Qtap** is the lightweight eBPF sensor that captures traffic on your hosts. It can run standalone (configured via YAML) or connected to Qplane. **Qplane** is the cloud-managed control plane that provides centralized configuration, dashboards, and multi-environment management for your Qtap agents. **Learn more:** [Choosing Your Deployment: Qplane vs Qtap](/readme/choosing-your-deployment) # How It Works Qtap is Qpoint's host-based eBPF sensor that provides deep visibility into application traffic through various kernel and library integration points. The sensor employs multiple complementary approaches to gather comprehensive data about application traffic, with all connections visible through base socket layer monitoring and extended payload visibility available through either native TLS integration or egress controller mode. ## Connections All outbound connections are visible through Qtap's eBPF-based socket layer integration, regardless of the runtime environment or TLS implementation. **What We Capture:** * Connection establishment and teardown * Source and destination information * Bandwidth usage * Protocol details * Process attribution **How It Works:** * eBPF programs attach to kernel socket functions * Captures events for TCP/IP operations * Provides real-time visibility into network flows * Associates connections with process context This base layer ensures complete visibility into all network connections, forming the foundation for Qpoint's observability capabilities.

An example of Socket Layer Connection Attribution

## Payloads Beyond connection visibility, Qtap can capture payload content through two different mechanisms: ### **Native TLS Integration** Qtap provides automatic payload visibility for applications using supported TLS libraries, requiring zero configuration. The process works by using eBPF uprobes to attach to specific SSL/TLS functions in the library, allowing us to capture data before encryption (for outbound traffic) or after decryption (for inbound traffic). **How It Works:** 1. When a program starts, Qtap scans its ELF (Executable and Linkable Format) file 2. Qtap identifies and locates SSL/TLS-related functions in memory 3. eBPF uprobes are attached to these functions (e.g., SSL\_read, SSL\_write) 4. When the functions are called, Qtap captures the raw data buffer 5. The captured data is processed locally on the host **Supported Libraries:** * OpenSSL * GoTLS * NodeTLS * Java 11 - 21 ### **Egress Controller** For runtimes where native TLS integration isn't possible or preferred, Qtap provides an egress controller mode: **How It Works:** * Local HTTP(S) proxy on the host * Transparent traffic redirection * Local TLS termination and re-encryption **Configuration:** ```bash # Required Environment Variables QPOINT_STRATEGY=forward # Enables egress controller mode TRUST_STORE=/path/to/certs # Certificate store location ``` **Common Use Cases:** * Custom TLS implementations * Legacy runtimes ## Runtime Support Matrix | Runtime | Base Visibility | Payload Visibility | Configuration Needed | | -------------- | --------------- | ------------------ | -------------------- | | OpenSSL | Yes | Native TLS | None | | GoTLS | Yes | Native TLS | None | | NodeTLS | Yes | Native TLS | None | | Java | Yes | Native TLS | None | | Custom TLS | Yes | Egress Controller | Environment Vars | | Other Runtimes | Yes | Egress Controller | Environment Vars | *** ## Try It Yourself Ready to see this in action? **Quick Start (Cloud Management):** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Deploy your first eBPF sensor and see TLS visibility in 10 minutes **Self-Managed (YAML):** * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial from basics to production (4 levels, 50 minutes) **Deep Dive:** * [Architecture Overview](/readme/architecture-overview) - Understand the complete system architecture * [eBPF Concepts](/readme/ebpf-concepts) - Learn about the underlying eBPF technology # Architecture Overview ## What is Qpoint? Qpoint provides eBPF-based traffic visibility for observability, security, and compliance. At its core, Qpoint consists of: * **Qtap (Data Plane)**: A lightweight eBPF sensor that captures network traffic * **Optional Control Plane**: Choose between cloud-managed (Qplane at app.qpoint.io) or self-managed (YAML) deployment ## Guiding Principles Qpoint is designed with two core principles in mind: 1. **Deep Visibility:** To provide a complete, unencrypted view of your application traffic, including payloads, with full context (process, container, host, user, etc.). 2. **Data Sovereignty:** To ensure that your sensitive data, the actual content of your traffic, **never leaves your environment**. You retain full control and ownership of your most critical data at all times. ## Core Components ### Qtap (The Data Plane) **Qtap** is a lightweight, high-performance eBPF sensor that you install on your Linux infrastructure. It is the data plane that captures and processes traffic directly from the Linux kernel. * **How it Works:** By attaching to low-level TLS/SSL functions within the kernel, Qtap intercepts data *before* it's encrypted and *after* it's decrypted. This provides access to the original, plain-text data without needing to manage certificates, modify applications, or install invasive proxies. * **Out-of-Band Operation:** Qtap operates out-of-band, meaning it observes traffic without being in the critical path. This ensures it adds no latency and cannot disrupt application performance. * **Rich Context:** It enriches the captured data with a wealth of context, including the associated process, container, host, user, and protocol information. * **Flexible Deployment:** Can run standalone with YAML configuration or connect to Qplane for centralized management. ### Qscan (Data Classification) **Qscan** is an optional service, typically run as a Docker container, that is responsible for deep inspection and classification of your data payloads. * **Function:** Qtap can be configured to send data to Qscan, which then identifies specific types of sensitive information (PII, credentials, secrets, etc.) based on predefined or custom rules. * **Deployment:** Generally you host Qscan within your own environment. This ensures that the process of scanning and classifying your sensitive data happens securely under your control. ### Object Storage (Your Data Warehouse) You provide your own S3-compatible object store (e.g., AWS S3, MinIO, Google Cloud Storage) where sensitive payload data is stored. * **Function:** The Qtap agent sends **Objects** (request/response headers and bodies) directly to this storage bucket. * **Security:** This data flow is direct from the agent in your environment to the object store in your environment. **This sensitive data never traverses any Qpoint-managed systems** (when using your own object store). ## Data Types: The Golden Rule Qpoint distinguishes between two types of data to ensure security and privacy: * **Events (Anonymized Metadata):** High-level, non-sensitive information about a connection (e.g., source/destination IP, process name, status codes, timing). Where these go depends on your deployment mode. * **Objects (Sensitive Payloads):** The actual content of your traffic (e.g., API request/response bodies, headers). These **always** go to your S3-compatible object store (or optionally Qplane's managed store for preview/testing). ## Choose Your Deployment Mode Qpoint offers two deployment approaches with different tradeoffs: | Feature | Cloud-Managed (Qplane) | Self-Managed (YAML) | | ------------------ | ------------------------- | ------------------------- | | **Event Storage** | app.qpoint.io | stdout/logs only\* | | **Object Storage** | Your S3 or Qplane managed | Your S3 required | | **Dashboards** | Yes | No | | **Alerting** | Yes | No | | **Configuration** | Web UI + API\*\* | YAML files | | **Agent Updates** | Manual | Manual | | **Best For** | Teams, multi-environment | Air-gapped, IaC workflows | \*Self-hosted Pulse/ClickHouse infrastructure support coming in future release. \*\* API coming soon ### Cloud-Managed (with Qplane) Connect QTap sensors (data plane) to QPlane (control plane at app.qpoint.io) for centralized management and analytics. **When to Use:** * Multi-environment deployments * Teams needing shared visibility * Organizations wanting managed analytics infrastructure **Learn More:** [Cloud-Managed Architecture Details](/readme/architecture-overview/architecture-qplane) ### Self-Managed (YAML Only) Run Qtap standalone with local YAML configuration files. **When to Use:** * Strict data residency requirements * Air-gapped or isolated environments * Infrastructure-as-code workflows * Single-server or simple deployments **Learn More:** [Self-Managed Architecture Details](/readme/architecture-overview/architecture-qtap-yaml) *** ## Quick Start Ready to get started? Choose your path: **Cloud-Managed:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Deploy with Qplane in 10 minutes **Self-Managed:** * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive YAML tutorial (4 levels, 50 minutes) ## Related Documentation * [How It Works](/readme/how-it-works) - eBPF and TLS visibility technology * [Use Cases](/readme/use-cases) - Real-world applications of this architecture # Cloud-Managed (Qplane) This page describes the architecture for cloud-managed deployments using Qplane (control plane at app.qpoint.io) with Qtap sensors (data plane). ## Overview In cloud-managed mode, Qtap sensors (data plane) connect to Qplane (control plane) for centralized configuration management and analytics. This architecture provides real-time dashboards, alerting, and team collaboration features while maintaining data sovereignty for sensitive payloads. {% hint style="info" %} **Enterprise Only:** Self-hosted Pulse/ClickHouse infrastructure is available for enterprise customers, enabling centralized event storage and analytics within your own environment. Contact us to find out more. {% endhint %} ## Data Flow **Events (Anonymized Metadata):** ``` Qtap (Data Plane) → Pulse API Gateway → ClickHouse Database → Qplane Dashboards (Control Plane) ``` **Objects (Sensitive Payloads):** ``` Qtap (Data Plane) → Your S3 Bucket (or Qplane managed store for POC/testing) ```

Reference Architecture

## Components ### Qplane (The Control Plane) **Qplane** is the centralized control plane for cloud-managed deployments, hosted by Qpoint at `app.qpoint.io`. * **Function:** This is where you configure your Qtap agents (data plane), define rules, view dashboards, and analyze the metadata collected from your services. * **Event Processing:** Qtap agents (data plane) send anonymized event metadata to the Pulse API gateway (`api-pulse.qpoint.io`) for authentication and ingestion. Events are then stored in ClickHouse, powering Qplane's dashboards, analytics, and alerting features. * **Security:** Qplane only receives and processes anonymized **Event metadata** (connection info, status codes, timing). When configured properly, it never has access to your sensitive **Object payloads** (request/response bodies). * **Features:** * Real-time dashboards and service dependency maps * Alerting and notifications (Slack, PagerDuty, webhooks) * Team collaboration and RBAC * Visual configuration management * Automatic agent configuration propagation ### Pulse (Event Gateway) Pulse is Qpoint's API gateway that handles event ingestion from the data plane: * **Authentication:** Validates registration tokens from Qtap agents (data plane) * **Ingestion:** Receives anonymized event metadata from data plane * **Routing:** Forwards events to ClickHouse for storage and analysis * **Endpoint:** `api-pulse.qpoint.io` ### ClickHouse (Event Database) ClickHouse is the analytics database that stores event metadata: * **Purpose:** Powers Qplane's dashboards, traffic analysis, and alerting * **Data:** Contains only anonymized event metadata (no sensitive payloads) * **Performance:** Optimized for real-time analytics on high-volume event streams ### Your Object Store Even in cloud-managed mode, sensitive payloads should be stored in your infrastructure: * **Recommended:** AWS S3, Google Cloud Storage, or MinIO in your environment * **Alternative:** Qplane's managed object store (preview/testing only) * **Access:** When viewing payloads in Qplane UI, your browser fetches directly from your S3 bucket using signed URLs ## AWS Installation & Usage Workflow ### Data Flow **Events:** Qtap (Data Plane) → Pulse → ClickHouse → Qplane Dashboards (Control Plane) **Objects:** Qtap (Data Plane) → Your S3 Bucket ### Steps 1. **Host Your Services in AWS:** * Create an S3 bucket in your AWS account to serve as your **Object Store**. * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification. 2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your EC2 instances or EKS cluster with a registration token from Qplane (control plane). Ensure the environment variables for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process. 3. **Configure via Qplane:** Log in to `app.qpoint.io`: * Configure your S3 bucket and optional Qscan endpoint in Settings → Deploy → Services * Set up stacks and plugins in Plugins → Stacks * Define traffic capture rules in Settings → Qtap * Configuration automatically propagates to all connected agents 4. **Visualize & Analyze:** View real-time dashboards showing: * Anonymized event data flowing through Pulse to ClickHouse * Service dependencies and traffic patterns * Alerts and anomalies 5. **Access Payloads Securely:** When you need to inspect the full payload of a request from the Qplane UI, your browser will be given a URL to retrieve it directly from *your* S3 bucket. This maintains the security boundary, as Qpoint's servers never access the payload data. ## GCP Installation & Usage Workflow ### Data Flow **Events:** Qtap (Data Plane) → Pulse → ClickHouse → Qplane Dashboards (Control Plane) **Objects:** Qtap (Data Plane) → Your GCS Bucket ### Steps 1. **Host Your Services in GCP:** * Create a Google Cloud Storage (GCS) bucket in your GCP project to serve as your **Object Store**. * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification. 2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your Compute Engine VMs or GKE cluster with a registration token from Qplane (control plane). Ensure the environment variables for `GCS_ACCESS_KEY`, `GCS_SECRET_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process (e.g., via metadata, secrets, or environment configuration). 3. **Configure via Qplane:** Log in to `app.qpoint.io`: * Configure your GCS bucket and optional Qscan endpoint in Settings → Deploy → Services * Set up stacks and plugins in Plugins → Stacks * Define traffic capture rules in Settings → Qtap * Configuration automatically propagates to all connected agents 4. **Visualize & Analyze:** View real-time dashboards showing: * Anonymized event data flowing through Pulse to ClickHouse * Service dependencies and traffic patterns * Alerts and anomalies 5. **Access Payloads Securely:** When you need to inspect the full payload of a request from the Qplane UI, your browser will be given a URL to retrieve it directly from your GCS bucket. This maintains the security boundary, as Qpoint's servers never access the payload data. *** ## Get Started Ready to deploy with Qplane? **Quick Start:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Complete setup in 10 minutes **Configuration Guides:** * [Settings (Qplane)](/getting-started/qplane/configuration/settings) - Configure storage services in cloud control plane * [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) - Set up traffic processing pipelines * [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) - Deep dive into Qplane architecture **Back to Overview:** * [Architecture Overview](/readme/architecture-overview) - Compare deployment modes # Self-Managed (YAML) This page describes the architecture for self-managed deployments using Qtap (data plane) with local YAML configuration files. ## Overview In self-managed mode, Qtap sensors (data plane) run standalone with all configuration defined in YAML files. This architecture provides full control, supports air-gapped deployments, and enables infrastructure-as-code workflows while maintaining complete data sovereignty. ## Data Flow **Events (Anonymized Metadata):** ``` Qtap (Data Plane) → stdout/logs → Your Log Aggregation (Fluentd, Logstash, CloudWatch, etc.) ``` **Objects (Sensitive Payloads):** ``` Qtap (Data Plane) → Your S3-Compatible Object Store ``` {% hint style="info" %} **Enterprise Only:** Self-hosted Pulse/ClickHouse infrastructure is available for enterprise customers, enabling centralized event storage and analytics within your own environment. Contact us to find out more. {% endhint %} ## Components ### Qtap (The Data Plane) In self-managed mode, Qtap (data plane) operates independently without a control plane: * **Configuration:** All settings defined in `qtap.yaml` (or `qpoint.yaml`) * **Event Output:** Events written to stdout/logs * **Object Storage:** Payloads sent directly to your S3-compatible store * **Updates:** Manual agent version management * **Deployment:** Docker, Kubernetes, or Linux binary ### Your Object Store (Required) S3-compatible object storage for sensitive payload data: * **Supported:** AWS S3, Google Cloud Storage, MinIO, any S3-compatible API * **Configuration:** Defined in YAML `services.object_stores` section * **Access:** Direct access via S3 console, CLI, or your own tools * **Security:** Data never leaves your infrastructure ### Your Log Aggregation (Recommended) Collect and analyze event logs from Qtap stdout: * **Options:** Fluentd, Logstash, CloudWatch Logs, Cloud Logging, Splunk, etc. * **Benefits:** Centralized logging, search, alerting, and analysis * **Configuration:** Standard log collection from container stdout ### Qscan (Optional) For sensitive data classification: * **Deployment:** Docker container in your environment * **Configuration:** Define `services.qscan` in YAML * **Function:** Identifies PII, credentials, secrets in payloads ## YAML Configuration ### Basic Structure ```yaml version: 2 services: event_stores: - id: console type: stdout # Events go to stdout/logs object_stores: - id: my_s3 type: s3 endpoint: s3.amazonaws.com bucket: qpoint-data region: us-east-1 access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY stacks: default_stack: plugins: - type: http_capture config: level: summary format: text tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: default_stack ``` ### Configuration Management **Version Control:** * Store YAML in Git for change tracking * Use branches for different environments * Code review configuration changes **Deployment:** * Mount YAML into Docker container * Use ConfigMaps/Secrets in Kubernetes * Redeploy agents to apply changes **Documentation:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) ## AWS Installation & Usage Workflow ### Data Flow **Events:** Qtap (Data Plane) → stdout → CloudWatch Logs (or your log aggregator) **Objects:** Qtap (Data Plane) → Your S3 Bucket ### Steps 1. **Host Your Services in AWS:** * Create an S3 bucket in your AWS account to serve as your **Object Store**. * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification. 2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your EC2 instances or EKS cluster. Ensure the environment variables for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process. 3. **Configure via YAML:** Create a `qtap.yaml` file * Mount the YAML file into your Qtap container or provide via `--config` flag * All configuration changes require editing YAML and redeploying 4. **Collect Events:** Set up log aggregation: * Events appear in stdout/logs where Qtap runs * Use existing infrastructure (Fluentd, Logstash, CloudWatch) to collect and analyze * No centralized dashboard available (coming in future) 5. **Access Payloads:** Retrieve objects directly from your S3 bucket using the S3 console, CLI, or your own tools. ## GCP Installation & Usage Workflow ### Data Flow **Events:** Qtap (Data Plane) → stdout → Cloud Logging (or your log aggregator) **Objects:** Qtap (Data Plane) → Your GCS Bucket ### Steps 1. **Host Your Services in GCP:** * Create a Google Cloud Storage (GCS) bucket in your GCP project to serve as your **Object Store**. * (Optional) Deploy the **Qscan** Docker container within your VPC for sensitive data classification. 2. **Install the Data Plane:** Deploy the **Qtap** agent (data plane) onto your Compute Engine VMs or GKE cluster. Ensure the environment variables for `GCS_ACCESS_KEY`, `GCS_SECRET_KEY`, and optionally `QSCAN_TOKEN` are available to the agent process. 3. **Configure via YAML:** Create a `qtap.yaml` file * Mount the YAML file into your Qtap container or provide via `--config` flag * All configuration changes require editing YAML and redeploying 4. **Collect Events:** Set up log aggregation * Events appear in stdout/logs where Qtap runs * Use existing infrastructure (Fluentd, Logstash, Cloud Logging) to collect and analyze * No centralized dashboard available (coming in future) 5. **Access Payloads:** Retrieve objects directly from your GCS bucket using the GCS console, gsutil CLI, or your own tools. *** ## Limitations & Roadmap **Current Limitations:** * No centralized dashboard (use log aggregation tools) * No built-in alerting (use log-based alerts) * Manual configuration management (no auto-propagation) * Manual agent updates **Coming in Future Releases:** * Self-hosted Pulse/ClickHouse infrastructure * Centralized event storage and analytics within your environment * Dashboard and alerting capabilities for air-gapped deployments ## Get Started Ready to deploy with YAML configuration? **Quick Start:** * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial (4 levels, 50 minutes) * [Starter Configuration - Stdout Only](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) - Simplest possible setup **Configuration Guides:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - Complete S3 setup and event stores * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - Configure stacks and plugins * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Define what traffic to capture **Back to Overview:** * [Architecture Overview](/readme/architecture-overview) - Compare deployment modes # Choosing Your Deployment: Qplane vs Qtap When deploying Qpoint, you have two main options: **Qplane** (cloud-managed control plane) or **Qtap** (self-managed with YAML configuration). This guide helps you decide which deployment model fits your requirements. ## Quick Decision Matrix | Factor | Qplane (Cloud-Managed) | Qtap (Self-Managed) | | -------------------------- | ------------------------------ | ------------------------------------ | | **Setup Time** | 10 minutes | 5-50 minutes (depends on complexity) | | **Management** | Centralized dashboard | YAML configuration files | | **Infrastructure** | Managed control plane | No control plane needed | | **Data Storage** | Your choice (can use S3) | Your choice (can use S3) | | **Configuration** | Web UI | YAML files (version controlled) | | **Multi-Environment** | Built-in support | Manual coordination | | **Air-Gapped Deployments** | Not supported | Fully supported | | **Team Collaboration** | Built-in RBAC | Via GitOps workflows | | **Best For** | Teams wanting managed solution | Teams needing full control | ## Understanding the Components ### Qtap: The eBPF Agent **Qtap** is the core eBPF agent that runs on your Linux hosts (bare metal, VMs, or containers). It: * Captures network traffic at the kernel level using eBPF * Provides process-aware visibility (knows which process made each request) * Sees inside TLS/HTTPS before encryption happens * Can run standalone or connected to Qplane **Every deployment needs Qtap agents** - it's the component that actually captures traffic. ### Qplane: The Control Plane (Optional) **Qplane** is an optional cloud-managed control plane that provides: * Centralized configuration management * Dashboard for visualization and exploration * Multi-environment orchestration * Team collaboration features **Qplane is optional** - Qtap agents work perfectly without it, configured via YAML files. ## Choose Qplane If You Want: ### 1. Centralized Management **Problem Qtap Alone:** Managing YAML config files across dozens of hosts/environments **Qplane Solution:** Single web UI to configure all agents across all environments ### 2. Quick POC or Evaluation **Problem Qtap Alone:** Requires understanding YAML structure and options **Qplane Solution:** 10-minute guided setup with dashboard for immediate insights ### 3. Multi-Environment Visibility **Problem Qtap Alone:** Each environment logs separately; correlating is manual **Qplane Solution:** Unified view across dev, staging, prod with environment filtering ### 4. Team Collaboration **Problem Qtap Alone:** Sharing YAML configs and log outputs via Slack/email **Qplane Solution:** Role-based access, shared dashboards, built-in collaboration ### 5. Dynamic Configuration **Problem Qtap Alone:** Config changes require updating YAML + agent restart **Qplane Solution:** Push config changes to agents dynamically via web UI ## Choose Qtap (Standalone) If You Want: ### 1. Air-Gapped or Isolated Environments **Benefit:** No external connectivity required; agents run completely offline **Use Case:** Government, finance, healthcare with strict network isolation requirements Qtap operates entirely within your infrastructure without any outbound connections. Perfect for environments where internet access is restricted or prohibited for security/compliance reasons. ### 2. GitOps Workflows **Benefit:** Configuration is version-controlled code, with CI/CD and rollback **Use Case:** Platform teams managing infrastructure as code YAML configurations integrate seamlessly with Git-based workflows. Track changes, review via pull requests, automate deployments with CI/CD pipelines, and roll back configurations with `git revert`. Configuration becomes auditable infrastructure-as-code. ### 3. Building Block for Custom Observability Solutions **Benefit:** Use Qtap as a data plane in your own observability architecture **Use Case:** Teams building custom monitoring, security, or compliance platforms Qtap serves as a powerful data collection layer that feeds into your own systems. Route captured traffic to your data lake, SIEM, custom analytics platform, or internal dashboards. Full control over how traffic data is processed, stored, and analyzed. **Examples:** * Feed Qtap output to your existing log aggregation pipeline (Fluent Bit, Logstash) * Build custom analytics on top of Qtap's structured JSON output * Integrate with internal security tools and compliance systems * Create domain-specific observability tailored to your business needs ## Migration Between Deployment Models ### Start with Qtap → Add Qplane Later **When:** Self-managed becomes complex as you scale to many environments **Process:** 1. Sign up for Qplane 2. Connect existing Qtap agents to Qplane (change agent connection settings) 3. Reconfigure stacks and plugins in Qplane UI 4. Gradually migrate management to Qplane UI **Note:** You'll need to manually recreate your Qtap YAML configurations in the Qplane UI, as automatic import is not yet supported. ### Start with Qplane → Move to Qtap **When:** Production requirements change (e.g., need air-gapped deployment) **Process:** 1. Export YAML configuration snapshot from Qplane dashboard 2. Deploy Qtap agents with the exported YAML config 3. Disconnect agents from Qplane **Note:** Qplane provides YAML configuration snapshots that can be used directly with standalone Qtap agents. This makes migration straightforward - just export and deploy. ## Common Misconceptions ### "Qplane stores my traffic data" **False.** Qplane only stores: * Agent configuration * Connection metadata (which process called which endpoint) * Dashboard state and queries **Sensitive HTTP/HTTPS payload data** is stored where YOU configure (typically your S3 bucket), whether using Qplane or standalone Qtap. ### "Qtap standalone can't scale" **False.** Qtap scales horizontally to thousands of agents. The difference is **how you manage them**: * Qplane: Centralized web UI * Qtap: Configuration management tools (Ansible, Terraform, Kubernetes ConfigMaps) ### "I have to choose one permanently" **False.** You can switch between deployment models as your needs change: * **Qplane → Qtap:** Export YAML snapshots from Qplane and use directly with standalone agents * **Qtap → Qplane:** Connect agents to Qplane and manually recreate configurations in the UI Migration from Qplane to Qtap is straightforward with configuration export. Migration from Qtap to Qplane requires manual reconfiguration (import not yet supported). ## Real-World Examples ### Startup Evaluating Qpoint **Situation:** 10-person engineering team, want to try Qpoint **Recommendation:** **Qplane** - Get running in 10 minutes, explore with dashboard, decide later **Path:** [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) ### Enterprise with Compliance Requirements **Situation:** Healthcare company, HIPAA compliance, data cannot leave infrastructure **Recommendation:** **Qtap standalone** with S3 storage in your VPC **Path:** [Complete Guide: Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) ### Platform Team Managing 100+ Microservices **Situation:** Large org, many environments (dev/staging/prod), multiple teams **Recommendation:** **Qplane** for centralized visibility and team collaboration **Path:** [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) ### Security Team for Incident Response **Situation:** Need to capture traffic during security incident, ephemeral setup **Recommendation:** **Qtap standalone** (minimal setup, no account needed) **Path:** [Production Debugging](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) ## Next Steps **Ready to get started?** * **Qplane (Cloud-Managed):** [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) * **Qtap (Self-Managed):** [Complete Guide: Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) * **Just Exploring:** [Production Debugging](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) (30-second setup, no commitment) **Still have questions?** * [Architecture Overview](/readme/architecture-overview) - Understand how the components fit together * [How It Works](/readme/how-it-works) - Technical deep dive into eBPF and TLS visibility * [Use Cases](/readme/use-cases) - See how other teams use Qpoint # Why another Agent? Qpoint operates directly on your Linux application servers through our Qtap eBPF sensor. This host-based approach is fundamental to providing the deep visibility and process-level traffic attribution that makes Qpoint powerful. ## The Power of Being Local Operating directly on the host gives us several key advantages: * **Process Attribution**: We can see exactly which processes are making external calls * **Pre-Encryption Visibility**: Access to traffic before TLS encryption occurs * **Container and Pod Context**: Direct access to container runtime information * **Efficient Operation**: Minimal overhead by operating at the source * **No Network Changes**: No need to redirect traffic or modify network architecture ## Technical Foundation ### eBPF Technology At the heart of Qtap is eBPF (Extended Berkeley Packet Filter), a powerful Linux kernel technology that allows us to: * Execute programs safely within kernel space * Attach to specific kernel functions and events * Access network events and socket operations * Collect detailed process and connection information ### Kernel Integration Points Qtap taps into several key areas of the Linux kernel: 1. **Network Stack** * Socket operations * TCP/IP protocol events 2. **Process Context** * Process creation and execution * Container runtime details 3. **TLS/SSL Libraries** * OpenSSL function calls * GoTLS operations * NodeTLS interactions * Java 11 - 21 ## Linux-Centric Design Qtap is specifically designed for Linux environments because: * eBPF is a Linux kernel technology * Most cloud-native workloads run on Linux * Linux provides rich kernel instrumentation capabilities * Strong container and orchestration support ## Operating Modes Qtap can operate in two primary modes: 1. **Native TLS Introspection** * Direct integration with SSL/TLS libraries * Works with OpenSSL, GoTLS, and NodeTLS * No configuration changes needed 2. **Egress Controller** * Support for other runtimes * Simple environment variable configuration * Local certificate injection # eBPF Concepts ### What is eBPF? **eBPF (Extended Berkeley Packet Filter)** is a technology in the Linux kernel that allows safe and efficient execution of user-defined programs within the kernel space. Originally designed for packet filtering, eBPF has evolved into a powerful tool for observing and modifying system behavior without changing kernel source code or adding additional modules. * **Safe Execution**: eBPF programs are analyzed by the Linux kernel's built-in verifier before execution. The verifier mathematically proves that programs cannot crash the kernel, access unauthorized memory, or loop infinitely. This makes eBPF fundamentally safer than traditional kernel modules, which have unrestricted access and have historically caused system crashes. See [eBPF Safety & Trust](/readme/ebpf-safety-and-trust) for details on enterprise adoption and third-party audits. * **Performance**: Runs in kernel space, reducing the overhead of context switches between user space and kernel space. * **Flexibility**: Can attach to various hook points in the kernel, including networking, security, and tracing. ### How eBPF Works * **eBPF Virtual Machine**: The Linux kernel includes an in-kernel VM that executes bytecode loaded from user space. * **Program Loading**: eBPF programs are loaded into the kernel via system calls and attached to specific hooks or trace points. * **Kernel Hooks**: eBPF can attach to socket events, trace points, kprobes (kernel functions), and uprobes (user-space functions). * **Maps and Data Sharing**: eBPF programs use maps to store and share data between the kernel and user space. ### Traditional Use Cases for eBPF * **Networking**: Packet filtering, load balancing, and network performance monitoring. * **Observability**: System tracing, logging, and performance profiling. * **Security**: Detecting anomalies, enforcing security policies, and auditing. *** ## eBPF in Qpoint ### Why Qpoint Uses eBPF Qpoint leverages eBPF to achieve deep visibility into network traffic without the need for intrusive instrumentation or code changes. * **Kernel-Level Visibility**: Access to network events at the kernel level before encryption, providing insights into unencrypted traffic. * **Minimal Overhead**: eBPF's efficient execution minimizes performance impact on applications. * **No Code Changes Required**: Eliminates the need to modify application code. ### How Qpoint Leverages eBPF * **The eBPF Sensor**: Qtap uses eBPF programs to monitor network sockets and collect metadata and payloads from network traffic. * **Attaching to Events**: Qtap attaches eBPF probes to specific kernel events related to network communication. * **Data Collection**: Collects detailed information such as process IDs, IP addresses, ports, payloads, and timing information. ### Benefits of eBPF in Qpoint * **Transparency**: Operates transparently to applications and services. * **Security**: Data remains within your environment; no data is sent to external servers unless configured. * **Scalability**: Efficient monitoring even in high-throughput environments. # eBPF Safety & Trust ## You're Right to Be Cautious When evaluating kernel-level software, a healthy skepticism is appropriate. Any technology that operates at this level deserves thorough scrutiny. This page explains exactly why eBPF is designed to be safe and why enterprises trust it in production. ## How eBPF Keeps Your Systems Safe ### The Linux Kernel Verifier The cornerstone of eBPF safety is the Linux kernel's built-in verifier. Before any eBPF program runs, the kernel performs static analysis to mathematically prove the program is safe: **What the verifier checks:** * **Bounded execution**: Programs must terminate; infinite loops are rejected * **Memory safety**: All memory accesses must be within valid bounds * **Type safety**: Operations must use correct data types * **No kernel modification**: Programs cannot alter kernel code or data structures {% hint style="info" %} **Key point**: If a program fails any verification check, the kernel rejects it entirely. The program never runs. This is enforced by Linux itself, not by Qpoint. {% endhint %} ### Sandboxed Execution eBPF programs run in a strictly controlled environment: * **Isolated execution context**: Cannot interfere with other programs or kernel operations * **Limited helper functions**: Can only call pre-approved kernel functions * **Read-only after loading**: Program code cannot be modified once loaded * **Bounded stack size**: Fixed 512-byte stack prevents overflow attacks * **No arbitrary memory access**: Can only access maps and approved memory regions ### The Safer Alternative to Kernel Modules Traditional kernel modules have full kernel access and have historically caused system crashes and outages. eBPF was designed specifically to avoid these risks: | Aspect | Traditional Kernel Modules | eBPF Programs | | ----------------------- | -------------------------- | ------------------------------ | | **Verification** | None - full kernel access | Verified before execution | | **Crash potential** | Can crash entire system | Cannot crash kernel | | **Memory access** | Unrestricted | Strictly bounded | | **Kernel modification** | Can modify kernel behavior | Read-only observation | | **Removal** | May require reboot | Hot-swappable, instant removal | | **Debugging** | Difficult, risky | Safe, observable | ## Enterprise Adoption ### Who Trusts eBPF in Production? eBPF is not experimental technology. It powers critical infrastructure at the world's largest organizations. **Financial Services:** Major financial institutions run eBPF across thousands of developers and millions of containers for container networking and security. The technology has been vetted by security teams at organizations with the strictest compliance requirements. **Cloud Providers:** All three major cloud providers have adopted eBPF-based networking as their default or recommended Container Network Interface (CNI): * Leading cloud Kubernetes services use eBPF-based CNI by default * eBPF powers network load balancing and security at global scale * The technology handles production traffic for some of the world's largest platforms **Infrastructure Leaders:** eBPF enables network observability and DDoS protection at major content delivery and streaming platforms, processing billions of requests daily. ## Third-Party Validation ### Independent Security Assessments The eBPF ecosystem has undergone rigorous independent security review: * **Verifier Security Audits**: Independent security assessments have focused on the correctness and robustness of the eBPF verifier itself, finding no systemic weaknesses * **Project Security Audits**: Major eBPF-based projects have been audited by respected security firms, with findings noting well-secured implementations * **Threat Model Documentation**: The eBPF community has published comprehensive threat models documenting security considerations and mitigations {% hint style="info" %} The eBPF community has been highly effective in identifying and fixing bugs, with efficient response times to security issues. {% endhint %} ## Your Data Never Leaves Your Environment Beyond eBPF safety, Qpoint is designed with data sovereignty as a core principle: * **Sensitive payloads stay local**: HTTP bodies and headers are stored in YOUR S3-compatible storage (MinIO, AWS S3, GCS) * **No "phone home"**: Qpoint never receives your sensitive traffic data * **Air-gapped support**: Full functionality without any internet connectivity * **You hold the keys**: We cannot access your data even if we wanted to For detailed information on data handling and compliance, see [Security & Compliance](/security-and-compliance). ## Common Questions ### How does eBPF ensure it won't crash my systems? The Linux kernel's built-in verifier mathematically proves that each eBPF program is safe before allowing it to run. Programs that could crash the kernel, access invalid memory, or loop infinitely are rejected outright. This verification happens at the kernel level and cannot be bypassed. ### What if the eBPF program has a bug? The verifier prevents bugs from causing system-level harm. Even if an eBPF program has a logic error, it cannot: * Crash the kernel * Access unauthorized memory * Corrupt system data * Interfere with other processes The worst case is incorrect output, not system instability. ### Could an attacker use this to escalate privileges? eBPF requires root/CAP\_BPF privileges to load programs. If an attacker already has root access, eBPF doesn't give them additional capabilities they don't already have. The verifier also prevents eBPF programs from being used to modify kernel behavior or escalate privileges further. ### Will this degrade performance or stability? Qtap operates out-of-band, observing traffic without being in the critical path: * **No added latency**: Traffic flows normally; Qtap observes copies * **Minimal CPU overhead**: eBPF is highly optimized, running in kernel space without context switches * **No stability risk**: Cannot crash or hang applications ### Will Qtap conflict with other eBPF tools? No. The Linux kernel is designed to support multiple eBPF programs simultaneously. Qtap can run alongside other eBPF-based tools for networking, security, and observability without conflicts. ## Summary Qtap uses eBPF to give you pre-encryption visibility without compromising safety. Our code is verified by the Linux kernel before it runs, sandboxed so it cannot crash your systems, and trusted by industry leaders who have already vetted this technology. Your data never leaves your environment. ## Learn More * [eBPF Concepts](/readme/ebpf-concepts) - Technical deep dive into how eBPF works * [Security & Compliance](/security-and-compliance) - Data handling and compliance frameworks * [How It Works](/readme/how-it-works) - Qtap technical architecture # Use Cases ## **Bridging the Gap in API Observability** **Challenge:** In today's interconnected digital landscape, your applications rely heavily on APIs. However, many organizations lack full visibility into these dependencies. Undocumented endpoints, unmonitored HTTPS traffic, and unknown vendor integrations create blind spots that complicate operations and elevate risks. Hidden connections can lead to unexpected issues, making it difficult to maintain control over your platform's external communications. **How Qpoint Helps:** Qpoint illuminates these blind spots by automatically discovering and cataloging all your third-party API connections. It provides a real-time, comprehensive inventory of your external dependencies, ensuring you know exactly which APIs and endpoints your applications are communicating with. With detailed traffic monitoring and actionable insights, Qpoint empowers your team to uncover redundant or unexpected integrations, optimize traffic flows, and reduce risks before they disrupt your platform. **Benefits:** * **Complete Visibility:** Gain a full understanding of your applications' external communications. * **Risk Reduction:** Identify and mitigate potential threats from unknown or undocumented integrations. * **Optimization Opportunities:** Discover inefficiencies and optimize your API usage to improve performance. *** ## **Enhancing Reliability and Performance of API Interactions** **Challenge:** Third-party APIs are essential to your operations, but their reliability issues can ripple across your internal platforms. Service disruptions, rate limits, and intermittent failures not only degrade user experiences but also stall critical workflows. Often, pinpointing the root cause of these issues is a daunting task, delaying resolution and prolonging downtime. **How Qpoint Helps:** Qpoint proactively mitigates these risks with real-time error detection and alerts, keeping your team informed the moment issues arise. By tracking API usage and monitoring rate-limit thresholds, Qpoint enables you to act before disruptions impact your users. Detailed logs and insightful analytics streamline root cause analysis, allowing for faster resolution and minimizing downtime. With Qpoint, you can ensure that your vendors meet their SLA obligations and maintain the quality of service your business depends on. **Benefits:** * **Proactive Issue Detection:** Address problems before they affect your users. * **Improved Uptime:** Reduce downtime by quickly identifying and resolving third-party API issues. * **Performance Optimization:** Monitor and optimize API performance to enhance user experience. *** ## **Uncovering and Eliminating Hidden Costs** **Challenge:** Third-party APIs can quietly inflate your cloud spend. Without clear visibility, overlapping services or redundant tools may go unnoticed, adding unnecessary expenses. Inefficient traffic patterns, such as excessive API calls, waste bandwidth and resources, leading to higher costs without corresponding benefits. **How Qpoint Helps:** Qpoint brings transparency to your API usage with detailed metrics that break down consumption by application, service, or team. By aligning costs with specific business units through accurate cost attribution, you can understand precisely where your budget is going. Qpoint identifies redundant or excessive API calls, providing opportunities to streamline operations and eliminate unnecessary expenses. Optimize your API usage to ensure you're only paying for what you actually need. **Benefits:** * **Cost Transparency:** Understand exactly how API usage impacts your budget. * **Expense Reduction:** Identify and eliminate unnecessary costs from redundant services. * **Resource Optimization:** Optimize API calls to save bandwidth and reduce expenses. *** ## **Strengthening Security and Compliance** **Challenge:** Misused or compromised API tokens pose significant security risks, leading to unintended access and operational vulnerabilities. Moreover, limited visibility into configurations like TLS versions can expose your organization to compliance breaches and security threats. Without clear oversight, it's challenging to enforce security protocols and maintain compliance standards. **How Qpoint Helps:** Qpoint enhances your security posture by inventorying all API tokens in use across your applications and third-party connections. It monitors critical configurations, such as TLS versions, to ensure they meet current security standards. With detailed audit trails of API interactions, Qpoint enables you to quickly identify and address token misuse or anomalies. Stay ahead of potential threats by resolving issues proactively and maintaining strict compliance with data governance standards. **Benefits:** * **Improved Security:** Protect your applications from unauthorized access through vigilant token monitoring. * **Compliance Assurance:** Ensure configurations meet industry standards and regulatory requirements. * **Risk Mitigation:** Proactively address vulnerabilities before they become critical issues. *** ## **Reducing Operational Inefficiencies** **Challenge:** Managing platforms that rely heavily on third-party APIs can be complex and time-consuming. Fragmented tools and information spread across teams and services lead to bottlenecks. Limited visibility into API interactions delays incident response and complicates troubleshooting. Manual processes and incomplete data make debugging workflows inefficient. **How Qpoint Helps:** Qpoint streamlines operations by centralizing API monitoring, error tracking, and cost attribution into one unified platform. Real-time alerts and detailed audit logs provide your team with the insights needed to diagnose and resolve issues faster. By consolidating critical information, Qpoint reduces downtime and improves operational efficiency, allowing your team to focus on delivering value rather than managing chaos. **Benefits:** * **Centralized Monitoring:** Access all API-related information in one place. * **Faster Troubleshooting:** Quickly identify and resolve issues with comprehensive data at your fingertips. * **Operational Efficiency:** Streamline workflows to save time and reduce operational costs. *** ## **Enhancing Visibility into Customer-Driven API Traffic** **Challenge:** Customer usage often drives significant third-party API activity, yet linking specific slowdowns or failures to individual users or workloads remains a challenge. This lack of visibility can lead to unmanaged costs, delayed issue resolution, and disruptions due to rate limits, negatively impacting customer experience. **How Qpoint Helps:** Qpoint offers customer-centric tracking that maps API interactions directly to user activity. This provides clear insights into how customer behavior affects API usage and performance. With detailed audit logs and real-time alerts, your team can swiftly resolve issues, optimize resources, and proactively manage high-usage customers to prevent costly disruptions. Qpoint enables you to deliver a superior customer experience by understanding and addressing the impact of user activities on your platform. **Benefits:** * **User-Level Insights:** Understand how individual customers influence API usage and performance. * **Improved Customer Experience:** Resolve customer-related issues promptly to enhance satisfaction. * **Proactive Management:** Anticipate and prevent disruptions caused by high-usage patterns. *** By addressing these critical challenges, Qpoint empowers your organization to gain full visibility into third-party API interactions, enhance reliability and performance, reduce unnecessary costs, strengthen security, and improve operational efficiency. Qpoint is your partner in navigating the complexities of modern application ecosystems, ensuring your platform operates smoothly, securely, and cost-effectively. *** ## Get Started Ready to address these challenges in your environment? ### Quick Start by Use Case **API Observability & Discovery:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) → See "Automatic Discovery" and "Vendor Inventory" **Reliability & Performance:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) → See "Set up error alerting" and "Debug slow API calls" * [Alerting Configuration](/getting-started/qplane/configuration/alerting) → Set up proactive monitoring **Cost Management:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) → See "Usage" dashboards for tracking API consumption * [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) → Configure usage reporting **Security & Compliance:** * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) → See "Security Monitoring: Scanners & Sensitive Data" * [Settings Guide](/getting-started/qplane/configuration/settings) → Configure S3 for sensitive data storage **Operational Efficiency:** * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) → Learn conditional capture to reduce noise (Level 3) * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) → Filter and optimize ### Choose Your Deployment **Cloud-Managed (Qplane):** [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Centralized dashboards, 10 minutes to value **Self-Managed (Qtap):** [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - YAML configuration, maximum control # Deployment Qpoint provides deep visibility into your service connections, helping you discover, monitor, and analyze all your internal and external API dependencies. You can manage the Qtap eBPF sensor in two different ways depending on your environment. {% hint style="success" %} **🚀 Recommended Learning Path:** * **Qplane users:** Follow the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) for the fastest path to value (10 minutes) * **Qtap users:** Follow the [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) for progressive learning (4 levels, 50 minutes) {% endhint %}

Manage Locally

  • YAML Based Configuration
  • Self-contained
  • Great for testing
  • Easy to build on top
/pages/AtjIu6aE4yUWdfm8GiL0/files/BiXJGvw8gvFvzRblbrMV

Manage with QPlane

  • Centralized cloud control plane
  • Beautiful Dashboards
  • Advanced analytics
  • Multi-environment visibility
  • SSO/RBAC
/pages/Y2y7825hG3qi7BEadNsF/files/MXe7kK3NRfkDOPAIx8AF
### Which Option Should You Choose? #### Choose Qtap Local Deployment if you want to: * Quickly try Qpoint without creating an account * Test in an isolated or air-gapped environment * Experiment with minimal setup #### Choose Qplane if you want to: * Manage configurations centrally across multiple environments * Access advanced analytics and beautiful dashboards * Monitor multiple deployments from a single interface * Share visibility with team members ### System Requirements Both deployment options require: * Linux host with kernel 5.10+ * Proper permissions for eBPF operations See [System Requirements](/getting-started/qtap/system-requirements) for more details. # Qtap Qtap is an eBPF agent that captures traffic flowing through the Linux kernel. By attaching to TLS/SSL functions, data is intercepted before and after encryption and then passed to flexible plugins with full visibility along with all of the available context - process/container/host/user/protocol/etc. Qtap makes it possible to understand what's happening with your egress traffic, without modifying apps, installing proxies, or managing certs. Qtap shows you exactly what data is being sent and received in its original, unencrypted form while operating out-of-band with minimal overhead, without adding latency or disrupting application performance. > **Note**: For cloud-managed deployments, see [Qplane Documentation](/getting-started/qplane). ### Try Qtap in 60 Seconds Want to give Qtap a test run? Spin up a temporary instance in Demo mode! See traffic in real time right in your terminal. ```bash # Run Qtap in demo mode curl -s https://get.qpoint.io/demo | sudo sh ```
Or install and start running right away: ```bash # Install the Qtap agent curl -s https://get.qpoint.io/install | sudo sh # Run with defaults! sudo qtap ``` ### Key Features * **Zero Configuration**: Works out of the box with sensible defaults * **Process Attribution**: See exactly which apps are making connections * **Pre-Encryption Visibility**: See data before TLS encryption * **Minimal Overhead**: Lightweight eBPF implementation * **Flexible Storage**: Send data to stdout, S3, or more Start simple, scale as needed. Qtap provides immediate insights with no configuration required. *** *For cloud-managed deployments with centralized configuration, see* [*Qplane Documentation*](/getting-started/qplane)*.* # Getting Started Qtap is Qpoint's lightweight eBPF agent that captures network traffic at the Linux kernel level. It can run standalone (configured via YAML) or connected to Qplane for centralized management. ## Choose Your Quick Start Path We have several getting started guides tailored to different needs: ### For First-Time Users [**5-Minute Quickstart**](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) - Fast preview with console output * **Time:** 5 minutes * **Best for:** Testing if Qtap works on your system, quick demo * **You'll get:** Real-time console output of HTTP/HTTPS traffic ### For Production Deployment [**Complete Guide: Hello World to Production**](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive 4-level tutorial * **Time:** 50 minutes (pauseable between levels) * **Best for:** Learning Qtap systematically, production deployment planning * **You'll learn:** Basic setup → Filtering → Rules → S3 storage ### For Urgent Debugging [**Production Debugging with HTTPS Visibility**](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) - Emergency troubleshooting * **Time:** 30 seconds to 5 minutes * **Best for:** Active production issues requiring immediate visibility * **You'll get:** HTTPS traffic inspection without certificates ### For Specific Use Cases * **Interactive debugging:** [DevTools UI - Browser-Based Traffic Inspection](/getting-started/qtap/configuration/devtools) * **Web servers (NGINX, Caddy, etc.):** [NGINX Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) * **Ingress traffic:** [Ingress Traffic Capture](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) * **All guides:** [Qtap Guides Overview](/guides/qtap-guides) *** ## System Requirements Before getting started, check the [system requirements](/getting-started/qtap/system-requirements) for details on Qtap compatibility. **Quick compatibility check:** {% code overflow="wrap" fullWidth="false" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ### Minimum Requirements * Linux kernel 5.10+ (check with `uname -r`) * Docker installed (for Docker-based deployment) * root/sudo permissions * eBPF support enabled *** ## Quick Demo Want to see Qtap in action right now? Run this one-liner: ```bash # Run Qtap in demo mode (captures traffic to console) curl -s https://get.qpoint.io/demo | sudo sh ``` This starts Qtap with a basic configuration that outputs captured traffic to your console. Generate some traffic with `curl` to see it in action. ## Docker Demo ### Step 1: Create a Simple Qpoint Configuration File ```bash # Create a directory for your configuration mkdir -p ~/qpoint-demo/config ``` ```bash # Create the configuration file cat > ~/qpoint-demo/config/qpoint.yaml << 'EOF' version: 2 services: event_stores: - id: console_stdout type: stdout object_stores: - id: console_stdout type: stdout stacks: default_stack: # Stack Name plugins: - type: access_logs config: mode: details # Default action (summary|details|full) format: console # (json|console) tap: direction: egress ignore_loopback: false audit_include_dns: true http: stack: default_stack EOF ``` ### Step 2: Run the Qpoint Tap Agent Now, deploy the Qtap agent: ```bash docker run -d \ --name qtap \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v ~/qpoint-demo/config:/app/config \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ tap \ --log-level=info \ --log-encoding=console \ --config=/app/config/qpoint.yaml ``` ### Step 3: Generate Some Test Traffic Let's create some test traffic to see Qpoint in action: ```bash # Create a test container and send HTTP requests docker run --rm --name test-client curlimages/curl \ sh -c "curl -v https://httpbin.org/get && \ curl -v https://httpbin.org/status/404 && \ curl -v https://httpbin.org/status/500" ``` ### Step 4: View the Captured Traffic Check the Qtap logs to see the traffic being captured: ```bash docker logs qtap -f ``` You should see log entries showing: * Connection details (source, destination, protocol) * HTTP request and response metadata * Full request and response payloads in the console * Error information for the 404 and 500 responses Example output snippet: {% code overflow="wrap" %} ``` =================================================================== curl → GET https://httpbin.org/status/500 503 Service Unavailable =================================================================== ------------------ META ------------------ PID: 194603 Exe: /usr/bin/curl Container ID: 6f4fccfae3f7 Direction: egress-external Bytes Sent: 46 Bytes Received: 232 ------------------ REQUEST ------------------ GET httpbin.org http2 Accept: */* Qpoint-Request-Id: cvct1q87p3qj89ieqtlg :authority: httpbin.org :method: GET :path: /status/500 :scheme: https User-Agent: curl/8.12.1 ------------------ RESPONSE ------------------ 503 Service Unavailable :status: 503 Server: awselb/2.0 Date: Tue, 18 Mar 2025 20:00:41 GMT Content-Type: text/html Content-Length: 162 ``` {% endcode %} ### Step 5: Test Specific Use Cases Now that Qpoint is running, you can try various use cases to see more capabilities: #### Capture API Authentication Issues ```bash # Test missing or invalid API key docker run --rm --name test-auth curlimages/curl \ -v https://httpbin.org/bearer -H "Authorization: Bearer invalid-token" ``` #### Monitor Third-Party API Health {% code overflow="wrap" %} ```bash # Create a loop to periodically check an API docker run --rm --name test-health curlimages/curl \ sh -c "for i in {1..5}; do curl -s https://httpbin.org/status/200,200,200,500 > /dev/null; sleep 2; done" ``` {% endcode %} #### Track Slow Responses ```bash # Test a delayed response docker run --rm --name test-delay curlimages/curl \ https://httpbin.org/delay/2 ``` # System Requirements ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Operating System Requirements * Linux-based operating system * Supports both x86\_64 and arm64 architectures * Kernel version 5.10 or higher ## System Configuration ### Kernel Settings * Kernel lockdown mode should be set to `none` (default on most systems) * Cgroups v2 enabled (default on most modern Linux distributions) ### Network * At least one network interface with outbound network access ### Access Requirements * Root access for installation * Read-write filesystem access ### Resource Requirements * Sufficient memory for BPF operations * Storage space for logs and configuration # Installation Qtap can be deployed and managed either through the [Qplane](/getting-started/qplane) cloud control plane or locally via a YAML configuration file, depending on your operational preferences. This flexibility allows users to choose the setup that best suits their network and security requirements. The installation process is streamlined across multiple platforms with support for both binary and containerized deployments.
Linux Binary Install/pages/Gj2Q71d398IG96Ca7kN2
Docker Container Install/pages/8Y3taNByykdY0gaWtZtS
Helm Chart Install/pages/8FPKmP98B1t2Eiy1VNpQ
# Linux Binary This guide provides a secure, best-practice approach to installing and configuring the Qtap agent on Linux systems, independent from [Qplane](/getting-started/qplane). ## Prerequisites Before installation, verify your environment's compatibility: {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Requirements: * Linux host with kernel version 5.10+ * x86\_64 or arm64 architecture * Root/sudo access ## Installation Methods ### Automated Installation The quickest way to install or update Qtap is via our installation script, which places Qtap into your path: ```bash curl -s https://get.qpoint.io/install | sudo sh ``` **Quick test (runs with default settings):** ```bash sudo qtap ``` {% hint style="info" %} Running `sudo qtap` without a configuration file starts Qtap with default settings (captures all egress traffic to stdout). This is useful for quick tests, but **not recommended for production**. You'll want to create a configuration file to: * Control what traffic to capture (filters, endpoints, direction) * Configure storage destinations (S3, Axiom, etc.) * Set up selective capture with rules (errors only, specific domains) * Define processing plugins and capture levels See [Configuration](#configuration-setup) below and [Configuration Documentation](/getting-started/qtap/configuration) for details. {% endhint %} ### Manual Installation For more control, you can manually install the binary: 1. Download the appropriate binary for your architecture: ```bash # For x86_64 curl -L https://downloads.qpoint.io/qpoint/qtap-v0.17.1-linux-amd64.tgz -o qtap.tgz # For arm64 curl -L https://downloads.qpoint.io/qpoint/qtap-v0.17.1-linux-arm64.tgz -o qtap.tgz ``` 2. Extract and install the binary: ```bash tar -xzf qtap.tgz sudo mv qtap-* /usr/local/bin/qtap sudo chmod +x /usr/local/bin/qtap ``` 3. Test the installation: ```bash sudo qtap ``` This runs Qtap with default settings. Press `Ctrl+C` to stop. {% hint style="warning" %} **Default behavior:** Running `sudo qtap` without arguments uses built-in defaults (captures all egress traffic to stdout). While this works for testing, **you should create a configuration file** before production use to control capture behavior, storage destinations, and processing rules. {% endhint %} ## Configuration Setup **Why you need a configuration file:** Qtap's configuration file (`qpoint.yaml`) allows you to: * **Control what to capture**: Filter by process, domain, direction (egress/ingress) * **Define storage**: Send data to S3, Axiom, or other destinations instead of stdout * **Selective capture**: Use rules to capture only errors, specific endpoints, or traffic patterns * **Processing control**: Choose capture levels (summary/details/full) and formats (text/json) Without a config file, Qtap captures everything to stdout with no filtering - rarely what you want in production. **Configuration resources:** * **Quick start configs**: [5-Minute Quickstart](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) * **Progressive tutorial**: [Complete Guide: Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) * **Full reference**: [Configuration Documentation](/getting-started/qtap/configuration) * **Examples**: [Configuration Examples](/getting-started/qtap/configuration/configuration-examples) ### Creating Your Configuration File 1. Create a standard configuration directory with appropriate permissions: ```bash sudo mkdir -p /etc/qtap ``` 2. Create your [configuration file](/getting-started/qtap/configuration): ```bash sudo nano /etc/qtap/qpoint.yaml ``` 3. Set appropriate permissions: ```bash sudo chmod 640 /etc/qtap/qpoint.yaml sudo chown root:root /etc/qtap/qpoint.yaml ``` 4. For sensitive credentials (if applicable), create a separate environment file: ```bash sudo touch /etc/qtap/environment sudo chmod 600 /etc/qtap/environment ``` 5. Add any sensitive environment variables to this file: ```bash # Example for S3 credentials if needed echo "S3_ACCESS_KEY=your_access_key" | sudo tee -a /etc/qtap/environment echo "S3_SECRET_KEY=your_secret_key" | sudo tee -a /etc/qtap/environment ``` ## Running as a Systemd Service 1. Create a systemd service file: ```bash sudo tee /etc/systemd/system/qtap.service << 'EOF' [Unit] Description=Qtap Traffic Analysis Service After=network.target [Service] Type=simple User=root # Only include Environment if you have sensitive environment variables EnvironmentFile=/etc/qtap/environment ExecStart=/usr/local/bin/qtap --config=/etc/qtap/qpoint.yaml Restart=always RestartSec=1 [Install] WantedBy=multi-user.target EOF ``` 2. Reload systemd, enable and start the service: ```bash sudo systemctl daemon-reload sudo systemctl enable qtap sudo systemctl start qtap ``` ## Verification 1. Check the service status: ```bash sudo systemctl status qtap ``` 2. Verify the running version: ```bash qtap --version ``` 3. Monitor the logs: ```bash sudo journalctl -u qtap -f ``` ## Updating Qtap Service ### Installation Script The quickest way to update Qtap is via our installation script: 1. Stop the service: ```bash sudo systemctl stop qtap ``` 2. Download and install the new version: ```bash curl -s https://get.qpoint.io/install | sudo sh ``` 3. Restart the service: ```bash sudo systemctl start qtap ``` 4. Verify the update: ```bash qtap --version sudo systemctl status qtap ``` ### Manually 1. Stop the service: ```bash sudo systemctl stop qtap ``` 2. Download and install the new version: ```bash # Download the new version curl -L https://downloads.qpoint.io/qpoint/qtap-vX.Y.Z-linux-amd64.tgz -o qtap-new.tgz # Extract and replace tar -xzf qtap-new.tgz sudo mv qtap-* /usr/local/bin/qtap sudo chmod +x /usr/local/bin/qtap ``` 3. Restart the service: ```bash sudo systemctl start qtap ``` 4. Verify the update: ```bash qtap --version sudo systemctl status qtap ``` ## Configuration Management Best Practices * **Version Control**: Track configuration changes in a version control system * **Regular Backups**: Include `/etc/qtap` in your backup strategy * **Configuration Reviews**: Implement a review process for configuration changes * **Automation**: Consider using configuration management tools (Ansible, Chef, Puppet) for deployment Remember that Qtap requires root permissions to work properly due to its use of eBPF for traffic monitoring. # Docker Container ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * A Linux host with supported kernel (5.10+) * `Docker`. For installation instructions, refer to the [Docker official documentation](https://docs.docker.com/get-docker/). * [Qtap YAML config file](/getting-started/qtap/configuration) ## Installation Steps Download the latest Docker Image: ```bash docker pull us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` ### Running Qtap in Docker {% tabs %} {% tab title="Local Only" %} ```bash docker run \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd):/app/config" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config=/app/config/qpoint.yaml ``` Replace `/app/config/qpoint.yaml` with the actual path to your YAML configuration file inside the container. You can exclude this for testing, and Qtap will run in a default configuration. Make sure to mount your local configuration file to this path using the `-v` option. {% endtab %} {% endtabs %} ## Docker Run Command Options Explained 1. `--user 0:0`: Runs the container as root (necessary for eBPF operations). 2. `--privileged`: Gives extended privileges to this container. 3. `--cap-add CAP_BPF`: Adds the CAP\_BPF capability (required for eBPF operations). 4. `--cap-add CAP_SYS_ADMIN`: Adds the CAP\_SYS\_ADMIN capability (for low-level system operations). 5. `--pid=host`: Shares the host's PID namespace with the container. 6. `--network=host`: Uses the host's network stack inside the container. 7. `-v /sys:/sys`: Mounts the host's /sys directory into the container. 8. `-v /var/run/docker.sock:/var/run/docker.sock:` Mounts the host's docker socket 9. `-v "$(pwd):/app/config"`: Mounts the current directory to /app/config in the container. 10. `-e TINI_SUBREAPER=1`: Sets up Tini as a subreaper for proper signal handling. 11. `--ulimit=memlock=-1`: Removes the memory lock limit for eBPF programs. ## Qtap-specific Flags * `--log-level=info`: Sets the logging level (debug, info, warn, error). * `--log-encoding=console`: Sets the log encoding format. * `--registration-token=$TOKEN`: (Cloud-connected mode) Provides the registration token for Qtap. * `--config=/app/config/qpoint.yaml`: (Local mode) Specifies the path to the Qtap configuration file. ## Available Flags and Options To see all available options and flags, run: ```bash docker run --rm us-docker.pkg.dev/qpoint-edge/public/qtap:v0 --help ``` # Helm Chart ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * Kubernetes cluster on a Linux Host with supported Kernel (5.10+) * `Helm` * `Kubectl` * [Qtap YAML config file](/getting-started/qtap/configuration) ## Install Qpoint Helm Repo First, add the Qpoint Helm Repo: ```bash helm repo add qpoint https://helm.qpoint.io/ ``` ```bash helm repo update ``` ```bash helm search repo qpoint/qtap --versions ``` ```bash helm show values qpoint/qtap ``` ## Deploy Qtap You'll need to provide your Qtap configuration using a YAML file. First, [create your configuration file](/getting-started/qtap/configuration) `qtap-config.yaml` and supply it via your helm command: ```bash helm install qtap qpoint/qtap \ -n qpoint \ --create-namespace \ --set logLevel=warn \ --set-file config=./qtap-config.yaml ``` ## Verifying the Deployment To verify that Qtap is running: ```bash kubectl get pods -n qpoint ``` You should see a pod named `qtap-xxxx` in the Running state. ## Uninstalling Qtap To uninstall Qtap: ```bash helm uninstall qtap -n qpoint ``` ## Important Notes 1. The Qtap pod requires privileged access for eBPF operations. Ensure your cluster's security policies allow this. 2. To set a specific version, use `--set image.tag="v0.17.1"` 3. For cloud-connected mode, keep your registration token secure and do not share it in public repositories. 4. For local mode, ensure your configuration is correctly formatted and contains all necessary settings. 5. The default configuration mounts the host's `/sys` directory. Ensure this is allowed in your cluster. 6. You may need to adjust resource requests and limits based on your cluster's capacity and Qtap's requirements. 7. When using secret management, ensure proper RBAC permissions are in place for accessing the secret. # Kubernetes Manifest This guide covers how to deploy Qtap, an eBPF-based traffic monitoring agent, in Kubernetes environments. ### Prerequisites * Kubernetes cluster on Linux hosts with supported kernel (5.10+) * `kubectl` * `Helm` ### Generating Base Kubernetes Manifest If you'd like to use the Helm chart as a base for building a Kubernetes manifest, you can do so with the following instructions: ```bash helm repo add qpoint https://helm.qpoint.io/ helm template qtap qpoint/qtap > qtap-base.yaml ``` This command generates a base manifest file named `qtap-base.yaml`. ### Customizing Qtap's Configuration If you wish to supply your own Qtap configuration file, first write it to a file named `qtap-config.yaml` . Refer to the [Configuration](/getting-started/qtap/configuration) page for further information on the configuration file structure. Then, instruct helm to use it like so: ```bash helm template qtap qpoint/qtap \ --set-file config=qtap-config.yaml \ > qtap-base.yaml ``` To see additional helm chart configuration options: ```bash helm show values qpoint/qtap ``` ### Deploying Qtap Create a namespace and apply the modified manifest to your cluster: ```bash kubectl create ns qpoint kubectl apply -f qtap-base.yaml -n qpoint ``` ### Verifying the Deployment To verify that Qtap is running: ```bash kubectl get pods -n qpoint ``` You should see pods named `qtap-xxxx` in the Running state. To check the deployment logs: ```bash kubectl logs -n qpoint pod qtap-xxxx ``` ### Uninstalling Qtap To uninstall Qtap: ```bash kubectl delete -f qtap-base.yaml -n qpoint ``` ### Understanding the Base Manifest The base manifest creates several Kubernetes resources: 1. **ServiceAccount**: Provides an identity for the Qtap pods 2. **ConfigMap**: Stores the Qtap configuration 3. **DaemonSet**: Ensures Qtap runs on every node in the cluster #### Key Components in the DaemonSet The DaemonSet specification includes several important settings: * **Host Access**: Uses `hostPID: true` and `hostNetwork: true` to access the host's process namespace and network * **Security Context**: Requires privileged access and specific capabilities (`CAP_BPF`, `CAP_SYS_ADMIN`) for eBPF operations * **Volume Mounts**: * `/sys`: Access to the host's system directories (required for eBPF) * `/run/containerd/containerd.sock`: Access to the container runtime socket (for rich container attribution). Optional. * Configuration file * **Probes**: Health checks to ensure the Qtap pod is running correctly * **Resource Limits**: Controls how much CPU and memory Qtap can use ### Example Here's a complete deployment example with an external object store: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: qtap labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap app.kubernetes.io/version: "v0.17.1" automountServiceAccountToken: true --- apiVersion: v1 kind: ConfigMap metadata: name: qtap-config labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap app.kubernetes.io/version: "v0.17.1" data: tap-config.yaml: | version: 2 services: event_stores: - id: console_stdout type: stdout object_stores: - id: minio type: s3 endpoint: minio.storage.svc.cluster.local:9000 bucket: qpoint region: us-east-1 access_url: http://minio.storage.svc.cluster.local:9000/{{BUCKET}}/{{DIGEST}} insecure: true access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY stacks: default_stack: plugins: - type: debug config: mode: summary - type: detect_errors config: rules: - name: "All Errors" trigger_status_codes: - '4xx' - '5xx' only_categories: - app report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: true tap: direction: egress ignore_loopback: true audit_include_dns: true http: stack: default_stack filters: groups: - eks - kubernetes --- apiVersion: apps/v1 kind: DaemonSet metadata: name: qtap labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap app.kubernetes.io/version: "v0.17.1" spec: selector: matchLabels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap template: metadata: labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap app.kubernetes.io/version: "v0.17.1" spec: hostPID: true hostNetwork: true serviceAccountName: qtap securityContext: null containers: - name: qtap securityContext: allowPrivilegeEscalation: true capabilities: add: - CAP_BPF - CAP_SYS_ADMIN privileged: true readOnlyRootFilesystem: false runAsGroup: 0 runAsNonRoot: false runAsUser: 0 image: "us-docker.pkg.dev/qpoint-edge/public/qtap:v0" imagePullPolicy: IfNotPresent args: [] env: - name: QPOINT_CONFIG value: "/app/tap-config.yaml" - name: S3_ACCESS_KEY valueFrom: secretKeyRef: name: minio-credentials key: access-key - name: S3_SECRET_KEY valueFrom: secretKeyRef: name: minio-credentials key: secret-key - name: STATUS_LISTEN value: "0.0.0.0:10001" - name: LOG_LEVEL value: "info" - name: LOG_ENCODING value: "json" - name: TINI_SUBREAPER value: "1" ports: - name: status containerPort: 10001 protocol: TCP startupProbe: httpGet: path: /readyz port: status initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 20 readinessProbe: httpGet: path: /readyz port: status initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 1 livenessProbe: httpGet: path: /healthz port: status initialDelaySeconds: 3 periodSeconds: 10 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 3 resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 100m memory: 128Mi volumeMounts: - name: config-volume mountPath: /app/tap-config.yaml subPath: tap-config.yaml - mountPath: /sys name: sys readOnly: true - name: containerd-socket mountPath: /run/containerd/containerd.sock volumes: - name: config-volume configMap: name: qtap-config items: - key: tap-config.yaml path: tap-config.yaml - hostPath: path: /sys type: Directory name: sys - name: containerd-socket hostPath: path: /run/containerd/containerd.sock type: Socket ``` ### Important Notes * **Privileged Access**: The Qtap pod requires privileged access for eBPF operations. Ensure your cluster's security policies allow this. * **Token Security**: For cloud-connected mode, keep your registration token secure and do not share it in public repositories. * **Configuration Format**: For local mode, ensure your configuration is correctly formatted and contains all necessary settings. * **Host Access**: The deployment mounts the host's `/sys` directory and container runtime socket. Ensure this is allowed in your cluster. * **Resource Management**: You may need to adjust resource requests and limits based on your cluster's capacity and Qtap's workload. * **DaemonSet Deployment**: This deployment uses a DaemonSet to ensure Qtap runs on every node in your cluster. Adjust if this is not your intended behavior. ### Common Customizations The generated manifest provides a starting point. You may need to customize various aspects such as: * **Resource limits and requests**: Adjust based on your workload and node capacity * **Node selectors or tolerations**: Target specific nodes or allow scheduling despite taints * **Environment variables**: Add additional configuration or credentials * **Volume mounts**: Access additional host resources if needed * **Security contexts**: Adjust permissions based on your security requirements Always review and test your modifications in a non-production environment before deploying to production. ### Troubleshooting If you encounter issues with your Qtap deployment, check the following: 1. **Pod Status**: Check if the pods are running ```bash kubectl get pods -n qpoint ``` 2. **Pod Logs**: Examine the logs for error messages ``` kubectl logs -n qpoint ds qtap # or, for a specific pod kubectl logs -n qpoint pod qtap-xxxx ``` 3. **Configuration:** Verify your configuration is correctly formatted ```bash kubectl describe configmap -n qpoint qtap-config ``` 4. **Permissions**: Ensure the pod has the necessary permissions ```bash kubectl describe pod -n qpoint qtap-xxxx ``` 5. **Kernel Support**: Verify your nodes are running a supported kernel version (5.10+) ```bash kubectl debug node/node-name -it --image=ubuntu uname -r ``` # Configuration Qtap can be managed locally with a `yaml` config file. This guide explains how to create, deploy, and maintain a local configuration file for Qtap. {% hint style="info" %} **New to Qtap?** Follow the [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) for hands-on examples progressing from basic to production-ready configurations (4 levels, 50 minutes). {% endhint %} {% hint style="success" %} **Want centralized management?** Check out [Qplane](/getting-started/qplane) for cloud-based configuration with visual dashboards, automatic propagation to all agents, and advanced analytics. See the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) to get started. {% endhint %} ## Configuration File Structure The `qpoint.yaml` file consists of three main sections: 1. [**Storage Configurations**](/getting-started/qtap/configuration/storage-configuration) **(Where processed data goes)** 2. [**Stacks**](/getting-started/qtap/configuration/traffic-processing-with-plugins) **(How data is processed)** 3. [**Traffic Capture Rules**](/getting-started/qtap/configuration/traffic-capture-settings) **(What data is processed)** Here's the basic structure: ```yaml version: 2 services: event_stores: # Event storage configuration object_stores: # Object storage configuration stacks: stack_1: plugins: # First plugin configuration stack_2: plugins: # Second plugin configurations (optional) tap: # Global traffic capture settings ``` ## Creating a Basic Configuration Let's walk through creating a simple configuration file step by step. ### Step 1: Set Version Start by setting the configuration version: ```yaml version: 2 ``` ### Step 2: Configure Data Storage Next, define where captured data will be stored: {% tabs %} {% tab title="Stdout" %} ```yaml services: # For connection metadata (anonymized) event_stores: - id: console_stdout type: stdout # For actual request/response content object_stores: - id: console_stdout type: stdout ``` This configuration: * Outputs events (connection metadata) to the console for debugging * Outputs objects (request/response content) to the console for debugging {% endtab %} {% tab title="S3 Endpoint" %} ```yaml services: # For connection metadata (anonymized) event_stores: - id: console_stdout type: stdout # For actual request/response content object_stores: - id: minio type: s3 endpoint: 127.0.0.1:9000 bucket: qpoint region: us-east-1 access_url: http://localhost:9000/{{BUCKET}}/{{DIGEST}} insecure: true access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY ``` This configuration: * Outputs events (connection metadata) to the console for debugging * Stores objects (request/response content) in a locally running MinIO S3-compatible store * Uses environment variables for S3 credentials (recommended for security) {% endtab %} {% endtabs %} ### Step 3: Set Up Processing Plugins Define how captured data will be processed: ```yaml stacks: default_stack: # Stack Name plugins: - type: access_logs config: mode: details # Default action (summary|details|full) format: console # (json|console) rules: - name: summary log example.com expr: request.host == "example.com" mode: summary - name: details log on anything above 400 expr: response.status >= 400 mode: full ``` This configuration: * Creates a stack named "default\_stack" * The access logs plugin provides summary-level debug information with specific capture definitions for example.com, or full payload capture for traffic with a status code above 400 ### Step 4: Configure Traffic Capture Finally, set up what traffic to capture and send to plugins: ```yaml tap: direction: egress ignore_loopback: false audit_include_dns: true http: stack: default_stack ``` This configuration: * Captures all outgoing (egress) traffic * Includes loopback traffic * Includes DNS information in audit logs * Applies the default\_stack to HTTP traffic for processing ## Example Putting it all together: ```yaml version: 2 services: event_stores: - id: console_stdout type: stdout object_stores: - id: console_stdout type: stdout stacks: default_stack: # Stack Name plugins: - type: access_logs config: mode: details # Default action (summary|details|full) format: console # (json|console) rules: - name: summary log example.com expr: request.host == "example.com" mode: summary - name: details log on anything above 400 expr: response.status >= 400 mode: full tap: direction: egress ignore_loopback: true audit_include_dns: true http: stack: default_stack ``` # Storage Configuration ## Understanding Qtap Storage Components Qtap captures two distinct types of data, each with different storage requirements: 1. [**Events**](#event-stores) (Connection Metadata): Anonymized information about connections, including timestamps, endpoints, and performance metrics 2. [**Objects**](#object-stores) (Payload Content): Actual request and response data including headers and bodies, which may contain sensitive information Each type has its own dedicated storage configuration in the `services` section of your `qpoint.yaml` file. {% hint style="info" %} Objects (Payload Content) are selectively captured based on the plugin configuration. See [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) {% endhint %} ```yaml services: event_stores: - id: console_stdout type: stdout object_stores: - id: console_stdout type: stdout ``` ## Event Stores Event stores handle anonymized metadata about network connections. This data is generally not sensitive but is useful for analytics, troubleshooting, and monitoring. ### **Console Output (stdout)** The simplest option for development and debugging: ```yaml event_stores: - id: console_stdout type: stdout ``` This configuration sends all event data to the console where Qtap is running, making it immediately visible but not persistent. ### **Axiom** For sending events to Axiom for analytics and monitoring: ```yaml event_stores: - type: axiom dataset: type: text value: qpoint-events token: type: env value: AXIOM_TOKEN ``` This configuration sends event data to an Axiom dataset for advanced analytics and visualization. **Axiom Configuration Parameters**
ParameterDescription
datasetThe name of the Axiom dataset to send events to
tokenAxiom API token
### **OpenTelemetry (OTLP)** Send events to any OpenTelemetry-compatible backend using the OTLP protocol. Qtap exports events as **OpenTelemetry Logs** with rich structured attributes. ```yaml event_stores: - type: otel endpoint: "localhost:4317" # OTLP endpoint (required for grpc/http, not needed for stdout) protocol: grpc # grpc, http, or stdout (required) service_name: "qtap" # Service name for resource attributes (optional) environment: "production" # Environment tag (optional) headers: # Custom headers (optional) api-key: type: env value: OTEL_API_KEY tls: enabled: false # Enable TLS (optional, default: false) ``` This configuration works with any OTLP-compatible backend including: * OpenTelemetry Collector * Datadog (OTLP ingestion) * Honeycomb (OTLP API) * New Relic (OTLP endpoint) * Grafana Cloud (OTLP) * Elastic (OTLP) **OpenTelemetry Configuration Parameters**
ParameterDescription
endpointOTLP endpoint address (e.g., "localhost:4317" for gRPC, "localhost:4318" for HTTP). Required for grpc and http protocols. Not needed for stdout protocol.
protocolProtocol to use: "grpc" (default port 4317), "http" (default port 4318), or "stdout" for debugging
service_nameService name added to resource attributes (default: "qtap")
environmentEnvironment name for filtering/grouping (e.g., "production", "staging")
headersCustom headers for authentication (e.g., API keys). Each header can use type: env to load from environment variables or type: text for direct values
tls.enabledEnable TLS for secure connections (default: false)
**Event Types Sent** Qtap sends two types of OpenTelemetry log events: 1. **Connection Events** (`event.type: connection`) - TCP connection lifecycle and metadata including TLS version, protocol detection, bytes transferred 2. **Artifact Records** (`event.type: artifact_record`) - HTTP transaction summaries with method, URL, status, duration, bytes, and links to full request/response data in object stores See the [OpenTelemetry Integration Guide](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) for detailed setup instructions. ### **Qpoint Pulse Service (Coming Soon)** For self-hosted environments with a Pulse instance: ```yaml event_stores: - id: pulse type: pulse endpoint: http://pulse-service:8000 token: type: env value: PULSE_TOKEN ``` This connects to a Pulse service for advanced analytics and visualization. ## Object Stores Object stores contain the actual content of requests and responses, which often includes sensitive information. This data requires more careful handling and secure storage. ### **Console Output (stdout)** For development and debugging: ```yaml object_stores: - id: console_stdout type: stdout ``` Sends all object data to the console. ### **OpenTelemetry (OTLP)** Send captured artifacts (request/response headers and bodies) through your existing OpenTelemetry pipeline. Artifacts are exported as **OpenTelemetry Logs**, flowing through the same collector as your events. ```yaml object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false ``` This pairs naturally with the OTel event store — point both at the same collector endpoint for a unified pipeline: ```yaml services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false ``` **OpenTelemetry Object Store Configuration Parameters**
ParameterDescription
otel_endpointOTLP endpoint address (e.g., "localhost:4317" for gRPC, "localhost:4318" for HTTP)
protocolProtocol to use: "grpc" (default port 4317), "http" (default port 4318), or "stdout" for debugging
service_nameService name added to resource attributes (default: "qtap")
environmentEnvironment name for filtering/grouping (e.g., "production", "staging")
headersCustom headers for authentication. Each header can use type: env to load from environment variables or type: text for direct values
tls.enabledEnable TLS for secure connections (default: false)
See the [sample ClickStack configuration](https://github.com/qpoint-io/qtap/blob/main/examples/sample-otel-objectstore-clickstack.yaml) for a complete working example, and the [OTel Object Store release notes](/release-notes/otel-object-store) for more details. ### **S3-Compatible Storage** For secure, persistent storage: ```yaml object_stores: - id: s3_store type: s3 endpoint: storage.example.com:9000 bucket: qpoint-objects region: us-east-1 access_url: https://storage.example.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY ``` This configuration: * Stores objects in an S3-compatible storage service * Uses HTTPS for secure transmission (`insecure: false`) * Retrieves credentials from environment variables * Provides a template URL for accessing stored objects **S3 Configuration Parameters** | Parameter | Description | Example | | ------------ | ------------------------------ | --------------------------------------------------- | | `endpoint` | S3 server hostname and port | `minio.example.com:9000` | | `bucket` | S3 bucket name | `qpoint-objects` | | `region` | S3 region name | `us-east-1` | | `access_url` | URL template for object access | `https://storage.example.com/{{BUCKET}}/{{DIGEST}}` | | `insecure` | Allow HTTP instead of HTTPS | `false` (recommended) | | `access_key` | S3 access key configuration | See credential management | | `secret_key` | S3 secret key configuration | See credential management | **URL Template Variables** The `access_url` parameter supports these template variables: * `{{ENDPOINT}}`: The S3 endpoint * `{{BUCKET}}`: The bucket name * `{{DIGEST}}`: The unique file identifier ## Credential Management For security, Qtap supports retrieving credentials from environment variables or using direct text values. **Environment Variable Configuration** ```yaml token: type: env value: AXIOM_TOKEN # Name of the environment variable ``` **Direct Text Configuration** ```yaml token: type: text value: your_actual_token # Direct token value (not recommended for production) ``` For S3 credentials, when running Qtap, ensure these environment variables are set: ```bash export S3_ACCESS_KEY=your_access_key export S3_SECRET_KEY=your_secret_key ``` For Docker: ```bash docker run \ # Other parameters... -e S3_ACCESS_KEY=your_access_key \ -e S3_SECRET_KEY=your_secret_key \ # Rest of command... ``` For Kubernetes, use secrets: ```bash kubectl create secret generic s3-credentials \ --from-literal=access-key='YOUR_ACCESS_KEY' \ --from-literal=secret-key='YOUR_SECRET_KEY' \ -n qpoint ``` And reference them in your Helm values: ```yaml extraEnv: - name: S3_ACCESS_KEY valueFrom: secretKeyRef: name: s3-credentials key: access-key - name: S3_SECRET_KEY valueFrom: secretKeyRef: name: s3-credentials key: secret-key ``` ## Object Storage Configuration Examples ### MinIO Configuration [MinIO](https://min.io/) is a popular self-hosted, S3-compatible object store: ```yaml object_stores: - id: minio type: s3 endpoint: minio.example.com:9000 bucket: qpoint region: us-east-1 access_url: https://minio.example.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` ### AWS S3 Configuration For AWS S3: ```yaml object_stores: - id: aws_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` ### Google Cloud Storage Configuration For Google Cloud Storage: ```yaml object_stores: - id: gcs type: s3 endpoint: storage.googleapis.com bucket: my-company-qpoint region: us-central1 access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: GCS_ACCESS_KEY secret_key: type: env value: GCS_SECRET_KEY ``` ## Security Best Practices When configuring storage, especially for production environments: 1. **Use HTTPS**: Always set `insecure: false` to enforce encrypted connections 2. **Environment Variables**: Never store credentials in the configuration file 3. **Bucket Policies**: Restrict access to your storage bucket with appropriate IAM policies 4. **Encryption**: Enable server-side encryption for stored objects 5. **Lifecycle Rules**: Configure automatic deletion of old data to comply with retention policies 6. **Audit Logging**: Enable access logging for your storage service ## Complete Storage Configuration Example ```yaml version: 2 services: event_stores: - id: console_stdout type: stdout object_stores: - id: minio type: s3 endpoint: minio.internal:9000 bucket: qpoint-objects region: us-east-1 access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY ``` This configuration sends connection metadata to the console for easy monitoring while securely storing the actual request and response content in MinIO. # Traffic Processing with Plugins ## Understanding Stacks and Plugins In Qtap's configuration, traffic processing is organized using two key concepts: * **Stacks**: Named collections of plugins that work together to process traffic * **Plugins**: Individual components that perform specific functions on captured traffic This structure allows you to create different processing configurations for different types of traffic. {% hint style="success" %} **New to plugins?** Follow the [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) for hands-on examples progressing through all 4 levels of plugin configuration (from basic to production-ready). {% endhint %} {% hint style="info" %} **Want centralized management?** [Qplane](/getting-started/qplane/configuration/stacks-and-plugins) provides visual configuration for stacks and plugins with automatic propagation to all agents. See the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) to get started. {% endhint %} ## Stack Configuration Stacks are defined in the `stacks` section of your `qpoint.yaml` file. Each stack has a unique name and contains one or more plugins: ```yaml stacks: default_stack: # Stack name plugins: # List of plugins in this stack - type: http_capture config: # Plugin-specific configuration ``` You can create multiple stacks for different purposes, each with its own set of plugins and configurations. ## Available Plugins Qtap includes several plugins that provide different processing capabilities. Rules use [Rulekit](https://github.com/qpoint-io/rulekit) - Qpoint's flexible expression-based rules engine for evaluating conditions against HTTP traffic. ### HTTP Capture Plugin The `http_capture` plugin provides comprehensive HTTP traffic capture with flexible logging levels and the ability to upload payloads to object storage. **Basic Configuration** ```yaml - type: http_capture config: level: summary # Default capture level (none|summary|headers|full) format: text # Output format (text|json) ``` **Level Options** * `none`: No capture (effectively disables the plugin) * `summary`: Basic information (method, path, status code) * `headers`: Includes headers * `full`: Complete information including request/response bodies **Example with Rules** ```yaml - type: http_capture config: level: summary # Default level for most traffic format: text # Human-readable format rules: - name: "Full capture for httpbin.org" expr: http.req.host == "httpbin.org" level: full - name: "Headers logging for server errors" expr: http.res.status >= 500 level: headers - name: "Full capture for client errors" expr: http.res.status >= 400 && http.res.status < 500 level: full ``` This configuration: 1. Uses summary level by default 2. Captures full details for all traffic to `httpbin.org` 3. Captures detailed information for server errors (5xx) 4. Captures full information for client errors (4xx) **Container and Pod-based Filtering** You can also create rules based on container or Kubernetes pod attributes: ```yaml - type: http_capture config: level: summary format: text rules: - name: "Debug specific container" expr: src.container.name == "my-app-container" level: full - name: "Debug by container label" expr: src.container.labels.qpoint_debug_level == "debug" level: full - name: "Debug specific pod" expr: src.pod.name == "frontend-deployment-abc123" level: full - name: "Debug by pod label" expr: src.pod.labels.app == "frontend" && src.pod.labels.debug == "true" level: full ``` The `http_capture` plugin uploads captured data to the configured object store at all capture levels. At `summary` level, the stored object contains metadata only (method, URL, status, duration). At `headers` level, request and response headers are included. At `full` level, both headers and bodies are captured and stored. ### Access Logs Plugin The `access_logs` plugin provides formatted logging of HTTP traffic to `stdout`. It does not upload to the object store. **Basic Configuration** ```yaml - type: access_logs config: mode: details # Default logging level (summary|details|full) format: console # Output format (console|json) ``` **Mode Options** * `summary`: Basic information (method, path, status code) * `details`: Includes headers and timing information * `full`: Complete information including request/response bodies **Example with Rules** ```yaml - type: access_logs config: mode: summary # Default mode for most traffic format: console # Human-readable format rules: - name: "Detailed API Logging" expr: http.req.host == "api.example.com" mode: details - name: "Full Error Logging" expr: http.res.status >= 400 mode: full ``` ### HTTP Metrics Plugin The `http_metrics` plugin exposes Prometheus-style metrics for HTTP traffic. It tracks request and response counts, durations, and payload sizes — useful for dashboards, alerting, and capacity planning. **Configuration** ```yaml - type: http_metrics ``` No additional configuration is needed. Once included in a stack, the plugin automatically records the following metrics: **Exported Metrics** | Metric | Type | Description | | ----------------------- | --------- | ------------------------------------ | | `requests_total` | Counter | Total number of requests | | `requests_duration_ms` | Histogram | Request duration in milliseconds | | `requests_size_bytes` | Histogram | Request payload size in bytes | | `responses_total` | Counter | Total number of responses | | `responses_duration_ms` | Histogram | Response duration in milliseconds | | `responses_size_bytes` | Histogram | Response payload size in bytes | | `duration_ms` | Histogram | Combined request + response duration | All counter and histogram metrics include labels for `method`, `host`, `status_code`, and `protocol`. **Example** ```yaml stacks: monitored: plugins: - type: http_metrics - type: access_logs config: mode: summary format: console ``` This gives you both Prometheus metrics and console logging in the same stack. ### Report Usage Plugin The `report_usage` plugin sends anonymized usage metrics to Pulse. This is mainly useful when using [Qplane](/getting-started/qplane). It works alongside other plugins and doesn't affect traffic capture. ```yaml - type: report_usage ``` You can also use `report_usage` to promote specific HTTP headers into your OTel event store as custom attributes. Add a `headers` list to extract headers from observed traffic: ```yaml - type: report_usage config: headers: - Cf-Access-Authenticated-User-Email - X-Forwarded-For - X-Request-Id ``` Promoted headers appear as `customHeaders.req.` and `customHeaders.res.` attributes in your OTel backend. See the full guide: [Exporting HTTP Headers to OpenTelemetry](/guides/qtap-guides/observability-and-integration/exporting-http-headers-to-opentelemetry). This plugin is optional and can be included in any stack. ### Error Detection Plugin The `detect_errors` plugin captures detailed information when HTTP responses meet specific error criteria. It provides granular control over what gets recorded — you can independently toggle request headers, request body, response headers, and response body per rule. This is especially useful when you need to capture request payloads without storing potentially massive response bodies (e.g., API calls that return large datasets). **Basic Configuration** ```yaml - type: detect_errors config: rules: - name: "Server Errors" trigger_status_codes: - '5xx' report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: true ``` **Rule Options** | Field | Type | Description | | ---------------------- | -------- | -------------------------------------------------------------------- | | `name` | string | Rule name for identification | | `trigger_status_codes` | list | Status code patterns to match (e.g., `'5xx'`, `'404'`, `'4xx'`) | | `trigger_empty_body` | bool | Trigger when response body is empty | | `trigger_duration` | duration | Trigger when request exceeds this duration (e.g., `'5s'`, `'500ms'`) | | `trigger_contains` | string | Trigger when response body contains this string | | `only_categories` | list | Only match specific MIME categories | | `only_urls` | list | Only match specific URL patterns | | `exclude_urls` | list | Exclude specific URL patterns | | `with_tags` | list | Only match connections with specific tags | | `report_as_issue` | bool | Flag matched requests as issues in the event store | | `record_req_headers` | bool | Record request headers | | `record_req_body` | bool | Record request body | | `record_res_headers` | bool | Record response headers | | `record_res_body` | bool | Record response body | **Example: Capture requests but skip large response bodies** ```yaml - type: detect_errors config: rules: - name: "API request capture" trigger_status_codes: - '2xx' - '4xx' - '5xx' record_req_headers: true record_req_body: true record_res_headers: true record_res_body: false # Skip potentially massive responses ``` **Example: Slow query detection with full capture** ```yaml - type: detect_errors config: rules: - name: "Slow requests" trigger_duration: '5s' report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: true - name: "Errors without response body" trigger_status_codes: - '5xx' report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: false ``` ### Debug Plugin (Deprecated) {% hint style="warning" %} **DEPRECATED:** The `debug` plugin is deprecated and will be removed in a future version. **Migration:** Use `access_logs` plugin for console output, or `http_capture` for more flexible capture. **Why deprecated:** The `access_logs` plugin provides better formatting and more features, making this plugin redundant. {% endhint %} The `debug` plugin provides basic logging of HTTP traffic. **Example of OLD (deprecated) approach:** ```yaml - type: debug config: mode: summary # or "details" for more information ``` **NEW approach using `access_logs`:** ```yaml - type: access_logs config: mode: details # summary, details, or full format: console # Human-readable console output ``` **Migration benefits:** * Better formatted output (Apache-style logs) * Support for both console and JSON formats * Rule-based selective logging * Consistent with other plugins ## Rule Expressions with Rulekit Both the `http_capture` and `access_logs` plugins use [Rulekit](https://github.com/qpoint-io/rulekit) for rule evaluation. Rulekit is Qpoint's expression-based rules engine that evaluates conditions against key-value data from HTTP traffic. ### Expression Syntax Rule expressions follow a straightforward pattern: ``` ``` Multiple conditions can be combined using logical operators: ``` and ``` ### Available Fields for Rule Expressions **Request Fields** | Field | Example Usage | Description | | ------------------------- | ----------------------------------------------------- | ----------------------------- | | `http.req.method` | `http.req.method == "POST"` | HTTP method (GET, POST, etc.) | | `http.req.path` | `http.req.path contains "/api/"` | Request path | | `http.req.host` | `http.req.host == "api.example.com"` | Host header value | | `http.req.url` | `http.req.url contains "search"` | Full URL | | `http.req.headers.` | `http.req.headers.content-type == "application/json"` | Request header by name | **Response Fields** | Field | Example Usage | Description | | ------------------------- | ----------------------------------------------- | ----------------------- | | `http.res.status` | `http.res.status >= 400` | HTTP status code | | `http.res.headers.` | `http.res.headers.content-type contains "json"` | Response header by name | **Source Context Fields** | Field | Example Usage | Description | | ---------------------------- | ---------------------------------------- | --------------------- | | `src.container.name` | `src.container.name == "my-app"` | Container name | | `src.container.labels.` | `src.container.labels.app == "frontend"` | Container label value | | `src.pod.name` | `src.pod.name == "frontend-abc123"` | Kubernetes pod name | | `src.pod.labels.` | `src.pod.labels.version == "v2"` | Pod label value | ### Operators for Rule Expressions Rulekit supports various operators for building expressions: **Comparison Operators** | Operator | Aliases | Description | Example | | ---------- | --------- | ------------------------ | ----------------------------------------- | | `==` | `eq` | Equal to | `http.req.method == "GET"` | | `!=` | `ne` | Not equal to | `http.req.host != "internal.example.com"` | | `>` | `gt` | Greater than | `http.res.status > 200` | | `>=` | `ge` | Greater than or equal to | `http.res.status >= 400` | | `<` | `lt` | Less than | `http.res.status < 300` | | `<=` | `le` | Less than or equal to | `http.res.status <= 399` | | `=~` | `matches` | Matches regex pattern | `http.req.path matches /^\/api\/v\d+\//` | | `contains` | | Contains substring | `http.req.url contains "search"` | | `in` | | Is contained in array | `http.req.method in ["GET", "HEAD"]` | **Logical Operators** | Operator | Aliases | Description | Example | | -------- | ------- | ----------- | ------------------------------------------------------------------- | | `and` | `&&` | Logical AND | `http.req.method == "POST" and http.res.status >= 400` | | `or` | `\|\|` | Logical OR | `http.req.path contains "/admin" or http.req.path contains "/auth"` | | `not` | `!` | Logical NOT | `not http.req.host == "public.example.com"` | ### Value Types Rulekit expressions support various value types: * **Boolean**: `true`, `false` * **Number**: Integer or floating-point values (e.g., `200`, `1.5`) * **String**: Text enclosed in double quotes (e.g., `"example.com"`) * **Regex Pattern**: Patterns enclosed in slashes `/pattern/` or vertical bars `|pattern|` * **Array**: Values in square brackets (e.g., `[200, 201, 204]`) ### Using Rulekit Macros You can define reusable expression macros in the `rulekit` section of your configuration: ```yaml rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 - name: is_api_request expr: http.req.path matches /^\/api\// - name: is_production expr: src.pod.labels.env == "production" || src.container.labels.env == "production" stacks: default_stack: plugins: - type: http_capture config: level: summary format: text rules: - name: "Capture production API errors" expr: is_production() && is_api_request() && is_error() level: full - name: "Debug all errors" expr: is_error() level: headers ``` {% hint style="warning" %} **Known Limitation:** The `not` operator does not currently work in rule expressions or macros. Use explicit negation with `!=` and `&&` operators instead. For example, instead of: ```yaml - name: is_not_production expr: not (src.pod.labels.env == "production") ``` Use: ```yaml - name: is_not_production expr: src.pod.labels.env != "production" ``` {% endhint %} ## Complete Configuration Example Here's a comprehensive example that demonstrates various plugin features: ```yaml version: 2 # Define reusable macros rulekit: macros: - name: is_4xx expr: http.res.status >= 400 && http.res.status < 500 - name: is_5xx expr: http.res.status >= 500 && http.res.status < 600 - name: is_debug_enabled expr: src.container.labels.debug == "true" || src.pod.labels.debug == "true" services: event_stores: - id: console_stdout type: stdout object_stores: - id: minio type: s3 endpoint: minio.internal:9000 bucket: qpoint-objects region: us-east-1 access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY stacks: production_stack: plugins: - type: http_capture config: level: summary format: text rules: # Debug containers with debug label - name: "Debug mode capture" expr: is_debug_enabled() level: full # Capture errors from specific service - name: "Payment service errors" expr: http.req.host == "payment.api.com" && (is_4xx() || is_5xx()) level: full # Monitor external API calls - name: "External API monitoring" expr: http.req.host matches /\.(googleapis\.com|amazonaws\.com)$/ && http.req.method == "POST" level: headers tap: direction: egress ignore_loopback: true audit_include_dns: true http: stack: production_stack ``` ## Additional Resources * **Rulekit Documentation**: For detailed information about rule expressions, operators, and advanced features, visit the [Rulekit GitHub repository](https://github.com/qpoint-io/rulekit) * **Rulekit Examples**: The repository includes an interactive CLI demo tool for testing rule expressions # Traffic Capture Settings The `tap` section in your Qtap configuration controls what traffic is processed. This section provides fine-grained control over which network connections to monitor, what traffic to include or exclude, and how to apply different processing stacks to specific domains. ### Basic Configuration Here's the structure of a basic `tap` configuration: ```yaml tap: direction: egress ignore_loopback: false audit_include_dns: true http: stack: default_stack ``` ### Traffic Direction Settings The `direction` parameter controls which traffic flows Qtap captures: | Option | Description | | ----------------- | ---------------------------------------------------------- | | `egress` | Captures all outgoing traffic (both internal and external) | | `egress-external` | Only captures traffic to external networks | | `egress-internal` | Only captures traffic to internal networks | | `ingress` | Only captures inbound traffic | | `all` | Captures all traffic (inbound and outbound) | ```yaml tap: direction: egress-external # Only capture traffic going to external networks ``` ### Network Interface Settings #### Loopback Traffic The `ignore_loopback` parameter controls whether traffic on the loopback interface (localhost) is captured: ```yaml tap: ignore_loopback: true # Don't capture localhost traffic ``` * When set to `true`, traffic to addresses like `127.0.0.1` or `localhost` is ignored * When set to `false`, local traffic is included in capture #### DNS Traffic The `audit_include_dns` parameter controls whether DNS queries are captured: ```yaml tap: audit_include_dns: true # Include DNS traffic in audit logs ``` This is useful for: * Troubleshooting DNS resolution issues * Tracking which domains your applications are trying to reach * Identifying potential DNS-based data exfiltration ### Process Filtering The `filters` section allows you to control which processes Qtap monitors: ```yaml tap: filters: groups: - kubernetes - qpoint custom: - exe: /usr/bin/curl strategy: exact - exe: /usr/bin/python3 strategy: prefix - exe: .*ruby.* strategy: regex ``` #### Predefined Process Groups The `groups` parameter allows you to ignore specific types of processes: | Group | Description | | ------------ | ------------------------------------------------- | | `kubernetes` | Standard Kubernetes processes | | `container` | Container runtime processes | | `gke` | Google Kubernetes Engine processes | | `eks` | Amazon Elastic Kubernetes Service processes | | `qpoint` | Qpoint's own processes (prevents self-monitoring) | When a group is included in the list, traffic from those processes is ignored. #### Custom Process Filters The `custom` parameter allows you to define specific processes to ignore: ```yaml custom: - exe: /usr/bin/curl strategy: exact ``` Each entry requires: * `exe`: The executable path * `strategy`: Matching strategy, which can be: * `exact`: Exact path match * `prefix`: Path prefix match * `regex`: Regular expression match **Strategy Examples** **Exact Match** - Blocks only the specific path: ```yaml filters: custom: - exe: /usr/bin/curl strategy: exact ``` * Blocks: `/usr/bin/curl` * Does NOT block: `/usr/local/bin/curl`, `/bin/curl`, `/opt/curl/bin/curl` **Prefix Match** - Blocks anything starting with the prefix: ```yaml filters: custom: - exe: /usr/bin/ strategy: prefix ``` * Blocks: `/usr/bin/curl`, `/usr/bin/wget`, `/usr/bin/python3` * Does NOT block: `/usr/local/bin/curl`, `/bin/sh` **Regex Match** - Blocks paths matching the regex pattern: ```yaml filters: custom: - exe: .*curl.* strategy: regex ``` * Blocks: `/usr/bin/curl`, `/usr/local/bin/curl`, `/opt/my-curl-app/bin/app` * Does NOT block: `/usr/bin/wget`, `/usr/bin/python3` **Common Use Cases** **Development - Filter CLI tools to reduce noise:** ```yaml filters: custom: # Block common HTTP clients - exe: /usr/bin/curl strategy: exact - exe: /usr/bin/wget strategy: exact # Block package managers - exe: /usr/bin/apt strategy: exact - exe: /usr/bin/yum strategy: exact ``` **Production - Filter system processes:** ```yaml filters: groups: - kubernetes - qpoint custom: # Block monitoring agents - exe: /opt/datadog-agent strategy: prefix - exe: /usr/bin/node_exporter strategy: exact # Block backup tools - exe: .*backup.* strategy: regex ``` **Container environments - Filter by container tools:** ```yaml filters: custom: # Docker containers often have curl at /usr/bin/curl - exe: /usr/bin/curl strategy: exact # Alpine-based containers - exe: /bin/wget strategy: exact ``` **Multiple installations - Use regex to catch all variants:** ```yaml filters: custom: # Catch curl wherever it's installed - exe: .*curl.* strategy: regex # Catch Python anywhere (python, python3, /opt/python, etc.) - exe: .*python.* strategy: regex ``` **Finding the correct path:** To determine what path Qtap sees for a process, check the logs: ```bash # Start Qtap and generate traffic docker logs qtap-container 2>&1 | grep '"exe"' | head -20 # Look for exe field in output: # "exe":"/usr/bin/curl" <- This is what you filter on ``` Or find the path on your system: ```bash which curl # Shows: /usr/bin/curl readlink -f $(which curl) # Shows real path if it's a symlink ``` ### Domain-Specific Rules The `endpoints` section allows you to apply different processing stacks to specific domains or patterns: ```yaml tap: http: stack: default_stack # Default stack for most traffic endpoints: - domain: 'api.payment-processor.com' http: stack: payment_stack # Special stack for payment processor traffic - domain: 'internal-api.example.com' http: stack: debug_stack # Debug stack for internal API traffic ``` Each endpoint entry consists of: * `domain`: The domain pattern to match (supports wildcards) * `http.stack`: The stack to apply for matching HTTP traffic This allows you to: * Apply detailed logging to specific APIs * Handle sensitive traffic differently * Focus debugging efforts on problematic domains ### Complete Example Here's a comprehensive example that demonstrates all the main features: ```yaml tap: direction: egress ignore_loopback: true audit_include_dns: true http: stack: default_stack filters: groups: - qpoint - kubernetes custom: - exe: /usr/bin/wget strategy: exact - exe: /usr/bin/curl strategy: exact endpoints: - domain: 'api.github.com' http: stack: debug_stack - domain: 'api.google.com' http: stack: debug_stack ``` This configuration: 1. Captures all outgoing traffic (`direction: egress`) 2. Ignores localhost traffic (`ignore_loopback: true`) 3. Includes DNS queries in audit logs (`audit_include_dns: true`) 4. Applies the `default_stack` to most HTTP traffic 5. Ignores traffic from Kubernetes and Qpoint processes 6. Ignores traffic from specific executables (wget, curl) 7. Applies the `debug_stack` to all traffic to `api.github.com` and `api.google.com` domains # Custom Tagging Custom tags allow you to attach metadata to captured connections based on process environment variables, container labels, or Kubernetes labels and annotations. This is especially useful for: * **Multi-tenant environments**: Tag traffic by team, customer, or tenant * **Environment identification**: Distinguish between production, staging, and development traffic * **Cost allocation**: Track usage by department, project, or business unit * **Service organization**: Identify traffic by service name, version, or component * **Qplane filtering**: Filter and organize captured traffic in the Qplane UI by custom dimensions Custom tags appear in the `tags` object of **Connection events** (event store) alongside default tags like `bin`, `host`, `ip`, `protocol`, and `strategy`. Tags are connection-level metadata and do not appear in HTTP transaction objects (object store). ## Configuration Custom tags are defined in the `tags` section of your qtap configuration: ```yaml tags: - key: team source: env location: TEAM - key: service source: container.label location: service - key: environment source: k8s.label location: env ``` ### Field Reference | Field | Type | Required | Description | | ---------- | ------ | -------- | ------------------------------------------------------------------------------- | | `key` | string | Yes | The name of the tag as it will appear in the `tags` object | | `source` | string | Yes | Where to extract the tag value from (see [Tag Sources](#tag-sources)) | | `location` | string | Yes | The specific environment variable name, label key, or annotation key to extract | ### Tag Sources Qtap supports extracting tag values from multiple sources: | Source | Description | Example | | ----------------- | ----------------------------------------------- | ----------------------------------------------------------------------------- | | `env` | Environment variable from the monitored process |

source: env
location: APP\_NAME

| | `container.label` | Docker container label |

source: container.label
location: service

| | `k8s.label` | Kubernetes pod label |

source: k8s.label
location: team

| | `k8s.annotation` | Kubernetes pod annotation |

source: k8s.annotation
location: version

| ## Examples ### Environment Variables Extract tags from environment variables set on your applications: ```yaml version: 2 services: event_stores: - type: stdout tags: - key: team source: env location: TEAM - key: app source: env location: APP_NAME - key: environment source: env location: ENVIRONMENT stacks: default_stack: plugins: - type: http_capture config: level: summary format: json tap: direction: egress ignore_loopback: true http: stack: default_stack ``` When you run your application with environment variables: ```bash TEAM=platform APP_NAME=api-gateway ENVIRONMENT=production python ./app.py ``` The captured connections will include: ```json { "tags": { "app": ["api-gateway"], "bin": ["python"], "environment": ["production"], "host": ["qpoint"], "ip": ["192.168.1.100"], "protocol": ["http2"], "strategy": ["observe"], "team": ["platform"] } } ``` ### Container Labels Extract tags from Docker container labels: ```yaml tags: - key: service source: container.label location: service - key: version source: container.label location: version ``` Run containers with labels: ```bash docker run --label service=web-frontend --label version=v2.1.0 myapp:latest ``` Tags in captured connections: ```json { "tags": { "service": ["web-frontend"], "version": ["v2.1.0"], "bin": ["node"], "host": ["qpoint"] } } ``` ### Kubernetes Labels Extract tags from Kubernetes pod labels: ```yaml tags: - key: team source: k8s.label location: team - key: app source: k8s.label location: app ``` Pod manifest with labels: ```yaml apiVersion: v1 kind: Pod metadata: name: api-server labels: app: api-server team: backend spec: containers: - name: api image: myapp:latest ``` Tags in captured connections: ```json { "tags": { "app": ["api-server"], "team": ["backend"], "bin": ["node"], "host": ["k8s-node-01"] } } ``` ### Kubernetes Annotations Extract tags from Kubernetes pod annotations: ```yaml tags: - key: version source: k8s.annotation location: app.version - key: owner source: k8s.annotation location: owner ``` Pod manifest with annotations: ```yaml apiVersion: v1 kind: Pod metadata: name: api-server annotations: app.version: "2.1.0" owner: "platform-team" spec: containers: - name: api image: myapp:latest ``` ### Combining Tag Sources You can combine tags from multiple sources: ```yaml tags: # From environment variables - key: environment source: env location: ENVIRONMENT # From container labels - key: service source: container.label location: service # From Kubernetes labels - key: team source: k8s.label location: team # From Kubernetes annotations - key: version source: k8s.annotation location: version ``` All tags from different sources are merged together in the Connection event. ## Static Tags (CLI Flag) In addition to dynamic tags extracted from the YAML configuration, you can add static tags to all connections using the `--tags` CLI flag: ```bash qtap --config=/path/to/qtap.yaml \ --tags='region:us-west,deployment:production,cluster:main' ``` {% hint style="info" %} **Format:** The `--tags` flag uses `key:value` pairs separated by commas (not `key=value`). {% endhint %} Static tags are useful for: * Identifying the qtap instance or node * Tagging by region, datacenter, or availability zone * Marking deployment environment at the infrastructure level Static tags and dynamic tags (from YAML) are merged together: ```json { "tags": { "app": ["api-gateway"], // from env var (YAML) "team": ["platform"], // from env var (YAML) "region": ["us-west"], // from CLI --tags "deployment": ["production"], // from CLI --tags "cluster": ["main"], // from CLI --tags "bin": ["python"], // default "host": ["qpoint"], // default "protocol": ["http2"], // default "strategy": ["observe"] // default } } ``` ## Tag Behavior ### Missing Values If a tag's source value doesn't exist, the tag is simply omitted from the output (no errors are logged): ```yaml tags: - key: team source: env location: TEAM # If TEAM env var not set, tag won't appear ``` If the `TEAM` environment variable is not set on the process, the `team` tag won't appear in the Connection event. ### Tag Merging When the same tag key is defined in multiple sources, **all values are merged into an array**: **Example with environment variable and CLI flag:** ```yaml tags: - key: team source: env location: TEAM ``` ```bash TEAM=platform-yaml ./my-app ``` ```bash qtap --tags='team:platform-cli' ... ``` **Result:** ```json { "tags": { "team": ["platform-yaml", "platform-cli"] } } ``` This applies to all tag sources: * YAML-defined tags (env, container.label, k8s.label, k8s.annotation) * CLI `--tags` flag * Default qtap tags (bin, host, ip, protocol, strategy) Even default tags can be extended: ```bash qtap --tags='bin:my-custom-bin,host:my-custom-host' ... ``` **Result:** ```json { "tags": { "bin": ["curl", "my-custom-bin"], "host": ["qpoint", "my-custom-host"] } } ``` All tag values from all sources are preserved in the array. ## Use Cases ### Multi-Tenant SaaS Platform Tag traffic by tenant and tier for cost allocation: ```yaml tags: - key: tenant_id source: env location: TENANT_ID - key: tier source: env location: SERVICE_TIER ``` Run application: ```bash TENANT_ID=customer-123 SERVICE_TIER=premium python ./app.py ``` ### Microservices Environment Identify traffic by service, version, and team: ```yaml tags: - key: service source: k8s.label location: app - key: version source: k8s.annotation location: version - key: team source: k8s.label location: team ``` ### Cost Tracking by Department Tag traffic for cost allocation across departments: ```yaml tags: - key: department source: env location: DEPARTMENT - key: cost_center source: env location: COST_CENTER - key: project source: env location: PROJECT_ID ``` Run application: ```bash DEPARTMENT=engineering COST_CENTER=R001 PROJECT_ID=api-v2 ./my-app ``` ### Environment-Specific Tagging Distinguish between production, staging, and development: ```yaml tags: - key: environment source: env location: ENV - key: region source: env location: AWS_REGION ``` Run in production: ```bash ENV=production AWS_REGION=us-west-2 ./my-app ``` Run in staging: ```bash ENV=staging AWS_REGION=us-east-1 ./my-app ``` ## Viewing Tags in Qplane When qtap is connected to Qplane, custom tags appear in the Qplane UI and can be used for: * **Filtering connections**: Filter by team, environment, service, etc. * **Grouping traffic**: Group connections by custom dimensions * **Creating dashboards**: Visualize traffic patterns by custom tags * **Setting alerts**: Alert on traffic patterns for specific tags Custom tags make it easy to organize and filter large volumes of captured traffic across multiple teams, services, and environments. ## Best Practices ### Security Considerations {% hint style="warning" %} **Avoid tagging sensitive data**: Don't extract environment variables or labels that contain secrets, credentials, or personally identifiable information (PII). {% endhint %} ```yaml # AVOID - these could contain sensitive data tags: - key: api_key source: env location: API_KEY # ❌ Never tag secrets - key: user_email source: env location: USER_EMAIL # ❌ Avoid PII # Good - safe metadata tags: - key: service source: env location: SERVICE_NAME # ✅ Safe - key: team source: env location: TEAM # ✅ Safe ``` ### Use Static Tags for Infrastructure Use the `--tags` CLI flag for infrastructure-level tags that don't change per process: ```bash # Tag by region and cluster qtap --tags='region:us-west-2,cluster:production,zone:a' # Tag by datacenter and rack qtap --tags='datacenter:dc1,rack:r42' ``` Use YAML tags for application-level tags that vary per process: ```yaml # Tag by application attributes tags: - key: service source: env location: SERVICE_NAME - key: version source: env location: APP_VERSION ``` ## Complete Example Here's a complete configuration combining multiple tag sources: ```yaml version: 2 services: event_stores: - type: s3 config: bucket: my-qtap-events region: us-west-2 object_stores: - type: s3 config: bucket: my-qtap-objects region: us-west-2 tags: # Application-level tags from environment - key: service source: env location: SERVICE_NAME - key: version source: env location: APP_VERSION - key: environment source: env location: ENVIRONMENT # Team ownership from Kubernetes - key: team source: k8s.label location: team - key: cost_center source: k8s.annotation location: cost-center stacks: default_stack: plugins: - type: http_capture config: level: summary format: json tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: default_stack ``` Start qtap with static infrastructure tags: ```bash qtap --config=/etc/qtap/config.yaml \ --tags='region:us-west-2,cluster:production' ``` Run your application with environment variables: ```bash SERVICE_NAME=api-gateway \ APP_VERSION=2.1.0 \ ENVIRONMENT=production \ python ./app.py ``` The resulting Connection events will include all tags: ```json { "meta": { "connectionId": "conn-123", "endpointId": "api.example.com" }, "tags": { "service": ["api-gateway"], "version": ["2.1.0"], "environment": ["production"], "team": ["platform"], "cost_center": ["engineering-001"], "region": ["us-west-2"], "cluster": ["production"], "bin": ["python"], "host": ["k8s-node-01"], "ip": ["10.0.1.50"], "protocol": ["http2"], "strategy": ["observe"] }, "timestamp": "2025-11-03T18:30:00Z", "direction": "egress-external", "source": { "exe": "/usr/bin/python3", "address": { "ip": "10.0.1.50", "port": 45678 } }, "destination": { "address": { "ip": "52.1.2.3", "port": 443 } } } ``` # Configuration Examples ### 1. Development ```yaml # Development configuration to see all HTTP traffic # Outputs everything to console for immediate debugging version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: development: plugins: - type: http_capture config: level: full # (none|summary|headers|full) format: text # (json|text) tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: false # Include localhost traffic audit_include_dns: true # See DNS queries too http: stack: development # Test with: curl -X POST https://httpbin.org/post -d "test=data" -H "Content-Type: application/json" ``` ### 2. API Error Debugging ```yaml # Production-safe configuration that only captures when things go wrong # Shows summary for all traffic, full details for errors version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: error_capture: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info for all requests format: json # (json|text) - Structured for analysis rules: # Capture everything for client errors - name: "Debug 4XX errors" expr: http.res.status >= 400 && http.res.status < 500 level: full # Capture everything for server errors - name: "Debug 5XX errors" expr: http.res.status >= 500 level: full tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # Skip internal traffic audit_include_dns: false http: stack: error_capture # Test with: curl https://httpbin.org/status/500 # Triggers 5XX error (full capture) # Test with: curl https://httpbin.org/status/404 # Triggers 4XX error (full capture) # Test with: curl https://httpbin.org/get # Normal request (summary only) ``` ### 3. Domain-Specific Monitoring ```yaml # Different capture levels for different domains # Shows summary by default, more for specific domains version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: multi_domain: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info format: json # (json|text) rules: # Full capture for test environment - name: "Debug httpbin.org" expr: http.req.host == "httpbin.org" level: full # Headers for GitHub API - name: "GitHub API monitoring" expr: http.req.host == "api.github.com" level: headers # Full capture for test APIs - name: "Test API services" expr: http.req.host matches /\.(typicode|mockapi)\.com$/ level: full # Skip health checks - name: "Ignore health checks" expr: http.req.path in ["/health", "/ping", "/metrics"] level: none tap: direction: egress-external # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: false http: stack: multi_domain # Test with: curl https://httpbin.org/get # Full capture # Test with: curl https://api.github.com/users # Headers only # Test with: curl https://jsonplaceholder.typicode.com/posts/1 # Full capture # Test with: curl https://httpbin.org/status/200 # Summary only (not a health path) ``` ### 4. Container-Based Debugging ```yaml # Debug specific containers in Docker/Kubernetes # Shows summary for all, full for specific containers version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: container_debug: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info format: text # (json|text) rules: # Debug specific container by name - name: "Debug payment service" expr: src.container.name == "payment-service" level: full # Debug by container image - name: "Debug Redis containers" expr: src.container.image contains "redis" level: headers # Debug containers with debug label - name: "Debug labeled containers" expr: src.container.labels.debug == "true" level: full # Debug specific process - name: "Debug curl requests" expr: src.process.binary == "curl" level: full tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: false # Include container-to-container audit_include_dns: false http: stack: container_debug # Test with: docker run --name payment-service --rm curlimages/curl https://httpbin.org/get # Test with: docker run --label debug=true --rm curlimages/curl https://httpbin.org/post -d "test" # Test with: curl https://httpbin.org/get # If running from host, captures curl binary (full) ``` ### 5. Kubernetes Pod Debugging ```yaml # Debug specific pods or namespaces in Kubernetes # Shows summary by default, more for specific pods version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: k8s_debug: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info format: text # (json|text) rules: # Debug specific pod pattern - name: "Debug frontend pods" expr: src.pod.name matches /^frontend-deployment-/ level: full # Debug by namespace - name: "Staging namespace debugging" expr: src.pod.namespace == "staging" level: headers # Debug by pod label - name: "Debug enabled pods" expr: src.pod.labels.debug == "true" level: full # Debug specific app - name: "Debug payment app pods" expr: src.pod.labels.app == "payment-service" level: full tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: false # Include pod-to-pod traffic audit_include_dns: false http: stack: k8s_debug # Test with: kubectl run test-curl --image=curlimages/curl --rm -it -- https://httpbin.org/get # Test with: kubectl run test-debug --labels="debug=true" --image=curlimages/curl --rm -it -- https://httpbin.org/post -d "test" # Test with: kubectl -n staging run test-staging --image=curlimages/curl --rm -it -- https://httpbin.org/get ``` ### 6. Security Monitoring ```yaml # Detect and capture potentially malicious traffic # Shows summary for normal traffic, full for suspicious patterns version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: security: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic monitoring format: json # (json|text) rules: # Capture authentication failures - name: "Authentication failures" expr: http.res.status == 401 || http.res.status == 403 level: full # SQL injection patterns in URL - name: "SQL injection attempts" expr: http.req.url matches /(\-\-|union\s+select|drop\s+table)/i level: full # Path traversal attempts - name: "Path traversal" expr: http.req.path contains "../" || http.req.path contains "..%2F" level: full # Suspicious user agents - name: "Malicious scanners" expr: http.req.headers.user-agent matches /(sqlmap|nikto|havij|acunetix)/i level: full tap: direction: ingress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: true # DNS can reveal C&C communication http: stack: security # Test with: curl https://httpbin.org/status/401 # Auth failure (full) # Test with: curl "https://httpbin.org/get?id=1' OR '1'='1' --" # SQL injection pattern (full) # Test with: curl https://httpbin.org/../../../etc/passwd # Path traversal (full) # Test with: curl https://httpbin.org/get -H "User-Agent: sqlmap/1.0" # Suspicious agent (full) # Test with: curl https://httpbin.org/get # Normal request (summary) ``` ### 7. API Endpoint Monitoring ```yaml # Monitor specific API endpoints with different rules # Shows summary by default, more for sensitive endpoints version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: api_monitoring: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info format: json # (json|text) rules: # Full capture for authentication endpoints - name: "Auth endpoints" expr: http.req.path in ["/login", "/logout", "/oauth/token"] level: full # Capture payment endpoints - name: "Payment processing" expr: http.req.path contains "/payment" || http.req.path contains "/checkout" level: headers # API v2 endpoints - name: "API v2 monitoring" expr: http.req.path matches /^\/api\/v2\// level: headers # GraphQL queries - name: "GraphQL monitoring" expr: http.req.path == "/graphql" && http.req.method == "POST" level: full tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: false http: stack: api_monitoring # Test with: curl https://httpbin.org/anything/login -X POST -d "user=test" # Auth endpoint (full) # Test with: curl https://httpbin.org/anything/payment/process # Payment endpoint (details) # Test with: curl https://httpbin.org/anything/api/v2/users # API v2 endpoint (details) # Test with: curl https://httpbin.org/anything/graphql -X POST -d '{"query":"test"}' # GraphQL (full) # Test with: curl https://httpbin.org/get # Normal request (summary) ``` ### 8. Method-Based Capture ```yaml # Different capture levels based on HTTP methods # Shows summary for GET, more for mutations version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: method_based: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) rules: # Full capture for mutations - name: "Capture POST requests" expr: http.req.method == "POST" level: full - name: "Capture PUT requests" expr: http.req.method == "PUT" level: full - name: "Capture DELETE requests" expr: http.req.method == "DELETE" level: full # Headers only for PATCH - name: "Monitor PATCH requests" expr: http.req.method == "PATCH" level: headers # Skip OPTIONS preflight - name: "Skip CORS preflight" expr: http.req.method == "OPTIONS" level: none tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: false http: stack: method_based # Test with: curl -X GET https://httpbin.org/get # Summary only # Test with: curl -X POST https://httpbin.org/post -d "data=test" # Full capture # Test with: curl -X PUT https://httpbin.org/put -d "data=update" # Full capture # Test with: curl -X DELETE https://httpbin.org/delete # Full capture # Test with: curl -X PATCH https://httpbin.org/patch -d "data=patch" # Headers only # Test with: curl -X OPTIONS https://httpbin.org/anything # No output (none) ``` ### 9. Header-Based Debugging ```yaml # Capture based on request/response headers # Shows summary by default, more for specific headers version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: header_based: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) rules: # Debug requests with trace headers - name: "Trace enabled requests" expr: http.req.headers.x-trace-enabled == "true" level: full # Capture requests with specific API keys - name: "Debug API key" expr: http.req.headers.x-api-key == "debug-key-123" level: full # Monitor JWT authentication - name: "JWT requests" expr: http.req.headers.authorization contains "Bearer" level: headers # Debug based on response headers - name: "Capture rate limited requests" expr: http.res.headers.x-ratelimit-remaining == "0" level: full tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: false http: stack: header_based # Test with: curl https://httpbin.org/get # Summary only # Test with: curl https://httpbin.org/get -H "X-Trace-Enabled: true" # Full capture # Test with: curl https://httpbin.org/get -H "X-API-Key: debug-key-123" # Full capture # Test with: curl https://httpbin.org/bearer -H "Authorization: Bearer abc123" # Details capture # Test with: curl https://httpbin.org/response-headers?X-RateLimit-Remaining=0 # Response header trigger (full) ``` ### 10. Production with Rulekit Macros ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: # Status code groups - name: is_error expr: http.res.status >= 400 - name: is_client_error expr: http.res.status >= 400 && http.res.status < 500 - name: is_server_error expr: http.res.status >= 500 # Create specific macros instead of parameterized ones - name: is_httpbin expr: http.req.host == "httpbin.org" || http.req.host matches /\.httpbin\.org$/ - name: is_github expr: http.req.host == "github.com" || http.req.host matches /\.github\.com$/ - name: is_googleapis expr: http.req.host matches /\.googleapis\.com$/ # Path patterns - name: is_api_call expr: http.req.path matches /^\/api\// - name: is_static_asset expr: http.req.path matches /\.(css|js|png|jpg|ico|woff|ttf)$/ # Authentication - name: is_authenticated expr: http.req.headers.authorization != "" stacks: production: plugins: - type: http_capture config: level: summary format: json rules: - name: "All errors" expr: is_error() level: full - name: "API errors" expr: is_api_call() && is_error() level: full - name: "Production monitoring" expr: is_httpbin() && is_server_error() level: full - name: "Ignore static files" expr: is_static_asset() level: none - name: "Auth failures" expr: is_authenticated() && http.res.status == 401 level: full - type: report_usage tap: direction: all ignore_loopback: true audit_include_dns: false http: stack: production ``` ### 11. Multi-Tenant with Custom Tags ```yaml # Tag traffic by team, service, and environment for multi-tenant observability # Custom tags enable filtering and cost allocation in Qplane version: 2 services: event_stores: - type: s3 config: bucket: my-qtap-events region: us-west-2 object_stores: - type: s3 config: bucket: my-qtap-objects region: us-west-2 tags: # Extract from environment variables - key: service source: env location: SERVICE_NAME - key: version source: env location: APP_VERSION - key: environment source: env location: ENVIRONMENT # Extract from container labels - key: team source: container.label location: team # Extract from Kubernetes labels - key: app source: k8s.label location: app - key: cost_center source: k8s.annotation location: cost-center stacks: multi_tenant: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) rules: # Full capture for errors - name: "Capture errors" expr: http.res.status >= 400 level: full tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true audit_include_dns: false http: stack: multi_tenant # Run application with environment variables: # SERVICE_NAME=api-gateway APP_VERSION=2.1.0 ENVIRONMENT=production python ./app.py # # Or in Docker with labels: # docker run --label team=platform myapp:latest # # Or in Kubernetes with labels and annotations: # kubectl run api-server --labels="app=api-server,team=backend" \ # --annotations="cost-center=eng-001" # # Start qtap with static infrastructure tags: # qtap --config=/path/to/config.yaml --tags='region:us-west-2,cluster:prod' # # Result: Connection events include all tags merged together: # "tags": { # "service": ["api-gateway"], // from env var # "version": ["2.1.0"], // from env var # "environment": ["production"], // from env var # "team": ["platform"], // from container label or k8s label # "app": ["api-server"], // from k8s label # "cost_center": ["eng-001"], // from k8s annotation # "region": ["us-west-2"], // from CLI --tags # "cluster": ["prod"], // from CLI --tags # "bin": ["python"], // default # "host": ["k8s-node-01"], // default # "protocol": ["http2"], // default # "strategy": ["observe"] // default # } ``` # Metrics Qtap exposes Prometheus-compatible metrics endpoints for monitoring HTTP traffic activity and agent health. These can be used to build your own dashboards with Grafana. We have a sample Grafana dashboard available [here](https://github.com/qpoint-io/qtap/blob/main/examples/dashboards/qtap-http-overview.json).
{% hint style="warning" %} **REQUIRED**: HTTP-level metrics (requests, responses, latency) require the `http_metrics` plugin in your stack configuration. Without this plugin, only connection-level metrics will be available. See [Configuration](#configuration-requirement) below. {% endhint %} ## Endpoints Qtap provides two metrics endpoints on `localhost:10001`: ### `/metrics` Application metrics for monitoring HTTP traffic observed by qtap: * Request rates and volumes * Error rates by status code * Response times and latency distributions * Request/response payload sizes * Traffic patterns by host, method, and protocol ### `/system/metrics` System metrics for monitoring qtap agent health: * eBPF program performance * Memory and CPU usage * Event processing rates * Internal error counts ### `/devtools/` (Optional) Interactive browser-based interface for debugging HTTP/S traffic in real-time. Requires `ENABLE_DEV_TOOLS=true` environment variable. See [DevTools - Interactive Traffic Inspection](/getting-started/qtap/configuration/devtools) for the complete guide. {% hint style="info" %} **Shared Port**: Metrics endpoints and DevTools all run on port `10001`: * `/metrics` - Application metrics (Prometheus) * `/system/metrics` - Agent health metrics * `/devtools/` - Interactive debugging UI (requires `ENABLE_DEV_TOOLS=true`) {% endhint %} ## Configuration Requirement {% hint style="danger" %} **CRITICAL**: To enable HTTP-level Prometheus metrics, you **must** add the `http_metrics` plugin to your stack configuration. {% endhint %} ### Required Configuration 
version: 2

# Storage Configuration - Everything to console
services:
  # Event metadata goes to stdout
  event_stores:
    - type: stdout
  
  # Object data (headers) goes to stdout
  object_stores:
    - type: stdout

# Processing Stack - Simple HTTP capture
stacks:
  starter_stack:
    plugins:
      # REQUIRED for prometheus endpoint
      - type: http_metrics
      # HTTP Capture plugin - outputs to console
      - type: http_capture
        config:
          level: headers  # Capture headers (use 'full' for bodies too)
          format: text    # Human-readable format (use 'json' for structured)

# Traffic Capture Settings
tap:
  direction: egress  # Capture outgoing traffic
  ignore_loopback: false  # Skip localhost traffic
  audit_include_dns: false  # Skip DNS queries for cleaner output
  http:
    stack: starter_stack
Without the `http_metrics` plugin, you will only see connection-level metrics like `qtap_connection_open_total`. The HTTP metrics like `qtap_http_requests_total` will not be available. ## Accessing Metrics ### From Host ```bash curl http://localhost:10001/metrics curl http://localhost:10001/system/metrics ``` ### From Docker When using `--network=host`, access directly from the host: ```bash curl http://localhost:10001/metrics ``` If NOT using host networking, expose port 10001: ```bash docker run -d --name qtap \ -p 10001:10001 \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/config.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --config="/app/config/qtap.yaml" ``` Then access: ```bash curl http://localhost:10001/metrics ``` ### From Kubernetes Forward the metrics port: ```bash kubectl port-forward pod/qtap-xxxxx 10001:10001 ``` Access locally: ```bash curl http://localhost:10001/metrics ``` ## Available Metrics ### HTTP Request Metrics | Metric | Type | Description | Labels | | --------------------------------------- | --------- | ------------------------------------ | ------------------------------------------------- | | `qtap_http_requests_total` | Counter | Total HTTP requests observed | `host`, `method`, `protocol`, `status_code` | | `qtap_http_requests_duration_ms_bucket` | Histogram | Request duration distribution | `host`, `method`, `protocol`, `status_code`, `le` | | `qtap_http_requests_duration_ms_sum` | Counter | Total request processing time (ms) | `host`, `method`, `protocol`, `status_code` | | `qtap_http_requests_duration_ms_count` | Counter | Number of requests measured | `host`, `method`, `protocol`, `status_code` | | `qtap_http_requests_size_bytes_bucket` | Histogram | Request payload size distribution | `host`, `method`, `protocol`, `status_code`, `le` | | `qtap_http_requests_size_bytes_sum` | Counter | Total request payload size (bytes) | `host`, `method`, `protocol`, `status_code` | | `qtap_http_requests_size_bytes_count` | Counter | Number of requests measured for size | `host`, `method`, `protocol`, `status_code` | ### HTTP Response Metrics | Metric | Type | Description | Labels | | ---------------------------------------- | --------- | ------------------------------------- | ------------------------------------------------- | | `qtap_http_responses_total` | Counter | Total HTTP responses observed | `host`, `method`, `protocol`, `status_code` | | `qtap_http_responses_duration_ms_bucket` | Histogram | Response duration distribution (TTFB) | `host`, `method`, `protocol`, `status_code`, `le` | | `qtap_http_responses_duration_ms_sum` | Counter | Total response time/TTFB (ms) | `host`, `method`, `protocol`, `status_code` | | `qtap_http_responses_duration_ms_count` | Counter | Number of responses measured | `host`, `method`, `protocol`, `status_code` | | `qtap_http_responses_size_bytes_bucket` | Histogram | Response payload size distribution | `host`, `method`, `protocol`, `status_code`, `le` | | `qtap_http_responses_size_bytes_sum` | Counter | Total response payload size (bytes) | `host`, `method`, `protocol`, `status_code` | | `qtap_http_responses_size_bytes_count` | Counter | Number of responses measured for size | `host`, `method`, `protocol`, `status_code` | ### Combined Metrics | Metric | Type | Description | Labels | | ------------------------------ | --------- | ----------------------------------------------- | ----------------------------- | | `qtap_http_duration_ms_bucket` | Histogram | Combined request+response duration distribution | None (aggregates all traffic) | | `qtap_http_duration_ms_sum` | Counter | Total combined duration (ms) | None | | `qtap_http_duration_ms_count` | Counter | Number of completed transactions | None | ### Connection-Level Metrics These are always available (no plugin required): | Metric | Type | Description | Labels | | ------------------------------------- | --------- | -------------------------------------- | ----------------------------------------------------------- | | `qtap_connection_open_total` | Counter | Total connections opened | `direction`, `remote_addr`, `remote_port` | | `qtap_connection_close_total` | Counter | Total connections closed | `direction`, `remote_addr`, `remote_port` | | `qtap_connection_active_total` | Gauge | Number of currently active connections | `direction`, `remote_addr`, `remote_port` | | `qtap_connection_protocol_total` | Counter | Connections by protocol | `protocol` | | `qtap_connection_bytes_sent_total` | Counter | Total bytes sent | `direction`, `remote_addr`, `remote_port` | | `qtap_connection_bytes_recv_total` | Counter | Total bytes received | `direction`, `remote_addr`, `remote_port` | | `qtap_connection_duration_ms` | Histogram | Connection duration distribution | `direction`, `remote_addr`, `remote_port`, `le` | | `qtap_connection_tls_handshake_total` | Counter | Total TLS handshakes completed | `direction`, `remote_addr`, `remote_port`, `sni`, `version` | ## Common Queries ### Request Rate ```promql rate(qtap_http_requests_total[5m]) ``` ### Error Rate ```promql sum(rate(qtap_http_requests_total{status_code=~"4..|5.."}[5m])) / sum(rate(qtap_http_requests_total[5m])) ``` ### Average Response Time ```promql rate(qtap_http_responses_duration_ms_sum[5m]) / rate(qtap_http_responses_duration_ms_count[5m]) ``` ### 95th Percentile Latency ```promql histogram_quantile(0.95, rate(qtap_http_requests_duration_ms_bucket[5m]) ) ``` ### Traffic by Host ```promql topk(10, sum by (host) (rate(qtap_http_requests_total[5m]))) ``` ### HTTP/2 Traffic ```promql sum(rate(qtap_http_requests_total{protocol="http2"}[5m])) ``` ### Average Request Size ```promql rate(qtap_http_requests_size_bytes_sum[5m]) / rate(qtap_http_requests_size_bytes_count[5m]) ``` ### Overall Transaction Duration ```promql # Average combined request+response time rate(qtap_http_duration_ms_sum[5m]) / rate(qtap_http_duration_ms_count[5m]) ``` ### Active Connections ```promql # Total active connections across all destinations sum(qtap_connection_active_total) ``` ### TLS Handshake Rate ```promql # Overall handshake rate sum(rate(qtap_connection_tls_handshake_total[5m])) ``` ## Label Cardinality Metrics include labels for `host`, `method`, `protocol`, and `status_code`. This enables powerful filtering but can increase cardinality in high-traffic environments. **To reduce cardinality:** 1. Filter noisy processes using [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings): ```yaml filters: groups: - qpoint custom: - exe: /usr/bin/noisy-app strategy: exact ``` 2. Use domain-specific capture to limit metrics to important services: ```yaml tap: endpoints: - domain: 'important-api.example.com' http: stack: monitored_stack ``` 3. Implement Prometheus recording rules to pre-aggregate metrics ## Troubleshooting ### No HTTP Metrics Appearing If you don't see `qtap_http_requests_total` or other HTTP metrics: 1. **Verify the `http_metrics` plugin is configured:** ```bash # Check your qtap config includes http_metrics plugin cat config.yaml | grep -A 5 "plugins:" ``` 2. **Restart qtap after adding the plugin:** ```bash docker restart qtap # Or for Kubernetes: kubectl rollout restart daemonset/qtap ``` 3. **Verify metrics are now available:** ```bash curl http://localhost:10001/metrics | grep "qtap_http_requests_total" ``` ### Metrics Show Zero Values If metrics exist but show zero counts despite HTTP traffic: 1. **Verify qtap is capturing traffic** - check logs for HTTP transactions 2. **Check capture direction** - ensure `tap.direction` matches your traffic flow 3. **Verify filters** - ensure you're not filtering out all traffic ### Only Connection Metrics Available If you only see `qtap_connection_*` metrics but no `qtap_http_*` metrics, the `http_metrics` plugin is missing from your configuration. ## Integration See the [Monitoring Qtap with Prometheus and Grafana](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) guide for step-by-step setup instructions. For a comprehensive metrics reference including alerting examples, see [Prometheus Metrics](/appendix/prometheus-metrics) in the Appendix. # DevTools QTap Dev Tools is a browser-based interface for debugging production API calls. It uses the same familiar visual approach as Chrome DevTools, but for a completely different use case: troubleshooting server-side HTTP/S traffic in real-time. When you need to see exactly what your application is sending to third-party APIs, or why an integration is failing, Dev Tools gives you instant visibility without deploying code changes. {% hint style="info" %} **Air-Gapped Ready**: DevTools runs entirely on `localhost` - no external services required. All data stays on your infrastructure, making it ideal for regulated environments and data sovereignty requirements. {% endhint %} *** ### How to Enable DevTools is built into the Qtap binary. You simply need to turn it on during startup using a flag or environment variable. | **Method** | **Syntax** | **Best For** | | ---------- | ----------------------- | ------------------------- | | CLI Flag | `--enable-dev-tools` | Binary usage, Quick demos | | Env Var | `ENABLE_DEV_TOOLS=true` | Docker, Kubernetes, Helm | Once enabled, the interface is accessible at: `http://localhost:10001/devtools` *** ### Deployment Options #### 1. Quick Start (Temporary) The fastest way to inspect traffic immediately. This downloads Qtap to `/tmp` and starts capturing. ```bash # Download and run with DevTools enabled curl -s https://get.qpoint.io/demo | sudo sh -s -- --enable-dev-tools ``` #### 2. Production Binary For persistent installation on a VM or bare metal server. ```bash # 1. Install Qtap curl -s https://get.qpoint.io/install | sudo sh # 2. Run with DevTools enabled sudo qtap --enable-dev-tools ``` #### 3. Docker Run Qtap as a privileged container to capture host traffic. ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info ``` #### 4. Kubernetes / Helm Add the environment variable to your Pod spec or Helm values. ```yaml # In your Pod spec or Helm values env: - name: ENABLE_DEV_TOOLS value: "true" ``` *See* [*Kubernetes Installation*](https://docs.qpoint.io/getting-started/qtap/installation/kubernetes-manifest) *for full manifest details.* *** ### Core Use Cases DevTools is designed for moments when you need to see "ground truth" data immediately. #### 1. API Integration Debugging The Problem: A third-party API is returning errors, but your logs don't show the full request context. DevTools Insight: * See the exact Authorization header sent (e.g., Did we send `Bearer` when they expected `Basic`?). * Read the full error response body which often contains specific error codes hidden by application logic. * View request timing to pinpoint latency. #### 2. Production Incident Response The Problem: A critical sync process is failing in production, and you can't wait for a deployment cycle to add debug logging. DevTools Insight: * SSH into the box, start Qtap, and watch the failure occur in real-time. * Identify issues like 404 Not Found (endpoint changed) or DNS resolution failures instantly. * Result: Diagnose the issue in minutes without touching application code. #### 3. Third-Party Service Changes The Problem: An integration that worked yesterday is broken today. Did the provider change their API? DevTools Insight: * Compare your application's requests (unchanged) against the provider's responses. * Spot new requirements, such as a missing `X-API-Version` header or a change in the required payload structure. #### 4. Webhook Debugging The Problem: You aren't receiving webhooks from a provider. Is the network blocking them, or is your app rejecting them? DevTools Insight: * Verify if the webhook actually reaches the network interface. * Inspect the signature headers and payload format to ensure they match your validation logic. *** ### Remote Access (SSH Tunnel) Since DevTools listens on `localhost` for security, use an SSH tunnel to access it on a remote server from your local device. ```bash # 1. Create tunnel (run on your laptop) ssh -L 10001:localhost:10001 -N user@production-server # 2. Open browser locally open http://localhost:10001/devtools ``` *** Note: You can use DevTools (for real-time viewing) and S3 Storage (for permanent archiving) simultaneously by configuring both in your Qtap settings. # CLI This guide covers the essential command-line options for configuring and running Qtap. ## Basic Syntax ```bash qtap [flags] ``` ## Essential Flags **Configuration** | Flag | Description | | ----------------------- | ------------------------------------------------- | | `--config=` | Path to YAML configuration file or HTTP/HTTPS URL | | `--registration-token=` | Token for cloud-connected mode | **Logging Options**
FlagDefaultDescription
--log-level=infoSet logging detail level (debug, info, warn, error)
--log-encoding=consoleLog format (console, json)
**TLS Configuration**
FlagDefaultDescription
--tls-probes=nodetls,openssl,gotls,javasslTLS libraries to monitor (nodetls,openssl,gotls,javassl)
**Tagging**
FlagDescription
--tags=Deployment tags for categorization (comma-separated key:value pairs)
**Debugging**
FlagDescription
--enable-dev-toolsEnable browser-based DevTools UI at http://localhost:10001/devtools
**Other** | Flag | Description | | --------------- | ------------------------ | | `-v, --version` | Display Qtap version | | `-h, --help` | Display help information | ## Common Usage Examples **Running with a Local Configuration File** ```bash qtap --config=/path/to/qpoint.yaml ``` **Running with a Remote Configuration File** ```bash qtap --config=https://config.example.com/qpoint.yaml ``` **Running with** [**Qplane**](https://claude.ai/qplane/) ```bash qtap --registration-token=YOUR_TOKEN ``` **Adjusting Log Level for Troubleshooting** ```bash qtap --config=/path/to/qpoint.yaml --log-level=debug ``` **Using JSON Log Format for Structured Logging** ```bash qtap --config=/path/to/qpoint.yaml --log-encoding=json ``` **Monitoring Specific TLS Libraries** ```bash qtap --config=/path/to/qpoint.yaml --tls-probes=openssl,gotls,javassl ``` **Cloud-Connected with Enhanced Debugging** ```bash qtap --registration-token=YOUR_TOKEN --log-level=debug --log-encoding=json ``` **Adding Deployment Tags** ```bash qtap --registration-token=YOUR_TOKEN --tags="env:prod,region:us-east" ``` **Enabling DevTools for Interactive Debugging** ```bash qtap --config=/path/to/qpoint.yaml --enable-dev-tools # Access at http://localhost:10001/devtools ``` **Demo Mode with DevTools** ```bash curl -s https://get.qpoint.io/demo | sudo sh -s -- --enable-dev-tools # Qtap runs in /tmp with DevTools enabled ``` **Remote Config with Debugging** ```bash qtap --config=https://config.example.com/qpoint.yaml --log-level=debug ``` **Checking the Qtap Version** ```bash qtap --version ``` ## Configuration Sources The `--config` flag supports multiple sources: * **Local files**: `/path/to/qpoint.yaml` * **Remote URLs**: `https://config.example.com/qpoint.yaml` Remote configurations are automatically cached locally and support reload functionality via SIGHUP. #### TLS Probes Available | Probe | Description | | --------- | ---------------------------- | | `openssl` | Monitor OpenSSL traffic | | `gotls` | Monitor Go TLS traffic | | `nodetls` | Monitor Node.js TLS traffic | | `javassl` | Monitor Java SSL/TLS traffic | To monitor multiple libraries, combine them with commas (no spaces): ``` --tls-probes=openssl,gotls,javassl ``` ## DevTools Interactive Debugging The `--enable-dev-tools` flag enables a browser-based interface for real-time HTTP/S traffic inspection. See the [DevTools Guide](/getting-started/qtap/configuration/devtools) for complete documentation including: * Quick start with demo mode * SSH tunnel setup for remote servers * Use cases and troubleshooting workflows * Screenshots and visual walkthrough # Qplane Qplane is Qpoint's cloud control plane that provides centralized management for all your Qtap agents. Effortlessly manage configuration and analyze traffic patterns across your entire infrastructure from a single dashboard. > **Note**: For YAML-based local deployments, see [Qtap Documentation.](/getting-started/qtap) {% embed url="" %} ### Key Features * **Centralized Management**: Configure and monitor hundreds of Qtap agents from one place * **Visual Dashboards**: Beautiful real-time visualizations of your service dependencies * **API-Driven Configuration**: Programmatically manage settings across all agents * **Smart Insights**: Analyze traffic patterns, detect anomalies, and optimize API usage * **Alerting**: Get notified when Qtap detects an issue * **Team Collaboration**: Share access and insights across your organization # Getting Started Qplane is Qpoint's cloud-managed control plane that provides centralized configuration, dashboards, and multi-environment visibility for your Qtap agents. Get from zero to insights in 10 minutes. ## What You'll Get * **Centralized Dashboard:** View traffic from all agents across all environments in one place * **Visual Configuration:** Configure stacks, plugins, and filters through a web UI instead of YAML * **Multi-Environment Management:** Organize agents by environment (dev, staging, prod) with unified visibility * **Team Collaboration:** Role-based access control and shared dashboards for your team * **Managed Infrastructure:** No control plane to maintain - we handle updates and scaling ## Three Simple Steps Follow these three steps to get Qplane running with your first agent:

Sign Up

Head to app.qpoint.io and create an account

/pages/kuG53Je7cXGuTz10z803/pages/kuG53Je7cXGuTz10z803

Install the Qtap Agent

Follow the installation Wizard

/pages/IIzD1YGdXz6X0OL3RB4x/pages/IIzD1YGdXz6X0OL3RB4x
Review your DashboardsSee your data immediately without instrumentation/pages/bdIu4o9km8bquoOHkuw3/pages/bdIu4o9km8bquoOHkuw3
## Next Steps After Setup Once you've completed the three steps above: * **Explore the dashboard:** [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Detailed walkthrough of common workflows * **Add more agents:** Install Qtap on additional hosts to see cross-environment traffic * **Configure alerting:** Set up notifications for errors and anomalies * **Understand the architecture:** [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) ## Data Privacy **Your traffic data stays in your infrastructure.** You configure where Qtap stores HTTP/HTTPS request/response payloads (typically your S3 bucket). Qplane only stores: * Agent configuration * Connection metadata (which process called which endpoint) * Dashboard state and queries Sensitive payload data never leaves your infrastructure. See [Choosing Your Deployment](/readme/choosing-your-deployment) for more details. # Create an Account Navigate to and sign up for an account.
## Create an Organization After signing up, you'll be prompted to create your first organization. Enter an organization name that describes your team or designated environment. For example: * Development * Staging * Production {% hint style="info" %} * We recommend creating a test organization first as a sandbox environment * Each account can have multiple organizations * You can create additional organizations later {% endhint %} # Install Qtap Before getting started, we recommend checking the [system requirements](/getting-started/qtap/system-requirements) for details on Qtap compatibility. If you'd like to automatically verify your environment's compatibility, use the following: {% code overflow="wrap" fullWidth="false" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} Once you've signed up for an account, the installation wizard at [app.qpoint.io](https://app.qpoint.io) will walk you through the initial configuration. In this getting started guide, we'll let the installation wizard do most of the heavy lifting. Follow the wizard's on screen instructions:

The installation wizard will walk you through configuration during account creation.

*** See below for more comprehensive guides!
Linux Binary Install/pages/Gj2Q71d398IG96Ca7kN2/files/9O3e4aHRFf1zRfeH47wD
Docker Container Install/pages/8Y3taNByykdY0gaWtZtS/files/D9aOmoASlhdZvkMT1yRn
Helm Chart Install/pages/8FPKmP98B1t2Eiy1VNpQ/files/S8dI4AWL24OiR5OV5RWv
# Review your Dashboards Once you've got the agent up and running, you're ready to review the data that's being collected automatically by the Qtap agent. No additional instrumentation is required. Head back to to drill into your inventory and traffic pages!
Next, head over to our [Configuration](/getting-started/qplane/configuration) configuration guide for an overview on tuning the agent to collect what you want. # System Requirements ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Operating System Requirements * Linux-based operating system * Supports both x86\_64 and arm64 architectures * Kernel version 5.10 or higher ## System Configuration ### Kernel Settings * Kernel lockdown mode should be set to `none` (default on most systems) * Cgroups v2 enabled (default on most modern Linux distributions) ### Network Requirements **Local Network:** * At least one network interface with outbound network access **Cloud Connectivity:** Cloud-connected mode requires outbound connectivity to: * **Control Plane**: `api.qpoint.io` (TCP 443/HTTPS) * **Event Store**: `api-pulse.qpoint.io` (TCP 443/HTTPS) Ensure your firewall allows these connections for Qtap agents. {% hint style="info" %} **No inbound connections required**: Qtap agents only make outbound connections to Qplane. No ports need to be opened for inbound traffic. {% endhint %} ### Access Requirements * Root access for installation * Read-write filesystem access ### Resource Requirements * Sufficient memory for BPF operations * Storage space for logs and configuration # Installation Qtap connects to Qplane using a registration token. Once connected, Qplane pushes all configuration (stacks, plugins, filters) to the agent — no local config file needed. Choose the installation method that fits your environment:
Linux Binary Installhttps://github.com/qpoint-io/documentation/blob/main/getting-started/qplane/installation/linux-binary/README.md
Docker Container Installhttps://github.com/qpoint-io/documentation/blob/main/getting-started/qplane/installation/docker-container/README.md
Helm Chart Installhttps://github.com/qpoint-io/documentation/blob/main/getting-started/qplane/installation/helm-chart/README.md
Kubernetes Manifesthttps://github.com/qpoint-io/documentation/blob/main/getting-started/qplane/installation/kubernetes-manifest/README.md
{% hint style="info" %} For your first installation, we recommend starting with our [Getting Started](/getting-started/qplane/getting-started) guide which walks through the installation wizard. {% endhint %} # Linux Binary This guide covers installing and running Qtap as a Linux binary in cloud-connected mode, managed through the Qplane control plane. ## Preflight Check Verify your environment's compatibility: {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * Linux host with kernel 5.10+ * Root/sudo access * x86\_64 or arm64 architecture * Registration token from [app.qpoint.io](https://app.qpoint.io/) (Settings → Installation) ## Installation ### Quick Install ```bash curl -s https://get.qpoint.io/install | sudo sh ``` ### Manual Installation 1. Download the appropriate binary: For x86\_64: ```bash curl -L https://downloads.qpoint.io/qpoint/qtap-v0.17.1-linux-amd64.tgz -o qtap.tgz ``` For arm64: ```bash curl -L https://downloads.qpoint.io/qpoint/qtap-v0.17.1-linux-arm64.tgz -o qtap.tgz ``` 2. Extract and install: ```bash tar -xzf qtap.tgz && \ sudo mv qtap-* /usr/local/bin/qtap && \ sudo chmod +x /usr/local/bin/qtap ``` ## Running with Registration Token Run Qtap in cloud-connected mode: ```bash sudo qtap --registration-token=$TOKEN ``` Replace `$TOKEN` with your actual registration token from [app.qpoint.io](https://app.qpoint.io/). ## Running as a Service ### Service Configuration 1. Create environment file for secure token storage: ```bash sudo mkdir -p /etc/qtap && \ sudo touch /etc/qtap/environment && \ sudo chmod 600 /etc/qtap/environment ``` 2. Add your registration token and optional tag variables: ```bash sudo tee /etc/qtap/environment << 'EOF' REGISTRATION_TOKEN=your_registration_token_here QTAP_ENVIRONMENT=Production QTAP_SERVICE=API EOF ``` {% hint style="info" %} **Tags are optional** but useful for filtering in Qplane dashboards (e.g., filter by environment, region, or cluster). See [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments#agent-tags) for when to use tags vs installations. {% endhint %} 3. Create systemd service file: ```bash sudo nano /etc/systemd/system/qtap.service ``` Add the following content: ```ini [Unit] Description=Qtap Traffic Analysis Service After=network.target StartLimitIntervalSec=0 [Service] Type=simple User=root EnvironmentFile=/etc/qtap/environment ExecStart=/usr/local/bin/qtap --registration-token=${REGISTRATION_TOKEN} --tags="Environment:${QTAP_ENVIRONMENT},Service:${QTAP_SERVICE}" Restart=always RestartSec=1 [Install] WantedBy=multi-user.target ``` ### Managing the Service 1. Reload systemd: ```bash sudo systemctl daemon-reload ``` 2. Start the service: ```bash sudo systemctl start qtap ``` 3. Enable service to start on boot: ```bash sudo systemctl enable qtap ``` 4. Check service status: ```bash sudo systemctl status qtap ``` 5. View service logs: ```bash sudo journalctl -u qtap -f ``` ## Updating Binary ### Installation Script The quickest way to update Qtap is via our installation script: 1. Stop the service: ```bash sudo systemctl stop qtap ``` 2. Download and install the new version: ```bash curl -s https://get.qpoint.io/install | sudo sh ``` 3. Restart the service: ```bash sudo systemctl start qtap ``` 4. Verify the update: ```bash qtap --version sudo systemctl status qtap ``` ### Manually 1. Stop the service: ```bash sudo systemctl stop qtap ``` 2. Download and install the new version: ```bash # Download the new version curl -L https://downloads.qpoint.io/qpoint/qtap-vX.Y.Z-linux-amd64.tgz -o qtap-new.tgz # Extract and replace tar -xzf qtap-new.tgz sudo mv qtap-* /usr/local/bin/qtap sudo chmod +x /usr/local/bin/qtap ``` 3. Restart the service: ```bash sudo systemctl start qtap ``` 4. Verify the update: ```bash qtap --version sudo systemctl status qtap ``` # Docker Container ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * A Linux host with supported kernel (5.10+) * `Docker`. For installation instructions, refer to the [Docker official documentation](https://docs.docker.com/get-docker/). * A valid registration token from [app.qpoint.io](https://app.qpoint.io) (Settings -> Installation) ## Installation Steps Download the latest Docker Image: ```bash docker pull us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` ### Running Qtap in Docker {% tabs %} {% tab title="Cloud Connected" %} ```bash docker run \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e QTAP_ENVIRONMENT="Production" \ -e QTAP_REGION="US-East" \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --registration-token=$TOKEN \ --tags="Environment:${QTAP_ENVIRONMENT},Region:${QTAP_REGION}" ``` Replace `$TOKEN` with your actual registration token obtained from [app.qpoint.io](https://app.qpoint.io). {% hint style="info" %} **Tags are optional** (`--tags` flag) but useful for filtering in Qplane dashboards. See [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments#agent-tags) for when to use tags vs installations. {% endhint %} {% endtab %} {% endtabs %} ## Docker Run Command Options Explained 1. `--user 0:0`: Runs the container as root (necessary for eBPF operations). 2. `--privileged`: Gives extended privileges to this container. 3. `--cap-add CAP_BPF`: Adds the CAP\_BPF capability (required for eBPF operations). 4. `--cap-add CAP_SYS_ADMIN`: Adds the CAP\_SYS\_ADMIN capability (for low-level system operations). 5. `--pid=host`: Shares the host's PID namespace with the container. 6. `--network=host`: Uses the host's network stack inside the container. 7. `-v /sys:/sys`: Mounts the host's /sys directory into the container. 8. `-v /var/run/docker.sock:/var/run/docker.sock:` Mounts the host's docker socket 9. `-v "$(pwd):/app/config"`: Mounts the current directory to /app/config in the container. 10. `-e TINI_SUBREAPER=1`: Sets up Tini as a subreaper for proper signal handling. 11. `--ulimit=memlock=-1`: Removes the memory lock limit for eBPF programs. ## Qtap-specific Flags * `--log-level=info`: Sets the logging level (debug, info, warn, error). * `--log-encoding=console`: Sets the log encoding format. * `--registration-token=$TOKEN`: Provides the registration token for Qtap. ## Available Flags and Options To see all available options and flags, run: ```bash docker run --rm us-docker.pkg.dev/qpoint-edge/public/qtap:v0 --help ``` # Helm Chart ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * Kubernetes cluster on a Linux Host with supported Kernel (5.10+) * `Helm` * `Kubectl` * A valid registration token from [app.qpoint.io](https://app.qpoint.io) (Settings -> Installation) ## Install Qpoint Helm Repo First, add the Qpoint Helm Repo: ```bash helm repo add qpoint https://helm.qpoint.io/ ``` ```bash helm repo update ``` ```bash helm search repo qpoint/qtap --versions ``` ```bash helm show values qpoint/qtap ``` ## Deploy Qtap {% tabs %} {% tab title="Direct" %} ```bash helm install qtap qpoint/qtap \ -n qpoint \ --create-namespace \ --set logLevel=warn \ --set registrationToken=$TOKEN \ --set extraArgs="{--tags=Environment:Production\,Cluster:EKS-1}" ``` {% hint style="info" %} **Tags are optional** (`--tags` flag) but useful for filtering in Qplane dashboards. When using `--set`, escape commas with `\,`. See [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments#agent-tags) for when to use tags vs installations. {% endhint %} {% endtab %} {% tab title="Kubernetes Secrets" %} 1. Create the `qpoint` namespace if it doesn't exist: ```bash kubectl create namespace qpoint ``` 2. Create a secret containing your registration token: ```bash kubectl create secret generic qtap-token \ --from-literal=token='' \ -n qpoint ``` 3. Deploy Qtap using Helm, referencing the secret: ```bash helm install qtap qpoint/qtap \ -n qpoint \ --set logLevel=warn \ --set registrationTokenSecretRefName="qtap-token" ``` **Note:** You can use your preferred secret management solution to create the secret, such as HashiCorp Vault, AWS Secrets Manager, or other tools integrated with your cluster. {% endtab %} {% endtabs %} ## Verifying the Deployment To verify that Qtap is running: ```bash kubectl get pods -n qpoint ``` You should see a pod named `qtap-xxxx` in the Running state. ## Uninstalling Qtap To uninstall Qtap: ```bash helm uninstall qtap -n qpoint ``` ## Important Notes 1. The Qtap pod requires privileged access for eBPF operations. Ensure your cluster's security policies allow this. 2. To set a specific version, use `--set image.tag="v0.17.1"` 3. For cloud-connected mode, keep your registration token secure and do not share it in public repositories. 4. For local mode, ensure your configuration is correctly formatted and contains all necessary settings. 5. The default configuration mounts the host's `/sys` directory. Ensure this is allowed in your cluster. 6. You may need to adjust resource requests and limits based on your cluster's capacity and Qtap's requirements. 7. When using secret management, ensure proper RBAC permissions are in place for accessing the secret. # Kubernetes Manifest This guide covers deploying Qtap in cloud-connected mode using a Kubernetes manifest, managed through the Qplane control plane. Use this approach when you need full control over your Kubernetes resources or can't use Helm directly. ## Preflight Check If you'd like to verify your environment's compatibility, use the [following script](https://github.com/qpoint-io/preflight/tree/main): {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} ## Prerequisites * Kubernetes cluster on Linux hosts with supported kernel (5.10+) * `kubectl` configured for your cluster * `helm` (for generating the base manifest) * A valid registration token from [app.qpoint.io](https://app.qpoint.io) (Settings → Installation) ## Generating the Base Manifest Use the Helm chart to generate a static Kubernetes manifest: ```bash helm repo add qpoint https://helm.qpoint.io/ helm repo update ``` Generate the manifest with your registration token stored in a Kubernetes secret: ```bash helm template qtap qpoint/qtap \ --set registrationTokenSecretRefName="qtap-token" \ --set logLevel=warn \ > qtap-manifest.yaml ``` To see all available chart values: ```bash helm show values qpoint/qtap ``` ## Creating the Registration Token Secret Store your registration token as a Kubernetes secret: ```bash kubectl create namespace qpoint ``` ```bash kubectl create secret generic qtap-token \ --from-literal=token='' \ -n qpoint ``` {% hint style="warning" %} Keep your registration token secure. Do not commit it to version control or include it directly in manifest files. {% endhint %} ## Deploying Apply the generated manifest: ```bash kubectl apply -f qtap-manifest.yaml -n qpoint ``` ## Verifying the Deployment 1. Check that pods are running: ```bash kubectl get pods -n qpoint ``` You should see `qtap-xxxx` pods in the Running state — one per node (DaemonSet). 2. Check the logs: ```bash kubectl logs -n qpoint daemonset/qtap --tail 30 ``` Look for a successful registration message indicating the agent has connected to Qplane. 3. Confirm the agent appears in the [Qplane dashboard](https://app.qpoint.io) under your environment. ## Example Manifest Here's a complete manifest for cloud-connected deployment with a registration token secret. You can use this directly instead of generating from Helm: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: qtap namespace: qpoint labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap automountServiceAccountToken: true --- apiVersion: apps/v1 kind: DaemonSet metadata: name: qtap namespace: qpoint labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap spec: selector: matchLabels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap template: metadata: labels: app.kubernetes.io/name: qtap app.kubernetes.io/instance: qtap spec: hostPID: true hostNetwork: true serviceAccountName: qtap containers: - name: qtap image: "us-docker.pkg.dev/qpoint-edge/public/qtap:v0" imagePullPolicy: IfNotPresent securityContext: privileged: true allowPrivilegeEscalation: true runAsUser: 0 runAsGroup: 0 runAsNonRoot: false readOnlyRootFilesystem: false capabilities: add: - CAP_BPF - CAP_SYS_ADMIN env: - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: name: qtap-token key: token - name: STATUS_LISTEN value: "0.0.0.0:10001" - name: LOG_LEVEL value: "warn" - name: LOG_ENCODING value: "json" - name: TINI_SUBREAPER value: "1" ports: - name: status containerPort: 10001 protocol: TCP startupProbe: httpGet: path: /readyz port: status initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 20 readinessProbe: httpGet: path: /readyz port: status initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 1 livenessProbe: httpGet: path: /healthz port: status initialDelaySeconds: 3 periodSeconds: 10 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 3 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 1000m memory: 1Gi volumeMounts: - name: sys mountPath: /sys readOnly: true - name: containerd-socket mountPath: /run/containerd/containerd.sock volumes: - name: sys hostPath: path: /sys type: Directory - name: containerd-socket hostPath: path: /run/containerd/containerd.sock type: Socket ``` {% hint style="info" %} No ConfigMap is needed for cloud-connected mode. Qplane pushes configuration (stacks, plugins, filters) to the agent after registration. Any changes you make in the Qplane UI are applied automatically. {% endhint %} ## Adding Tags Tags help you filter and organize agents in the Qplane dashboard. Add them as extra arguments in the container spec: ```yaml args: - "--tags=Environment:Production,Cluster:EKS-1" ``` See [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments#agent-tags) for when to use tags vs separate installations. ## Adding S3 Credentials If your Qplane environment uses an S3 object store for payload storage, inject the credentials alongside the registration token: ```bash kubectl create secret generic qpoint-s3-creds \ --from-literal=AWS_ACCESS_KEY_ID='' \ --from-literal=AWS_SECRET_ACCESS_KEY='' \ -n qpoint ``` Add `envFrom` to the container spec: ```yaml envFrom: - secretRef: name: qpoint-s3-creds ``` ## Uninstalling ```bash kubectl delete -f qtap-manifest.yaml -n qpoint kubectl delete secret qtap-token -n qpoint ``` ## Understanding the Manifest The manifest creates the following resources: | Resource | Purpose | | ------------------ | -------------------------------------- | | **ServiceAccount** | Identity for Qtap pods | | **DaemonSet** | Runs Qtap on every node in the cluster | Key DaemonSet settings: * **`hostPID: true`** and **`hostNetwork: true`** — Required for eBPF to observe host processes and network traffic * **Privileged security context** — Required for loading BPF programs (`CAP_BPF`, `CAP_SYS_ADMIN`) * **`/sys` mount** — Access to kernel interfaces for eBPF * **containerd socket** — Enables container attribution (maps traffic to container names and labels). Optional; remove if not using containerd. * **Probes** — Startup, readiness, and liveness checks on the `/readyz` and `/healthz` endpoints ## Common Customizations * **Node selectors or tolerations** — Target specific nodes or schedule on tainted nodes * **Resource limits** — Adjust CPU and memory based on your traffic volume * **Container runtime socket** — Change from containerd to CRI-O (`/var/run/crio/crio.sock`) if applicable * **Image tag** — Pin to a specific version (e.g., `qtap:v0.17.1`) instead of the rolling `v0` tag ## Troubleshooting 1. **Pods not starting:** ```bash kubectl describe pod -n qpoint -l app.kubernetes.io/name=qtap ``` Check for security policy violations or missing secrets. 2. **Agent not appearing in Qplane:** ```bash kubectl logs -n qpoint daemonset/qtap --tail 50 ``` Verify the registration token is correct and the cluster can reach `api.qpoint.io:443`. 3. **Kernel compatibility:** ```bash kubectl debug node/ -it --image=ubuntu -- uname -r ``` Kernel must be 5.10 or later. # Configuration Configure and manage your Qplane deployment. The cloud control plane provides centralized configuration for all connected Qtap agents across your organization. {% hint style="info" %} **New to Qplane?** Start with the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) for practical examples (10 minutes), or follow the [Getting Started](/getting-started/qplane/getting-started) wizard for step-by-step setup. {% endhint %} *** ## How Qplane Configuration Works In cloud-connected mode, your Qtap agents connect to Qplane at [app.qpoint.io](https://app.qpoint.io) to receive their configuration. Key principles: * **Organization-wide configuration**: Stacks, plugins, routing rules, and filters apply to all installations within an organization * **Real-time updates**: Changes propagate to connected agents within a minute * **No restarts required**: Agents apply new configuration automatically * **Environment identification**: Use installations to identify different environments (prod, staging, dev) *** ## Configuration Topics ### [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments) Structure your Qplane deployment using organizations, installations, and role-based access control. **Learn how to:** * Organize your deployment (organizations vs installations) * Use RBAC roles to control team access * Implement multi-environment strategies * Choose the right access control pattern **Best for:** Admins setting up team access and organizational structure *** ### [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) Configure how Qtap processes captured traffic using stacks and plugins. **Learn how to:** * Create stacks for different traffic types * Configure plugins (Report Usage, Detect Errors, Access Logs, Qscan) * Set up domain-specific routing rules * Use the Reliability dashboard for error detection **Best for:** Engineers configuring traffic capture and processing *** ### [Settings](/getting-started/qplane/configuration/settings) Manage global settings, storage services, filters, and agent configuration. **Learn how to:** * Configure traffic direction (egress-only in Qplane, ingress/all requires [local YAML](/getting-started/qtap/configuration/traffic-capture-settings)) * Set up object stores (S3, MinIO) for sensitive data * Filter out noisy processes * Manage installations and registration tokens **Best for:** Operations teams managing infrastructure and data storage {% hint style="info" %} **Traffic Direction Limitations:** Qplane currently supports egress traffic only (all outbound, external-only, or internal-only). To capture ingress or bidirectional (all) traffic, you must use [local YAML configuration](/getting-started/qtap/configuration/traffic-capture-settings) with standalone Qtap deployment. {% endhint %} *** ### [Alerting](/getting-started/qplane/configuration/alerting) Set up real-time alerts for API performance, reliability, and security events. **Learn how to:** * Create alerting rules (critical latency, root user, vendor availability) * Configure integrations (webhooks, Slack, PagerDuty) * Filter alerts to reduce noise * Review alert events and history **Best for:** SREs, Secops, and on-call teams monitoring production traffic *** ### [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) Understand the complete Qpoint architecture and how all the pieces connect. **Learn about:** * Data flow (events vs objects vs Qscan) * YAML configuration vs Qplane UI * Plugin architecture and processing pipelines * Control plane vs data plane responsibilities **Best for:** Technical leads and architects understanding the full system *** ### [Object Storage Configuration](/getting-started/qplane/configuration/object-storage) Configure S3-compatible storage for captured traffic data. **Learn how to:** * Set up AWS S3, MinIO, or GCS buckets * Configure access credentials * Manage retention policies **Best for:** Operations teams configuring data storage *** ### [MCP Integration](/getting-started/qplane/configuration/mcp) Connect AI assistants to query your traffic data via Model Context Protocol. **Learn how to:** * Create API keys for MCP access * Connect Claude Code, Codex, Cursor, or other MCP clients * Query connections, requests, issues, and PII **Best for:** Teams using AI assistants for traffic analysis and debugging *** ## Quick Start **1. Already have agents deployed?** * Start with [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments) to structure your deployment * Then configure [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) for traffic processing **2. Need to configure data storage?** * Go to [Settings](/getting-started/qplane/configuration/settings) to set up S3-compatible object storage **3. Want to set up monitoring?** * Jump to [Alerting](/getting-started/qplane/configuration/alerting) to configure rules and integrations **4. Understanding the system?** * Read [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) for the complete picture *** ## Common Configuration Tasks **Configure S3 storage for sensitive data:** 1. Navigate to **Settings → Deploy → Services** 2. Add an S3-compatible object store (MinIO, AWS S3, GCS) 3. Configure access credentials 4. See [Settings](/getting-started/qplane/configuration/settings) for details **Create a custom stack for specific traffic:** 1. Navigate to **Plugins → Stacks** 2. Create a new stack with desired plugins 3. Set up routing rules to assign domains to your stack 4. See [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) for details **Set up team access control:** 1. Decide on your organization structure 2. Invite team members with appropriate roles 3. Create views for granular filtering (optional) 4. See [Organizations & Environments](/getting-started/qplane/configuration/organizations-and-environments) for details **Configure error alerting:** 1. Navigate to **Alerting → Rules** 2. Create or enable alerting rules 3. Configure webhook/Slack integrations 4. See [Alerting](/getting-started/qplane/configuration/alerting) for details *** ## Related Documentation * [Installation](/getting-started/qplane/installation) - Deploy additional Qtap agents * [Getting Started](/getting-started/qplane/getting-started) - First-time setup walkthrough * [Guides](/guides) - Step-by-step tutorials # Organizations & Environments Understanding how to structure your Qplane deployment using organizations, installations, and role-based access control (RBAC). *** ## Hierarchy Overview Qplane uses a flexible hierarchy that allows you to organize your infrastructure and control access at multiple levels: ``` Account (User) └── Organization 1 ├── Installation 1 (Production) │ ├── Registration Token │ └── Connected Agents (with optional tags) ├── Installation 2 (Staging) │ ├── Registration Token │ └── Connected Agents (with optional tags) └── Installation 3 (Development) ├── Registration Token └── Connected Agents (with optional tags) └── Organization 2 ├── Installation 1 (Customer A - Production) └── Installation 2 (Customer A - Staging) └── Organization 3 └── Installation 1 (Team B - Shared Environment) └── Connected Agents - Agent with tags: Environment:Prod,Region:US-East - Agent with tags: Environment:Staging,Region:EU-West ``` **Key Concepts:** * **Account**: Your individual user account * **Organization**: Top-level isolation boundary (unlimited per account) - separate users, configs, data * **Installation**: Token boundary within an organization (e.g., prod, staging, dev, or clusters) * **Registration Token**: Unique credential for each installation to connect agents * **Agents**: Individual Qtap instances (typically DaemonSets in Kubernetes) connected to an installation * **Agent Tags**: Optional key:value metadata on agents for UI filtering within Qplane dashboards *** ## Organizations Organizations are the primary way to separate concerns in Qplane. Each organization is completely isolated with its own: * Configuration (stacks, plugins, routing rules) * Data (traffic, inventory, dashboards) * Team members and permissions * Installations and agents ### When to Create Multiple Organizations **Separate customers or tenants:** ``` Organization: Customer A Organization: Customer B Organization: Customer C ``` **Separate business units or teams:** ``` Organization: Engineering Organization: Security Organization: Platform ``` **Separate testing from production:** ``` Organization: Production Organization: Non-Production ``` ### Creating an Organization 1. Click the **organization selector** in the top-left corner 2. Click **+ New Organization** 3. Enter an organization name 4. Click **Create** You can switch between organizations using the organization selector at any time. *** ## Installations (Environments) Installations represent environments within an organization. Each installation has: * A unique **registration token** for connecting agents * Its own set of **connected agents** * Shared configuration from the parent organization ### Common Installation Patterns **Traditional environments:** ``` Organization: My Company ├── Installation: Production ├── Installation: Staging └── Installation: Development ``` **Geographic regions:** ``` Organization: Global Infrastructure ├── Installation: US East ├── Installation: US West ├── Installation: EU Central └── Installation: APAC ``` **Kubernetes clusters:** ``` Organization: Platform Team ├── Installation: EKS Cluster 1 (us-east-1) ├── Installation: EKS Cluster 2 (us-west-2) └── Installation: GKE Cluster 1 (europe-west1) ``` **Per-customer deployments:** ``` Organization: SaaS Product ├── Installation: Customer A - Production ├── Installation: Customer A - Staging ├── Installation: Customer B - Production └── Installation: Customer B - Staging ``` ### Creating an Installation 1. Navigate to **Settings → Installations** 2. Click **+ Installation** 3. Enter an installation name (e.g., "Production", "Staging-US-East") 4. Copy the generated registration token 5. Use the token when deploying Qtap agents {% hint style="info" %} **Configuration is organization-wide**: All installations in an organization share the same stacks, plugins, routing rules, and filters. Changes to configuration affect all installations within that organization. {% endhint %} *** ## Agent Tags Agent tags are optional key:value pairs (e.g., `Environment:Production,Region:US-East`) that you pass when starting Qtap agents. Tags appear in Qplane dashboards and enable UI-level filtering of traffic, inventory, and logs. ### When to Use Tags Tags provide **soft partitioning** - they don't change what gets captured or create access boundaries, but they let you filter within Qplane: * Filter dashboards: "Show only Production traffic" * Compare across dimensions: "US-East vs EU-West error rates" * Create Views: "Payment team sees only Region:US-East traffic" * Flexible organization without creating multiple installations ### Setting Tags Tags are passed via the `--tags` CLI flag when starting Qtap: ```bash # Linux binary qtap --registration-token=$TOKEN --tags="Environment:Production,Region:US-East" # Docker docker run ... \ -e QTAP_ENVIRONMENT="Production" \ -e QTAP_REGION="US-East" \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --registration-token=$TOKEN \ --tags="Environment:${QTAP_ENVIRONMENT},Region:${QTAP_REGION}" # Helm helm install qtap qpoint/qtap \ --set registrationToken=$TOKEN \ --set extraArgs="{--tags=Environment:Production\,Region:US-East}" ``` See installation guides ([Linux Binary](/getting-started/qplane/installation/linux-binary), [Docker](/getting-started/qplane/installation/docker-container), [Helm](/getting-started/qplane/installation/helm-chart)) for detailed examples. ### Using Tags in Qplane Once agents are tagged: 1. **Filter Traffic**: Navigate to Traffic → Logs, click filter icon, add tag filters (Environment=Production) 2. **Create Views**: Settings → Views → Add tag-based filters → Assign to team members 3. **Monitor Agents**: Settings → Deploy → Environments → See all connected agents with their tags *** ## Three Ways to Partition Qplane offers **three complementary levels of partitioning**. You can use them individually or combine them depending on your needs: | Level | Separation Type | Use When | Example | | ----------------- | ------------------------------------ | ------------------------------ | ---------------------------------------------------------------- | | **Organizations** | Hard - separate users, configs, data | Need complete isolation | Separate orgs for customers, or prod vs dev with different teams | | **Installations** | Medium - separate tokens | Need token boundaries | Prod vs staging, or multi-cluster | | **Tags** | Soft - UI filtering only | Just need to filter dashboards | Environment, region, cluster name | **The same deployment can be organized multiple ways:** ### Pattern A: Use Installations for Environments **Best for:** Teams that want hard token boundaries between environments ``` Organization: MyCompany Installation: Production (token A) └── Agents with tags: Region:US-East, Region:EU-West Installation: Staging (token B) └── Agents with tags: Region:US-East ``` **Why:** * ✅ Separate registration tokens (security boundary) * ✅ Can use Views to limit staging access to certain users * ✅ Still use tags for regional filtering within each environment *** ### Pattern B: Use Tags for Environments **Best for:** Teams that want one token across all environments ``` Organization: MyCompany Installation: All-Clusters (one token) └── Agents with tags: - Environment:Production,Region:US-East - Environment:Production,Region:EU-West - Environment:Staging,Region:US-East ``` **Why:** * ✅ Simpler: one registration token * ✅ Easier config management (one installation) * ✅ Use Views to filter prod vs staging in UI * ⚠️ No token boundary between environments *** ### Pattern C: Use Organizations for Environments **Best for:** Teams that need separate user access and different configs ``` Organization: Production └── Installation: Prod-Cluster └── Agents (maybe tagged by region) Organization: Staging └── Installation: Staging-Cluster └── Agents (maybe tagged by region) ``` **Why:** * ✅ Complete isolation (different configs possible) * ✅ Different users per environment * ✅ Strictest separation * ⚠️ More overhead (managing multiple orgs) *** ### Choosing Your Pattern **Ask yourself:** 1. **Do environments need different users?** * Yes → Use separate Organizations (Pattern C) * No → Use Installations or Tags (Pattern A or B) 2. **Do environments need separate registration tokens?** * Yes → Use separate Installations (Pattern A) * No → Use Tags (Pattern B) 3. **Do environments need different configurations?** * Yes → Use separate Organizations (Pattern C) * No → Single organization works **Most common:** Pattern A (Installations for environments, Tags for regions/clusters) *** ## Role-Based Access Control (RBAC) Qplane provides granular roles to control user permissions across different aspects of the platform. ### Roles **Full Access** | Role | Permissions | | ----------------- | ------------------------------------------ | | **Owner** | Complete control with no restrictions | | **Administrator** | Full account access except team management | **Security Roles** | Role | Permissions | | -------------------- | -------------------------------------------------------------- | | **Security Manager** | Can modify security policies and vendor firewall configuration | | **Security Viewer** | Read-only access to security policies and firewall settings | **View Management** | Role | Permissions | | ---------------- | -------------------------------------------------------------------------------- | | **View Manager** | Can create and manage organization views (RBAC filters) | | **Root Viewer** | Can view inventory and traffic without custom views (bypasses view restrictions) | **Traffic & Configuration** | Role | Permissions | | ------------------- | ---------------------------------------------------------------- | | **Traffic Manager** | Can modify rules and configuration (stacks, rules, alerts, etc.) | | **Traffic Viewer** | Read-only access to inventory, traffic, analytics, and errors | **Limited Access** | Role | Permissions | | ------------------ | -------------------------------------------- | | **Payload Viewer** | Can view payloads persisted via rules engine | | **Log Viewer** | Can view traffic logs and errors | {% hint style="info" %} **Key Distinction:** "Manager" roles can modify, "Viewer" roles are read-only. The hierarchy goes from Owner (full control) down to specific read-only roles for different data types. {% endhint %} ### Inviting Team Members 1. Navigate to **Settings → Team** 2. Click **+ Invite Member** 3. Enter their email address 4. Select a role from the list above 5. Click **Send Invitation** The user will receive an email invitation to join your organization. *** ## Access Control Strategies Qplane offers multiple ways to control access to your data: ### Strategy 1: Organization-Level Separation **Use Case:** Completely isolate different teams, customers, or environments **Approach:** Create separate organizations **Example:** ``` Organization: Production (Owners: SRE team) └── Installation: Production US └── Installation: Production EU Organization: Development (Owners: Engineering team) └── Installation: Dev Environment └── Installation: QA Environment ``` **Benefits:** * Complete data isolation * Different teams manage different organizations * Independent configurations **Access Control:** * SRE team has Owner access to Production organization only * Engineering team has Owner access to Development organization only * Security team could have Root Viewer access to both *** ### Strategy 2: Installation-Level Identification **Use Case:** Same team manages multiple environments, but you want to identify which environment generated traffic **Approach:** Use a single organization with multiple installations **Example:** ``` Organization: My Company ├── Installation: Production ├── Installation: Staging └── Installation: Development ``` **Benefits:** * Centralized configuration management * All team members see all environments * Easy to compare traffic across environments **Access Control:** * All team members have access to all installations * Use views (see below) to filter data by environment if needed *** ### Strategy 3: View-Based Filtering **Use Case:** Granular access control within a single organization **Approach:** Use Views to create filtered perspectives on your data **Example:** ``` Organization: Platform Team └── Installation: Shared Kubernetes Cluster Views: - "Payment Service" (filters to payment-api container) - Team: Alice, Bob (Payment team) - "Auth Service" (filters to auth-api container) - Team: Carol, Dave (Auth team) - "Frontend" (filters to web-frontend container) - Team: Eve, Frank (Frontend team) ``` **How Views Work:** 1. Navigate to **Settings → Views** 2. Click **+ New View** 3. Enter a name and description 4. Add filters (by container, pod labels, domain, etc.) 5. Assign team members who can use this view **Access Control:** * **Root Viewer** role can see all data without a view * **Non-Root Viewers** only see data through their assigned views * Each view acts as a lens that filters Traffic, Inventory, and Dashboard pages {% hint style="info" %} **Views are UI-only filters**: They don't change what Qtap captures or stores. They only control what users see in the Qplane dashboards. A user without access to a view will see "No Records" for data outside their view. {% endhint %} *** ## Recommended Patterns ### Pattern 1: Environment Separation (Small Teams) **Best for:** Small teams managing dev, staging, and production ``` Account: Your Account └── Organization: My Company ├── Installation: Production ├── Installation: Staging └── Installation: Development ``` **Access:** * Engineers: Traffic Manager role (can modify stacks, rules, alerts) * Leadership: Root Viewer role (can view all data) * On-call rotation: Traffic Viewer + Log Viewer roles (read-only access to troubleshoot) **Why it works:** Simple structure, centralized configuration, everyone has visibility **Alternative with tags:** Could use single installation with `Environment:Production` and `Environment:Staging` tags if you don't need separate tokens. Filter environments using tag-based Views. *** ### Pattern 2: Multi-Tenant SaaS (Managed Service) **Best for:** SaaS providers managing multiple customer environments ``` Account: Your Account ├── Organization: Customer A │ ├── Installation: Production │ └── Installation: Staging ├── Organization: Customer B │ ├── Installation: Production │ └── Installation: Staging └── Organization: Internal └── Installation: Our Infrastructure ``` **Access:** * Your team: Owner on all organizations * Customer A stakeholders: Traffic Viewer on Customer A organization only * Customer B stakeholders: Traffic Viewer on Customer B organization only **Why it works:** Complete customer isolation, customers can view their own data *** ### Pattern 3: Large Organization with Multiple Teams **Best for:** Enterprises with many teams sharing infrastructure ``` Account: Your Account └── Organization: Company Platform └── Installation: Production Kubernetes Cluster Views: - "Payment Services" → Payment team - "Auth Services" → Security team - "Data Pipeline" → Data team - "API Gateway" → Platform team - "Frontend Apps" → Product team ``` **Access:** * Platform team: Owner role (manages everything) * Individual teams: Traffic Viewer role + assigned views (see only their services) * Leadership: Root Viewer role (see everything without view restrictions) **Why it works:** Centralized platform management, teams see only their services, no configuration conflicts *** ### Pattern 4: Multi-Region Deployment **Best for:** Teams managing multiple geographic regions or cloud providers ``` Account: Your Account └── Organization: Global Platform ├── Installation: AWS US-East-1 ├── Installation: AWS US-West-2 ├── Installation: GCP Europe-West1 └── Installation: Azure East-Asia ``` **Access:** * Global SRE team: Owner role * Regional teams: Traffic Manager role (can modify configs for all regions) * Regional on-call: Traffic Viewer role (read-only access to their region via views) **Why it works:** One configuration applied globally, installation names identify traffic source **Alternative with tags:** If regions share the same environment (e.g., all production), use one installation with tags like `Region:US-East`, `Region:EU-West`, `Cloud:AWS`, `Cloud:GCP`. Filter regions in dashboards using tag filters or Views. *** ## Configuration Sharing {% hint style="warning" %} **Important:** Configuration settings (stacks, plugins, routing, filters) are **organization-wide**. All installations within an organization share the same configuration. {% endhint %} This means: * ✅ Change a stack → affects all installations in the organization * ✅ Add a routing rule → applies to all installations in the organization * ✅ Update filters → filters all installations in the organization **If you need different configurations:** * Create separate organizations (each has independent configuration) * Example: "Production" organization with strict error-only capture, "Development" organization with full debug capture *** ## Decision Matrix Use this matrix to decide how to structure your deployment: | Need | Solution | | --------------------------------------------- | ------------------------------------------------------------------------------------------- | | Completely isolate customer data | Separate organizations per customer | | Different configurations for prod vs dev | Separate organizations (one for prod, one for dev) | | Different users for prod vs dev | Separate organizations | | Separate tokens for prod vs dev | Separate installations (Production, Staging) | | Same config + same token, just filter UI | Single installation + agent tags (Environment:Prod, Environment:Staging) | | Limit team access to specific services | Single organization, use Views (can filter by tags) | | Geographic/region identification | Single installation + agent tags (Region:US-East, Region:EU-West) OR multiple installations | | Multi-cloud visibility | Single installation + agent tags (Cloud:AWS, Cloud:GCP) OR multiple installations | | Filter by cluster in multi-cluster deployment | Agent tags (Cluster:EKS-Prod-1, Cluster:GKE-Prod-2) | | Compare metrics across dimensions | Agent tags + Qplane dashboard filters | *** ## Next Steps **Set up your organization structure:** 1. Decide on your organization strategy using the patterns above 2. Create organizations via the organization selector 3. Create installations for each environment **Configure access control:** 1. Invite team members via **Settings → Team** 2. Assign appropriate roles (Owner, Administrator, Root Viewer) 3. Create Views if you need granular filtering **Deploy agents:** 1. Get registration tokens from **Settings → Installations** 2. Deploy Qtap agents with the appropriate token 3. Verify agents are connected **Related Documentation:** * [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) - Configure traffic processing * [Settings](/getting-started/qplane/configuration/settings) - Manage team members and RBAC * [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) - Understand the overall architecture # Alerting The alerting system provides real-time monitoring and notification capabilities for your API infrastructure, enabling proactive response to performance issues and outages. {% hint style="success" %} **Quick start:** See the [POC Kick Off Guide - Set up error alerting](/guides/qplane-guides/poc-kick-off-guide#set-up-error-alerting) section for a simple working example showing how to create rules and configure integrations. {% endhint %} ### System Overview The alerting system consists of four main components: 1. **Rules** - Define monitoring conditions and thresholds 2. **Filters** - Narrow alert scope to specific conditions 3. **Integrations** - Configure where alerts are sent 4. **Events** - View alert history and patterns ### Rules Configuration Rules are the foundation of your alerting system, continuously monitoring metrics and triggering notifications when thresholds are crossed. #### Pre-Built Rule Templates Qpoint provides pre-built templates organized by category to help you get started quickly. **Discovery Templates** Monitor when new entities appear in your traffic: * **New Vendor** - Detects when a new vendor is used * **New Endpoint** - Detects when a new endpoint (domain or IP) is accessed * **New Client + Vendor** - Detects when a client uses a vendor for the first time * **New Sensitive Data Type** - Detects when a new sensitive data type is used * **New Client + Sensitive Data Type + Vendor** - Detects when a client sends a sensitive data type to a vendor for the first time * **New Token** - Detects when a new token is used * **New Country** - Detects when a new country is connected to **Risk Templates** Identify security and compliance risks: * **Root User Connections** - Detect when connections are established from processes running as root user * **Unencrypted Data Transmission** - Alert when data is transmitted over unencrypted connections * **Direct IP Access** - Detect when connections bypass DNS resolution and use direct IP addresses * **Shell Access Attempts** - Detect when shell access is attempted or established * **Weak TLS Usage** - Alert when deprecated or weak TLS versions are used * **Authentication Failures** - Detect when authentication errors occur * **Token-Based Authentication** - Track when authentication tokens are used * **Unknown TLS Configuration** - Alert when unknown or unsupported TLS configurations are detected * **High-Risk Connection Pattern** - Detect high-risk connections combining root access with unencrypted transmission * **Authentication Token from Root** - Detect when authentication tokens are used from root user processes **Reliability Templates** Monitor service health and availability: * **High Error Rate** - Alerts when error rate exceeds threshold * **Sudden Error Rate Increase** - Detects when error rate increases suddenly * **Vendor Availability Unacceptable** - Monitors when service availability falls below acceptable levels * **Sudden Availability Drop** - Detects when service availability drops suddenly **Performance Templates** Track response times and latency: * **Slow Response Time** - Detects when API response times degrade beyond acceptable thresholds * **Sudden Response Time Increase** - Detects when response time increases suddenly #### Custom Rule Creation When the pre-built templates don't meet your needs, create custom rules with complete control over monitoring conditions. There are four types of custom alerts, each designed for different monitoring scenarios. **Alert Types** **1. Discover New Entities** Detect when new entities appear in your traffic for the first time. **Configuration:** * **Check Interval**: How frequently to check for new entities (1 minute to 24 hours) * **Entity Combinations**: Select which entities to monitor (Vendor, Endpoint, Client, PII Data, Token, Country) * **Report Frequency**: How often to notify (Always, On State Change, Don't Report, Custom Schedule) **Example**: Alert when your application connects to a new vendor you haven't seen before, which could indicate shadow IT or unauthorized integrations. **2. Risk Detection** Identify security and compliance risks in your connections based on predefined risk patterns. **Configuration:** * **Check Interval**: How frequently to evaluate risk (1 minute to 24 hours) * **Define Source By**: What initiated the connection (Bin, Container, Pod, Process, etc.) * **Define Destination By**: Where the connection is going (Vendor, Endpoint, IP, etc.) * **Risk Labels**: Select which risk patterns to monitor from the available labels: * `user-shell` - Shell access attempts or established shell connections * `direct-ip` - Direct IP address connections bypassing DNS * `is-root` - Connections from processes running as root user * `unencrypted` - Unencrypted data transmission * `deprecated-tls` - Usage of deprecated or weak TLS versions * `unknown-tls` - Unknown or unsupported TLS configurations * `auth-error` - Authentication failures during connections * `auth-token` - Token-based authentication usage * **Report Frequency**: How often to notify **Example**: Detect when root user processes establish connections to external vendors, or when unencrypted connections are made to sensitive endpoints. **3. Percentage Changes in a Metric** Monitor for significant increases or decreases in metrics compared to historical baselines. **Configuration:** * **Check Interval**: How frequently to evaluate changes (1 minute to 24 hours) * **Metric**: Select the metric to monitor (same metrics as threshold alerts) * **Increase Percentage**: Alert when metric increases by this percentage (e.g., 50%) * **Decrease Percentage**: Alert when metric decreases by this percentage (e.g., 25%) * **Group By**: Segment alerts by Vendor, Endpoint, or Client (optional) * **Report Frequency**: How often to notify **Example**: Alert when error rates suddenly spike by 50% or when traffic volume drops by 25%, indicating potential issues even if absolute thresholds aren't crossed. **4. Metric Surpasses a Threshold** Alert when a metric crosses a defined absolute threshold value. **Configuration:** * **Check Interval**: How frequently to evaluate the metric (1 minute to 24 hours) * **Metric**: Select from 50+ available metrics (see metric categories below) * **Operator**: How to compare the metric (>, <, =, ≥, ≤) * **Threshold**: The numeric value that triggers the alert * **Group By**: Segment alerts by Vendor, Endpoint, or Client (optional) * **Report Frequency**: How often to notify **Example**: Alert when P99 response time exceeds 500ms or when error count goes above 100 errors. **Check Intervals** How frequently the rule evaluates your metrics: * 1 Minute (most responsive) * 5 Minutes * 15 Minutes * 30 Minutes * 1 Hour * 2, 4, 8, 12, 24 Hours (for less volatile metrics) **Available Metrics** Choose from 50+ available metrics organized by category. **Availability Metrics:** * Availability - Overall service availability percentage * Average Availability - Mean availability across time period * P99/P95/P90 Availability - Percentile-based availability metrics **Performance Metrics:** * Average Duration - Mean response time * P99/P95/P90/P50 Duration - Percentile response times * Maximum Duration - Worst-case response time **Traffic Metrics:** * Total Connections - Active connection count * Connections per Second - Connection rate * Total Requests - Request volume * Requests per Second - Request rate **Error Metrics:** * Errors - Total error count * Errors per Second - Error rate **Data Transfer Metrics:** * Total Bytes In/Out - Combined data transfer * Total Bytes Sent/Received - Directional data transfer * Bytes Sent/Received per Second - Transfer rates **Operators (for Threshold alerts)** Choose how to compare the metric against your threshold: * Greater Than (>) * Less Than (<) * Equal (=) * Greater Than or Equal (≥) * Less Than or Equal (≤) **Grouping (Optional)** Segment alerts by specific dimensions to get granular notifications: * **Vendor** - Monitor each vendor separately * **Endpoint** - Track individual endpoints * **Client** - Monitor per-client metrics **Report Frequency** Control how often you want to be notified: * **Don't Report** - Evaluate only, no notifications (useful for testing) * **On State Change** - Notify when alert state changes (recommended to avoid alert fatigue) * **Always** - Notify every time condition is met * **Custom Schedule** - Define specific intervals for notifications ### Filters (Advanced Configuration) Filters allow you to create highly specific alert conditions by narrowing the scope of monitored data. This significantly reduces false positives by targeting specific scenarios. #### Available Filter Dimensions **Network & Infrastructure:** * Source IP / Destination IP * Source Port / Destination Port * Protocol (HTTP/HTTPS, etc.) * Direction (inbound/outbound) * Host / Hostname **Geographic:** * Country * Continent * Region * City **Application Layer:** * Endpoint * HTTP Method * HTTP Status * Content Type * Error * TLS Version **Container/Kubernetes:** * Container Image * Container Name * Pod Name * Pod Namespace **Business Logic:** * Vendor * Environment * Strategy * Data Type **Security & Access:** * Agent * System User * Bin * Executable * IP #### How Filters Work 1. Add a filter by clicking the "+" button in the Filters section 2. Select the dimension to filter on 3. Choose the matching criteria (equals, contains, etc.) 4. Multiple filters can be combined to create precise conditions {% hint style="info" %} Filter operators (AND/OR logic) are coming soon to enable even more sophisticated filtering. {% endhint %} ### Integrations Integrations define where and how alerts are delivered. The system supports any webhook-based integration, making it compatible with Slack, PagerDuty, Microsoft Teams, custom internal systems, and more. #### Creating an Integration 1. **Name**: Descriptive identifier for your integration (e.g., "Production Alerts Slack") 2. **Description**: Optional details about purpose/destination 3. **Status**: Enable/Disable toggle for quickly turning integrations on/off 4. **Configuration**: * **Method**: HTTP method (typically POST) * **URL**: Webhook endpoint URL * **Auth Token**: Optional authentication header for secured webhooks #### Alert Payload Format When an alert fires, the following JSON payload is sent to your configured webhook: ```json { "orgId": "X0ysTipXUDzr9JPkY4W6", "contextId": "", "alertId": "cv4umbq9io6g00lgdpog", "name": "Low Availability Warning", "description": "Early warning when availability starts to drop", "message": "P99 Availability of 0.00% for cloudflare.com is below threshold of 99.50%", "locationId": "cloudflare.com", "locationType": "vendor", "timestamp": "2025-07-29T19:24:52.726139471Z" } ``` **Payload Fields:** * **orgId**: Your organization identifier * **contextId**: Additional context identifier (if applicable) * **alertId**: Unique identifier for this alert instance * **name**: The rule name that triggered * **description**: The rule description * **message**: Human-readable alert message with current value, threshold, and location * **locationId**: The specific entity that triggered the alert (vendor, endpoint, or client) * **locationType**: Type of entity ("vendor", "endpoint", or "client") * **timestamp**: ISO 8601 timestamp when the alert was triggered ### Events Dashboard The Events page provides comprehensive visibility into your alert history and helps you identify patterns, false positives, and optimization opportunities. #### Overview Features * **Time Range Selector**: View events from the past 15 minutes to 30 days * **Summary Table**: Total occurrences per rule for quick pattern identification * **Activity Timeline**: Visual representation of alert patterns over time #### Per-Rule Analytics For each alerting rule, you can view: * **Occurrence count**: How many times the alert fired * **Time-series graph**: Visual timeline showing when alerts fired * **Pattern identification**: Spot recurring issues or correlation with known incidents * **Peak activity periods**: Identify when alerts are most frequent #### Using Events Data The Events dashboard helps you: * Identify false positive patterns (rules firing too frequently may need threshold adjustment) * Adjust thresholds based on historical data (use actual performance patterns to tune alerts) * Correlate alerts with known incidents (match alert timelines with deployment or infrastructure changes) * Track improvement over time (verify that fixes reduce alert frequency) ### Common Alerting Scenarios #### API Health Monitoring Monitor overall API availability in production: ```yaml Rule: Availability < 99.5% Filter: Environment = Production Group By: Endpoint Check Interval: 1 minute Report Frequency: On State Change ``` #### Vendor SLA Tracking Track response time SLAs for third-party vendors: ```yaml Rule: P99 Duration > 500ms Filter: Vendor exists Group By: Vendor Check Interval: 5 minutes Report Frequency: On State Change ``` #### Error Spike Detection Detect sudden increases in client errors: ```yaml Rule: Errors per Second > 10 Filter: HTTP Status = 5xx Group By: Client Check Interval: 1 minute Report Frequency: Always ``` #### Geographic Performance Monitor performance for specific regions: ```yaml Rule: Average Duration > 200ms Filter: Country = "United States" Group By: Region Check Interval: 15 minutes Report Frequency: On State Change ``` #### Security Monitoring Detect unencrypted data transmission in production: ```yaml Template: Unencrypted Data Transmission Filter: Environment = Production Check Interval: 1 minute Report Frequency: Always ``` #### Discovery Monitoring Get notified when new vendors are discovered: ```yaml Template: New Vendor Check Interval: 5 minutes Report Frequency: Always Entity Combinations: Vendor ``` ### Best Practices 1. **Start with templates**: Use pre-built templates and customize them to your needs rather than building from scratch 2. **Use "On State Change" reporting**: Reduces alert fatigue while ensuring you're notified of important changes 3. **Group by relevant dimensions**: Creates actionable alerts by identifying exactly which vendor, endpoint, or client is affected 4. **Apply filters liberally**: Narrow alert scope to reduce noise and false positives 5. **Use the Events dashboard**: Regularly review alert patterns to tune thresholds and identify improvements 6. **Test with "Don't Report"**: When creating new rules, start with "Don't Report" to validate the rule logic before enabling notifications 7. **Longer intervals for stable metrics**: Use 15+ minute intervals for metrics that don't change rapidly to reduce overhead 8. **Combine Discovery and Risk alerts**: Use Discovery templates to know what's connecting, and Risk templates to monitor how it's connecting ### Important Notes * Alerting operates exclusively in Qplane and is not part of YAML configuration. * The Report Usage plugin must be included in your stacks for alerting to function. * Alerts are generated from anonymized event metadata collected by the Report Usage plugin * All alert data is stored in Pulse (ClickHouse) and does not include payload data from object stores # How It Fits Together This guide ties together the Qpoint platform’s architecture, the Qplane (web console) control plane and the Qtap agent/YAML configuration. It explains what each component does, how they interact, and where specific functionality lives—whether in YAML, in Qplane UI or both. *** #### 1. Qpoint Architecture and Data Flow Qpoint is built on a multi-layer architecture designed for deep visibility while preserving data sovereignty: **Qtap (Data Plane)** – A lightweight eBPF agent you run on your servers. It attaches to kernel socket functions and TLS/SSL library calls to capture connection metadata and payloads with minimal overhead. Qtap supports native TLS integration for common libraries (OpenSSL, GoTLS, NodeTLS, Java) or an egress-controller mode for custom TLS implementations. **Qplane (Control Plane)** – Hosted at app.qpoint.io, Qplane is the centralized management and visualization interface. You use it to configure Qtap agents, manage plugins and stacks, define filters and routing rules, create alerting rules and view dashboards. Qplane only processes anonymized event metadata (connection attributes, status codes, etc.) and never accesses sensitive payloads. Payloads remain in your environment. **Qscan (Data Classification)** – An optional service you run to perform deep inspection and classification of payloads (PII, credentials, secrets). Qtap sends payloads to Qscan for analysis, and Qscan runs entirely within your environment. **Object Storage** – Your S3-compatible bucket (AWS S3, MinIO, GCS) stores captured payload data. Qtap uploads request/response headers and bodies directly to this bucket. Qplane never sees this data; instead, it provides signed URLs so your browser can fetch the payloads directly. The data flow distinguishes between **events** (non-sensitive metadata) and **objects** (sensitive payloads). Events go to Qplane/Pulse (ClickHouse) for dashboards and alerting, while objects go to your object store or Qscan. A typical qtap.yaml shows this separation: event stores of type pulse send anonymized metadata to the Qpoint cloud, while object stores of type s3 send payloads to your bucket; a qscan entry sends payloads to your Qscan instance. *** #### 2. Qtap Configuration via YAML Qtap’s behavior is controlled by a YAML file (qpoint.yaml/qtap.yaml). Key sections include: **services** – Defines where events, objects and Qscan payloads are sent. Examples: ```yaml services: event_stores: - id: qpoint_cloud type: pulse url: https://api-pulse.qpoint.io object_stores: - id: aws_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint-data region: us-west-2 access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY qscan: type: client url: https://qscan.internal.example.com token: type: env value: QSCAN_TOKEN ``` **stacks** – Named collections of plugins that define how to process traffic. Each stack contains one or more plugins (e.g., http\_capture, access\_logs, report\_usage, detect\_errors). You can create multiple stacks for different traffic types. **tap** – Defines what traffic to capture and how to apply stacks. Parameters include: * `direction` – e.g., egress, egress-external, egress-internal, ingress, all. * `ignore_loopback` and `audit_include_dns` – control loopback and DNS capture. * `filters` – ignore processes by group (kubernetes, container, qpoint, gke, eks) or custom executables. * `http.stack` – default stack for HTTP traffic. * `endpoints` – assign specific domains to specific stacks for fine-grained control. **rulekit** – Optional; defines macros and expressions used in plugin rules (e.g., capture POST requests or errors). These rules are used by plugins like http\_capture for conditional capture. Qtap can be deployed using Docker, Kubernetes or directly on Linux. You can either create the YAML manually or configure it through Qplane and download the snapshot YAML. *** #### 2.1 Config File vs. Cloud-Connected Registration Tokens Qpoint offers two deployment models for Qtap, each suited to different scenarios: * **Local deployment with a self-managed configuration file.** You supply a complete qtap.yaml to the agent at startup (via a volume mount or environment variable). This file defines all services, stacks, tap settings and rulekit rules. Benefits: no Qpoint account is required, immediate visibility, self-contained operation and minimal setup—perfect for testing or air-gapped environments. Drawback: any configuration changes require editing the file and redeploying the agent. * **Cloud-connected mode using a registration token.** Instead of providing a config file, you register each Qtap agent with Qplane by supplying a registration token. Generate the token in the Qplane dashboard under Settings → Installations. Start Qtap with this token by passing the --registration-token flag to the agent (for example, `sudo qtap --registration-token=$TOKEN` or `docker run … tap --registration-token=YOUR_TOKEN`). The agent then fetches the organization-wide configuration from the control plane and remains synchronized. This model provides centralized management, advanced analytics and multi-environment visibility. Configuration changes in Qplane automatically propagate to all connected agents within minutes, and you can issue separate tokens per environment (dev, staging, prod) to track installations. Because registration tokens grant access to your organization, treat them as sensitive: store them securely, restrict access, rotate them periodically and revoke them immediately if compromised. *** #### 3. Qplane: Control Plane and UI Qplane extends beyond simple visualization—it's where you centrally manage Qtap agents and configuration without editing YAML directly. Key control features include: **3.1 Stacks & Plugins** Under Plugins → Stacks, you see all defined stacks (Default, Sensitive Data, Debug, Ignore) and can create new ones. Each stack corresponds to a YAML entry under stacks. When you edit a stack in Qplane, you choose plugins and set their parameters; for example, the http\_capture plugin supports capture levels (none, summary, details, full) and can include rulekit rules. Changes made here are reflected in the snapshot YAML. Routing (under Plugins) lets you map domains to stacks visually. This UI maps directly to the endpoints section of the YAML. **3.2 Qtap Settings** In Settings, you control organization-wide and agent-specific settings: * **Qtap → Config** – Choose the audit log destination (event store), traffic direction and advanced capture options (loopback, DNS). These map to services.event\_stores and tap settings. * **Qtap → Filters** – Toggle ignoring Kubernetes, container runtimes, GKE/EKS or Qpoint processes, and define custom filters. Maps to tap.filters settings. * **Deploy → Services** – Configure Object Stores, Event Stores and Qscans. This corresponds to services.object\_stores, services.event\_stores and services.qscan. * **Deploy → Snapshot** – Displays the auto-generated qpoint.yaml reflecting all current settings. You can copy or download this YAML to use in code-based deployments. **3.3 Alerting (Exclusive to Qplane)** Alerting is not configured via YAML; it is managed entirely within Qplane. The Alerting module provides real-time monitoring and notification capabilities for API performance, reliability and infrastructure metrics. It consists of four parts: * **Rules** – Define when an alert should trigger. Pre-built templates (Critical Latency, High Error Rate, Vendor Availability Drop) or custom rules. * **Filters** – Narrow a rule’s scope to vendors, endpoints, clients or environments. * **Integrations** – Define where alerts are delivered (webhooks, Slack, PagerDuty). * **Events Dashboard** – View alert history and timelines. Built on anonymized event data. Because alerting is exclusive to Qplane, these rules do not appear in the snapshot YAML. **3.4 Filtering & Views** Qplane provides two mechanisms for slicing anonymized event data: **Ad-hoc filters** – Available on Inventory and Traffic pages. Click the purple “+” button near the date range to open filtering options by metadata such as Agent, Container, Vendor or HTTP Method. Filters are temporary, resetting when the page reloads. **Views (preconfigured filters & RBAC)** – If you routinely need the same filter combination, Qplane lets you save it as a View and optionally restrict access to specific users. A View is essentially a saved query: it stores one or more filter criteria and can be assigned to team members. You manage Views under Settings → Views, where you’ll see a list of existing views and a New View button. Each view has a name, description, a Filters section and a Team section. The Filters section lists the criteria (e.g., Container Image: ...or-go:1.17), and the Team section lists users allowed to use that view. Only users added here will see the view in their View drop-down; removing a user revokes access. Switching views is done via the View drop-down in the top-left of the dashboard (beside the organization selector). By default, All is selected, showing unfiltered data. Choosing a custom view applies its filters across applicable pages. For example, selecting Hood’s View (a view filtered to a specific container image) causes the Inventory and Traffic tables to display No Records for entities outside that container, effectively sandboxing the user’s visibility. This mechanism supports role-based access control (RBAC): you can create views that limit certain users to a subset of the data without granting them global access. Views live entirely in Qplane; there is no YAML representation for them. Persistent views are particularly useful when different teams need different perspectives—APM teams might have a view filtered to application domains, security teams might view only high-severity issues, and SRE teams might focus on a particular environment. Because views are UI-only constructs, you can adjust filters or reassign users without redeploying agents or modifying the YAML. *** #### 4. Putting It All Together Here’s how you might deploy and manage Qpoint end-to-end: 1. **Plan your data flow.** Decide where anonymized events should go (Pulse/Qplane) and where payloads should be stored (S3/GCS/MinIO) and classified (Qscan). Define these in the services section of your YAML or through Qplane’s Services UI. 2. **Define stacks and plugins.** For basic setups, create a default\_stack with the http\_capture plugin at summary or full level. For more advanced use cases, define additional stacks (Debug, Sensitive Data) and configure plugins like report\_usage, detect\_errors, access\_logs or qscan. Use Qplane’s Stacks UI to create and manage these stacks. 3. **Configure capture settings.** Use the tap section (or Qplane’s Config/Filters pages) to choose traffic direction, ignore loopback or DNS, filter out noisy processes and map specific domains to specific stacks. These settings ensure you capture only what matters. 4. **Deploy the agent.** Install the Qtap agent on your servers or Kubernetes nodes. Provide the generated YAML via volume mount or environment variable. 5. **Use Qplane for control and visibility.** Once agents are running, log into Qplane to: * Monitor live metrics and dashboards. * Adjust stacks, plugins and filters; watch the snapshot YAML update accordingly. * Configure alerting rules, integrations and events dashboards. Remember: alerting lives exclusively in Qplane and is not part of YAML. * Invite team members and set roles; configure SSO. * Download the Snapshot YAML at any time to reconcile UI settings with code-based deployments or version control your configuration. 6. **Iterate and optimize.** Use historical events and alert patterns to refine your rules and thresholds. Tune capture levels and filters to balance visibility with cost and performance. *** #### 5. Key Takeaways * **Separation of Control and Data:** Qplane handles management and visualization; Qtap handles data capture. Events (metadata) go to Qplane/Pulse; objects (payloads) go to your object store and optional Qscan. * **YAML vs UI:** The YAML file controls agent behavior (services, stacks, tap settings and rulekit rules). Qplane can generate and update this YAML through the snapshot, but some features—particularly alerting, dashboards, team management and SSO—exist only in the UI and are not present in the YAML. * **Plugin & Stack Management:** Stacks group plugins to process traffic. Use http\_capture for capturing HTTP transactions with flexible levels; report\_usage and detect\_errors generate events; access\_logs provides aggregated logs; qscan sends payloads to Qscan for classification. * **Alerting:** Qplane provides a rich alerting system with templates and custom rules, filters to reduce noise, webhook integrations for notifications and a comprehensive events dashboard. This system operates on aggregated event data and does not require YAML configuration. * **Centralized Management:** With Qplane you can manage hundreds of agents, share dashboards, collaborate with your team and integrate notifications—all from a single portal. By understanding where each capability resides, you can confidently design your Qpoint deployment—using YAML for repeatable agent configuration and Qplane for centralized control, alerting and visualization. # Stacks & Plugins This guide explains how to configure traffic processing in Qplane using Stacks and Plugins. When you use Qplane (cloud-connected mode), you manage stacks visually through the UI instead of editing YAML files directly. {% hint style="success" %} **New to stacks and plugins?** See practical examples in the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) before diving into the details. The guide shows you how to use the default stack, create custom stacks, and set up routing rules. {% endhint %} *** ### Understanding Stacks & Plugins **Stacks** are named collections of plugins that work together to process your traffic. Think of a stack as a processing pipeline: traffic flows in, plugins process it according to their rules, and results flow out to your configured destinations. **Plugins** are individual processing components that perform specific functions: * **Report Usage**: Send anonymized metrics to Qplane for analytics and dashboards * **Detect Errors**: Capture detailed information when errors occur and upload to your object store * **Scan Payloads & Data**: Classify sensitive data (PII, credentials, secrets) using Qscan * **Access Logs**: Write formatted logs to stdout (console output) * **HTTP Metrics**: Expose Prometheus metrics for HTTP traffic You can create multiple stacks for different traffic types. For example, use one stack for general traffic and another for payment APIs that need extra scrutiny. *** ### The Default Configuration When you first set up Qplane, a default stack called **"Basic - Reporting and Error Detection"** is automatically created. This stack includes these plugins: #### Report Usage Sends anonymized usage metrics to Qplane (Pulse) so you can see traffic patterns, vendor analytics, and endpoint usage in your dashboards. **What it does:** * Tracks connection counts, request volumes, and response times * Groups metrics by vendor, endpoint, and client * Powers the Traffic, Inventory, and Dashboard pages * Required for alerting to work **Configuration:** None required. This plugin runs automatically when added to a stack. #### Detect Errors Captures full request/response details when HTTP errors occur and uploads them to your object store for debugging. **What it does:** * Monitors HTTP status codes * When a matching error occurs, captures headers and bodies * Uploads captured data to your configured object store * Optionally reports the error as an "issue" in Qplane for visibility **Default Rules:** The default stack includes these pre-configured error detection rules: 1. **App Error** (500) * Triggers on: 500 Internal Server Error * Captures: Request headers, body, response headers, body * Reports as issue: Yes 2. **Infrastructure Outage** (502, 503, 520-523, 525-526, 530) * Triggers on: Bad Gateway, Service Unavailable, Cloudflare errors * Captures: Full request/response details * Reports as issue: Yes 3. **Client Error** (400) * Triggers on: 400 Bad Request * Captures: Full request/response details * Reports as issue: Yes 4. **Authentication Error** (401, 403, 407) * Triggers on: Unauthorized, Forbidden, Proxy Authentication Required * Captures: Full request/response details * Reports as issue: Yes 5. **Rate Limited** (429) * Triggers on: 429 Too Many Requests * Captures: Full request/response details * Reports as issue: Yes 6. **Not Found** (404) * Triggers on: 404 Not Found * Captures: Full request/response details * Reports as issue: Yes **Important:** The Detect Errors plugin uploads payloads to your object store. Make sure you have configured an object store in Settings → Deploy → Services (by default, this will send to QPlane at app.qpoint.io). *** ### Navigating the Plugins Interface #### Viewing All Stacks **Location:** Plugins → Stacks Here you'll see all your configured stacks. By default, you'll only see: * **Basic - Reporting and Error Detection**: The default stack with metrics, reports, and error detection capabilities You can create additional stacks for different purposes such as: * **Sensitive Data Stack**: For traffic requiring PII/credential scanning * **Debug Stack**: For detailed troubleshooting with full logging * Or any custom stack for your specific needs Each stack shows: * Name * Description * "Manage" button to configure plugins #### Creating a New Stack 1. Click **"+ New Stack"** button (top right) 2. Enter a name and description 3. Click into the new stack to add plugins 4. Click **"+ Add Plugin"** to see available plugins 5. Select plugins and configure them 6. Enable "Auto generate release on every change" to push updates to agents immediately *** ### Managing Plugins in a Stack When you click "Manage" on a stack, you see the stack's configuration interface: #### Stack Settings Tab **Name**: The stack identifier used in routing rules **Description**: Human-readable description of the stack's purpose **Auto generate release on every change**: * When enabled (recommended), changes propagate to connected agents upon saving * When disabled, changes are saved but not deployed until you manually create a release #### Plugins Tab This shows all plugins currently in the stack. You can: * Reorder plugins by dragging the handle icons * Click the plugin to configure individual plugin settings * Click "Delete Plugin" to remove #### Reliability Tab (Rules) This tab shows specialized rules that can trigger based on traffic patterns when the Detect Errors plugin is enabled. Default rules include: * **Debug**: Enable debug mode on all traffic (disabled by default) * **App Error**: Errors within the application * **Infrastructure Outage**: Networks down or infrastructure offline * **Client Error**: Malformed or bad requests * **Authentication Error**: Invalid authentication or unauthorized requests * **Rate Limited**: Rate limiting threshold exceeded * **Not Found**: Requested resource doesn't exist * **Slow Response**: Application taking too long to respond Each rule can be enabled/disabled with a toggle and configured by clicking "Manage". *** ### Available Plugins in Detail #### 1. Report Usage **Purpose:** Powers Qplane analytics and dashboards **When to use:** * Always include in production stacks * Required for Qplane's Traffic, Inventory, and Dashboard pages * Necessary for alerting to function **Configuration:** None. Just add it to your stack. **Output:** Sends anonymized event metadata to Pulse (ClickHouse) **YAML equivalent:** ```yaml plugins: - type: report_usage ``` *** #### 2. Detect Errors **Purpose:** Capture and upload full request/response details when errors occur **When to use:** * Debugging integration problems * Understanding why API calls fail * Compliance requirements to log errors **Important Note:** While this plugin is marked as deprecated in the local YAML documentation (which recommends using `http_capture` or `access_logs` instead), Qplane still uses it extensively. Qplane uses `detect_errors` for: * Automatic error categorization (App Error, Infrastructure Outage, etc.) * Selective capture based on status codes * Issue reporting in the Qplane UI * Integration with the Reliability dashboard **Configuration Options:** **Trigger Status Codes**: Which HTTP status codes trigger capture * Individual codes: `404`, `500` * Ranges: `4xx`, `5xx` **Categories**: Only capture errors from specific sources * `app`: Application errors only * Leave empty to capture all **Capture Settings**: * Record request headers: Yes/No * Record request body: Yes/No * Record response headers: Yes/No * Record response body: Yes/No **Report as Issue**: Flag the error for visibility in Qplane's issues list **How it Works:** 1. Traffic flows through the stack 2. Detect Errors monitors HTTP status codes 3. When a configured status code occurs (e.g., 500), the plugin captures the specified headers/bodies 4. Data uploads to your configured object store 5. If "Report as Issue" is enabled, the error appears in Qplane for investigation **YAML equivalent:** ```yaml plugins: - type: detect_errors config: rules: - name: "Server Errors" trigger_status_codes: - '5xx' only_categories: - app report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: true ``` *** #### 3. Scan Payloads & Data (Beta) **Purpose:** Classify sensitive data in your traffic **When to use:** * Compliance requirements (GDPR, CCPA, PCI-DSS, HIPAA) * Security audits of third-party data sharing * Understanding what sensitive data your apps send **What it detects (20+ types):** * Personal: Names, emails, phone numbers, locations, dates of birth * Financial: Credit cards, crypto addresses, IBAN codes, bank accounts * Government IDs: SSNs, driver's licenses, passport numbers * Technical: IP addresses, URLs, API keys, credentials * Medical: License numbers **Configuration Options:** **Cache Settings:** * **Cache TTL**: How long to keep URLs in cache (default: 24h) * **Cache Size**: Maximum URLs to track (default: 4096) **Sampling:** * **Sample Baseline**: Always sample first N requests per URL (default: 10) * **Sample Rate**: Percentage to sample after baseline (default: 0.9 = 90%) **Storage:** * **Record matching metadata**: Store detection results (path, score) in object store **Data Types:** Enable/disable detection for each type individually. For each enabled type: * **Store value with metadata**: Save the actual detected value (e.g., the email address) or just store that something was detected **How it Works:** 1. Traffic flows through the stack 2. Qscan plugin sends payloads to your Qscan service 3. Qscan analyzes content 4. Detected sensitive data is classified and optionally stored 5. Results appear in Qplane for analysis **Important:** Requires a Qscan service running in your environment. Configure in Settings → Deploy → Services → Qscan. **YAML equivalent:** ```yaml plugins: - type: qscan services: qscan: type: client url: https://qscan.internal.example.com token: type: env value: QSCAN_TOKEN ``` *** #### 4. Access Logs **Purpose:** Generate formatted HTTP access logs to stdout (console) **When to use:** * Viewing traffic in real-time during development * Integration with log aggregation systems (Fluentd, Logstash) * Lightweight logging without object store uploads **Configuration Options:** **Mode**: Default logging level * `summary`: Method, path, status code * `details`: Includes headers * `full`: Complete headers and bodies **Format**: Output style * `console`: Human-readable format * `json`: Structured JSON logs **Rules**: Override the default mode for specific traffic (uses Rulekit expressions) **How it Works:** 1. Traffic flows through the stack 2. Access Logs plugin formats each request 3. Logs write to stdout (visible in agent logs) 4. NO uploads to object store occur **YAML equivalent:** ```yaml plugins: - type: access_logs config: mode: details format: console rules: - name: "Full logging for errors" expr: res.status >= 400 mode: full ``` *** #### 5. HTTP Metrics **Purpose:** Expose Prometheus-compatible metrics for HTTP traffic **When to use:** * Integration with existing Prometheus/Grafana monitoring * Custom metrics dashboards * Advanced observability pipelines **Configuration:** Typically none. The plugin exposes metrics on the agent's metrics endpoint. **Metrics Exposed:** * Request counts by method, status, endpoint * Response time histograms * Error rates * Bytes sent/received **YAML equivalent:** ```yaml plugins: - type: http_metrics ``` *** *** ### Routing: Assigning Domains to Stacks **Location:** Plugins → Routing Routing lets you send specific domains to specific stacks. This enables fine-grained control over how different traffic is processed. By default, all traffic uses your default stack (Basic - Reporting and Error Detection), but you can create routing rules to send specific domains to custom stacks you've created. #### Default Stack All traffic not matching a specific endpoint rule uses this stack. By default, this is set to "Basic - Reporting and Error Detection", but you can select any stack you've created. #### Endpoints Map domains to stacks: **Example Setup:** | Domain | Stack | Purpose | | ------------------------- | -------------------- | -------------------------- | | `github.com` | Debug Stack | Full logging for debugging | | `postman-echo.com` | Sensitive Data Stack | Scan test payloads | | `echo.free.beeceptor.com` | Sensitive Data Stack | Scan test traffic | | `httpbin.org` | Sensitive Data Stack | Scan public test API | **To Add an Endpoint:** 1. Click "+ Add Endpoint" 2. Enter domain (e.g., `api.stripe.com`) 3. Select stack from dropdown 4. Click Save **Domain Matching:** * Exact: `api.stripe.com` matches only that domain **Use Cases:** **Payment APIs:** ``` Domain: api.stripe.com Stack: Sensitive Data Stack (with Qscan enabled) Reason: Compliance requires PII detection on payment flows ``` **Internal Services:** ``` Domain: api.internal.company.com Stack: Default Stack (basic reporting only) Reason: Internal traffic doesn't need deep inspection ``` **Debugging Specific Integrations:** ``` Domain: problematic-vendor.com Stack: Debug Stack (with Access Logs at full level) Reason: Troubleshooting integration issues ``` *** ### Common Stack Configurations Here are examples of stacks you might create for different use cases. Remember, only "Basic - Reporting and Error Detection" exists by default - you'll need to create these others yourself. #### Basic - Reporting and Error Detection (Default) **Purpose:** Standard visibility with error capture **Plugins:** * Report Usage * Detect Errors (with all default rules) **Use Case:** Most production deployments. This is created automatically. *** #### Minimal Production Stack **Purpose:** Basic visibility without payload capture **Plugins:** * Report Usage **Use Case:** Simple monitoring, minimal data storage. Good for high-volume internal traffic. **How to Create:** 1. Click "+ New Stack" 2. Name it "Minimal Production" or similar 3. Add only the "Report Usage" plugin 4. Save *** #### Compliance Stack **Purpose:** Detect and classify sensitive data **Plugins:** * Report Usage * Scan Payloads & Data (configured for required data types) * Detect Errors (for error investigation) **Use Case:** GDPR, CCPA, PCI-DSS, HIPAA compliance **Routing:** Apply to domains handling sensitive data (payment APIs, user data APIs) **How to Create:** 1. Click "+ New Stack" 2. Name it "Compliance" or "Sensitive Data" 3. Add "Report Usage" plugin 4. Add "Scan Payloads & Data" plugin and configure data types 5. Add "Detect Errors" plugin 6. Save 7. In Plugins → Routing, route sensitive domains to this stack *** #### Debug Stack **Purpose:** Maximum visibility for troubleshooting **Plugins:** * Report Usage * Access Logs (mode: full, format: console) * Detect Errors (capture everything) **Use Case:** Temporary debugging of specific integrations **Routing:** Apply to problematic domains during investigation, then revert **How to Create:** 1. Click "+ New Stack" 2. Name it "Debug" 3. Add "Report Usage" plugin 4. Add "Access Logs" plugin, set mode to "full" 5. Add "Detect Errors" plugin 6. Save 7. Route specific problematic domains to this stack when needed *** #### Ignore Stack **Purpose:** Disable processing for specific traffic **Plugins:** None **Use Case:** * High-volume internal traffic that doesn't need monitoring * Reducing data storage costs * Excluding specific domains from analysis (health checks, monitoring endpoints) **Routing:** Apply to domains you want to ignore **How to Create:** 1. Click "+ New Stack" 2. Name it "Ignore" 3. Don't add any plugins 4. Save 5. Route noisy domains to this stack *** ### Configuration Workflow #### 1. Plan Your Stacks The default "Basic - Reporting and Error Detection" stack works well for most traffic, but consider creating additional stacks for special cases: **Questions to ask:** * Do we need PII detection for compliance? → Create a Compliance/Sensitive Data stack * Which vendors handle our most sensitive data? → Route those domains to the Compliance stack * Are we troubleshooting specific integrations? → Create a Debug stack temporarily * Do we have high-volume health checks or internal endpoints? → Create an Ignore stack * What's our data retention strategy? → Minimal stacks reduce storage costs #### 2. Create Stacks in Qplane Start with the default "Basic - Reporting and Error Detection" stack. Create additional stacks only when you need different processing: 1. Navigate to Plugins → Stacks 2. Click "+ New Stack" for each additional stack you need 3. Name descriptively (e.g., "Payment APIs - PII Scanning", "Debug Mode", "Ignore Health Checks") 4. Add relevant plugins to each stack 5. Configure plugin settings #### 3. Configure Plugins For each plugin: 1. Click "Manage" to open configuration 2. Set appropriate options (capture levels, data types, etc.) 3. Save changes #### 4. Set Up Routing 1. Navigate to Plugins → Routing 2. Verify your default stack (usually "Basic - Reporting and Error Detection") 3. Add endpoint rules only for domains that need different processing than the default 4. Test with a few domains first before expanding **Example:** If you created a "Sensitive Data" stack with Qscan enabled, route your payment API domain to it: * Domain: `api.stripe.com` * Stack: Sensitive Data * Result: Payment API traffic gets PII scanning; everything else uses default stack #### 5. Deploy and Monitor 1. Enable "Auto generate release on every change" for production stacks 2. Watch the Snapshot YAML to see configuration updates 3. Monitor Traffic and Inventory pages to verify agents are processing correctly 4. Adjust rules and thresholds based on observed patterns #### 6. Iterate As you learn more about your traffic: * Refine routing rules * Adjust capture levels to balance visibility with storage costs * Add or remove plugins based on changing needs * Create new stacks for new use cases *** #### Services Configuration Qplane automatically configures these services for cloud-connected agents: ```yaml services: event_stores: - id: qpoint_cloud type: pulse url: https://api-pulse.qpoint.io token: object_stores: - id: qpoint_cloud type: qpoint url: https://warehouse.qpoint.io token: qscan: type: client url: https://api-s.qpoint.io ``` You don't configure these manually; they're managed by Qplane. However, you can add your own object stores and Qscan instances in Settings → Deploy → Services. # Object Storage Configuration Object storage is where Qpoint stores sensitive request and response payloads captured by your Qtap agents. ⚠️ **Before Production Deployment:** By default, new Qplane accounts are configured to use Qpoint Cloud object storage for testing and development. **You must configure your own object storage (S3, GCS, Azure Blob, or MinIO) before deploying Qtap agents in any production or sensitive environment** to ensure sensitive data stays within your infrastructure. This guide explains when data is stored, how to configure your own object storage, and security best practices for production deployments. ## Understanding Qpoint's Data Architecture Qpoint uses a "separation of data" architecture to preserve data sovereignty and give you complete control over sensitive information: **Anonymized Metadata → Pulse (Qpoint Cloud)** * Connection metadata: IPs, ports, timings, status codes * Sent to Qpoint's cloud control plane (Pulse) for dashboards, metrics, and alerts * Contains no sensitive payload data * Required for Qplane dashboards to function **Sensitive Payloads → Object Storage** * Actual request/response headers and bodies * By default, sent to Qpoint Cloud object storage (testing only) * **For production:** Configure your own object storage so payloads are sent directly to YOUR infrastructure (S3, GCS, MinIO, etc.) * Qplane UI fetches payloads on-demand directly to your browser * Configuring your own storage preserves data sovereignty and regulatory compliance ## When Data is Stored in Object Storage ⚠️ **Critical:** Not all traffic is stored to object storage. Payloads are only captured and stored when specific plugins are configured to do so. ### Plugins That Write to Object Storage **1. Detect Errors Plugin** Captures full request/response details when HTTP errors occur. **When it writes:** * HTTP status codes: 404, 500, 502, 503, 504, and other configurable error codes * Automatically enabled in Qplane with pre-configured error detection rules **What it stores:** * Complete request headers and body * Complete response headers and body * Timing information **Learn more:** [Detect Errors Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md#detect-errors) **2. HTTP Capture Plugin** More flexible, rule-based capture for any traffic matching your criteria. **When it writes:** * Only when capture level is set to `details` or `full` * Based on configurable rulekit expressions (domain, path, method, headers, etc.) **Capture levels:** * `none` - No object storage writes * `summary` - No object storage writes * `details` - Writes headers to object storage * `full` - Writes headers AND bodies to object storage **Example use case:** Capture all calls to `api.openai.com` to audit what prompts are being sent. **Learn more:** [HTTP Capture Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/traffic-processing-with-plugins.md#http-capture) **3. Scan Payloads & Data (QScan) Plugin** Classifies sensitive data in traffic (PII, credentials, API keys, etc.). **When it writes:** * When **"Record matching metadata in object store"** is enabled * Stores metadata about detections: path in payload, detection score, data type found * When **"Store value with metadata"** is checked for specific data types * Also stores the actual matched sensitive values (e.g., the email address, phone number, or credential found) **What it detects:** * PII (names, emails, phone numbers, addresses, SSNs, dates of birth) * Credentials (passwords, API keys, tokens, usernames) * Financial data (credit cards, bank accounts, IBAN codes) * Location data (IP addresses, street addresses, cities, zip codes) * Medical identifiers (medical license numbers) * Custom patterns you define **What gets stored in object storage:** * **Metadata (when enabled):** Detection type, location in payload, confidence score * **Values (per data type):** The actual sensitive data matched (if "Store value with metadata" is checked for that type) **Learn more:** [QScan Plugin Documentation](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md#scan-payloads-data-qscan) ### Plugins That Do NOT Write to Object Storage **Report Usage Plugin** * Only sends anonymized metadata to Pulse * No payloads stored anywhere **Access Logs Plugin** * Writes formatted logs to stdout only * No object storage interaction **HTTP Metrics Plugin** * Exposes Prometheus metrics only * No object storage interaction ## Before You Deploy: Configuration Requirements ⚠️ **WARNING: Configure Your Object Storage BEFORE Deploying in Sensitive Environments** **Default Behavior:** * Qpoint Cloud object storage is configured by default * Designed for testing, preview, and proof-of-concept only * **NOT suitable for production or sensitive data** **Why This Matters:** * **Default error detection is active immediately** - The pre-configured "Basic Reporting and Error Detection" stack starts capturing error payloads (400, 401, 403, 404, 429, 500, 502, 503, etc.) as soon as an agent deploys * Once an agent starts capturing traffic, payloads are immediately sent to the configured object storage * If you haven't configured your own storage, **sensitive data in error responses will be sent to Qpoint Cloud** * Reconfiguring later doesn't retroactively move already-captured data * See [What's Captured by Default](#whats-captured-by-default-in-new-accounts) above for the complete list of error codes that trigger captures **Before deploying Qtap in any environment with sensitive data:** 1. ✅ Configure your own object storage (S3, GCS, Azure Blob, MinIO) 2. ✅ Verify the configuration in Snapshot YAML 3. ✅ Test with a non-sensitive agent deployment first 4. ✅ Only then deploy to production/sensitive environments ## What's Captured by Default in New Accounts ⚠️ **Important:** Every new Qplane account automatically includes a "Basic Reporting and Error Detection" stack with pre-configured error detection rules. This means **object storage writes begin immediately** when your agents detect errors. **Default Error Detection Rules:** Every new account comes with 6 pre-configured rules that automatically capture errors: By default, **full request and response data** (headers AND bodies) is captured to object storage for these status codes: | Error Type | Status Codes | What's Captured | | ------------------------- | ------------------------------- | ----------------------- | | **App Error** | 500 | Full request + response | | **Infrastructure Outage** | 502, 503, 520-523, 525-526, 530 | Full request + response | | **Client Error** | 400 | Full request + response | | **Authentication Error** | 401, 403, 407 | Full request + response | | **Rate Limited** | 429 | Full request + response | | **Not Found** | 404 | Full request + response | **Why This Matters:** * ✅ **Benefit:** Immediate error visibility and debugging capability from day one * ⚠️ **Risk:** If you haven't configured your own object storage, these payloads go to Qpoint Cloud (testing-only storage) * 🔒 **Action Required:** Configure your own object storage **BEFORE** deploying agents in any environment with sensitive data **Where Object Storage is NOT Used:** The `report_usage` plugin (also in the default stack) only sends anonymized metadata to Pulse - it does **not** write to object storage. ## Supported Object Storage Providers Qpoint supports any S3-compatible object storage: | Provider | Type | Deployment | Best For | | ------------------------- | ----------- | ------------------- | ------------------------------------- | | **AWS S3** | Cloud | AWS-managed | AWS customers, production at scale | | **Google Cloud Storage** | Cloud | GCP-managed | GCP customers, multi-cloud | | **Azure Blob Storage** | Cloud | Azure-managed | Azure customers | | **MinIO** | Self-hosted | Your infrastructure | Air-gapped environments, full control | | **Any S3-compatible API** | Varies | Varies | Custom requirements | ## Configuration Guide ### Step 1: Navigate to Object Stores 1. In Qplane, go to **Settings → Deploy → Services** 2. Find the **Object Stores** section 3. Click **"+ Add Object Store"** ### Step 2: Configure Your Provider #### AWS S3 Configuration **Prerequisites:** * S3 bucket created (e.g., `my-company-qpoint-payloads`) * IAM user with S3 write permissions and an access key pair (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`) **Configuration parameters:** ```yaml object_stores: - id: production_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint-payloads region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` **Parameter explanations:** * `endpoint` - S3 service endpoint (use `s3.amazonaws.com` for AWS) * `bucket` - Your bucket name (must exist before deploying agents) * `region` - AWS region where bucket is located * `access_url` - Template for retrieving stored objects (Qplane uses this to fetch payloads for display) * `access_key` / `secret_key` - Required for S3 authentication (IAM roles and ambient credential chains are not currently supported) **IAM Policy Example:** ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::my-company-qpoint-payloads/*" } ] } ``` **See also:** [AWS S3 Configuration Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/aws-s3.md) #### Google Cloud Storage Configuration **Configuration parameters:** ```yaml object_stores: - id: production_gcs type: gcs endpoint: storage.googleapis.com bucket: my-company-qpoint-payloads region: us-west1 access_url: https://storage.googleapis.com/{{BUCKET}}/{{DIGEST}} credentials: type: env value: GOOGLE_APPLICATION_CREDENTIALS ``` **See also:** [Google Cloud Storage Guide](/appendix/object-storage/google-cloud-storage) #### MinIO Configuration (Self-Hosted) **Configuration parameters:** ```yaml object_stores: - id: internal_minio type: s3 endpoint: minio.internal.company.com:9000 bucket: qpoint-payloads region: us-east-1 access_url: https://minio.internal.company.com:9000/{{BUCKET}}/{{DIGEST}} access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` **Notes:** * MinIO uses the S3-compatible API (`type: s3`) * Endpoint should include your MinIO host and port * Region can be any value (MinIO doesn't enforce AWS regions) **See also:** [MinIO Self-Hosted Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/minio.md) ### Step 3: Set Environment Variables (if using credentials) If your configuration references environment variables for credentials, you must set these on the hosts running Qtap agents. **Example for systemd:** ```bash # /etc/systemd/system/qtap.service.d/override.conf [Service] Environment="AWS_ACCESS_KEY_ID=AKIA..." Environment="AWS_SECRET_ACCESS_KEY=..." ``` **Example for Kubernetes:** ```yaml apiVersion: v1 kind: Secret metadata: name: qpoint-object-storage type: Opaque data: AWS_ACCESS_KEY_ID: AWS_SECRET_ACCESS_KEY: --- apiVersion: apps/v1 kind: DaemonSet metadata: name: qtap spec: template: spec: containers: - name: qtap envFrom: - secretRef: name: qpoint-object-storage ``` **See also:** [Qtap Storage Configuration](/getting-started/qtap/configuration/storage-configuration) for local YAML configuration examples ## Verifying Your Configuration ### 1. Check Snapshot YAML After adding your object storage: 1. Go to **Settings → Deploy → Snapshot** 2. Look for your object store in the `services.object_stores` section 3. Verify all parameters are correct (endpoint, bucket, region) **Example of what you should see:** ```yaml services: object_stores: - id: production_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint-payloads region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} ``` ### 2. Deploy a Test Agent Before deploying to production: 1. Deploy Qtap in a non-sensitive test environment 2. Configure a plugin to capture traffic (e.g., HTTP Capture with `level: full`) 3. Generate test traffic that matches your capture rules 4. Check your object storage bucket for uploaded objects **Expected behavior:** * Objects appear in your bucket within seconds of captured requests * Object keys are SHA256 digests (e.g., `abc123def456...`) * Objects contain JSON-encoded request/response data ### 3. Monitor Agent Logs Qtap logs object storage upload attempts: ```bash # View Qtap logs (systemd) journalctl -u qtap -f # View Qtap logs (Kubernetes) kubectl logs -n qpoint daemonset/qtap -f ``` **What to look for:** ✅ **Success:** ``` [INFO] Uploaded payload to s3://my-bucket/abc123... ``` ❌ **Failure:** ``` [ERROR] Failed to upload to object storage: Access Denied [ERROR] Failed to upload to object storage: No such bucket ``` **Common issues:** * Incorrect credentials or IAM permissions * Bucket doesn't exist * Network connectivity to object storage endpoint * Incorrect region configuration ## Related Documentation * [Settings Overview](/getting-started/qplane/configuration/settings) - Complete settings reference * [Stacks & Plugins](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/stacks-and-plugins.md) - Learn about plugins that capture data * [Traffic Processing with Plugins](https://github.com/qpoint-io/documentation/blob/main/how-to-guides/traffic-processing-with-plugins.md) - HTTP Capture plugin details * [Qtap Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - Local YAML configuration examples * [AWS S3 Configuration Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/aws-s3.md) - Provider-specific setup * [Google Cloud Storage Guide](/appendix/object-storage/google-cloud-storage) - GCS-specific setup * [MinIO Self-Hosted Guide](https://github.com/qpoint-io/documentation/blob/main/appendix/object-storage/minio.md) - Self-hosted MinIO setup # MCP Integration The Model Context Protocol (MCP) lets AI assistants query your Qplane traffic data directly. Ask natural language questions about connections, requests, errors, and PII exposure—your AI translates them into structured queries against your live data. ## What You Can Do With MCP, AI assistants can: * Investigate incidents by querying real-time traffic patterns * Audit security by checking authentication failures and PII exposure * Debug performance by analyzing latency and error rates * Discover shadow IT by identifying unexpected API connections MCP provides **read-only access** to your traffic data. It cannot modify configurations or affect traffic flow. ## Connection Details | Setting | Value | | ------------------ | -------------------------------- | | **Endpoint** | `https://pulse.qpoint.io/mcp/v1` | | **Transport** | Streamable HTTP | | **Authentication** | Bearer token | ## Get Your Token 1. Log in to [app.qpoint.io](https://app.qpoint.io) 2. Navigate to **Settings → Deployments** 3. Click **+ New Registration Token** 4. Name it (e.g., "MCP Access") 5. Copy the generated **Auth Token** {% hint style="warning" %} Store your token securely. It provides access to your organization's traffic data. {% endhint %} ## Connect Your MCP Client Use your token as a Bearer token with the MCP endpoint. Configuration varies by client: **Claude Code:** ```bash claude mcp add qpoint --transport http https://pulse.qpoint.io/mcp/v1 \ --header "Authorization: Bearer YOUR_TOKEN" ``` **Codex CLI** (`~/.codex/config.toml`): ```toml [mcp_servers.qpoint] url = "https://pulse.qpoint.io/mcp/v1" bearer_token_env_var = "QPOINT_TOKEN" ``` Then set `export QPOINT_TOKEN=your-token` in your shell. **Other Clients:** Configure your MCP client with: * URL: `https://pulse.qpoint.io/mcp/v1` * Header: `Authorization: Bearer YOUR_TOKEN` ## Example Queries Once connected, try: > "What endpoints have I connected to in the last hour?" > "Show me any 500 errors from today" > "Is there PII detected in outbound traffic?" ## Next Steps * [**MCP Tools Reference**](/guides/qplane-guides/mcp-tools-reference) — Complete reference for all available queries # Settings This guide walks through the Settings section of Qplane, explaining how to configure your Qtap agents, manage data services, and control what traffic gets captured. Settings apply globally to all connected agents in your organization. *** ### Overview **Location:** Settings (top navigation) The Settings area is organized into three main sections: **Organizations** * General: Organization details * Team: User management (covered in Teams & RBAC guide) * Views: Filtered dashboards and RBAC (covered in Teams & RBAC guide) * SSO: Single sign-on configuration **Qtap** * Config: Core capture settings * Filters: Process filtering to reduce noise **Deploy** * Environments: Registration token management * Services: Data destinations (event stores, object stores, Qscan) * Releases: Configuration version control (covered in Releases guide) * Snapshot: View auto-generated YAML *** ### Qtap Configuration **Location:** Settings → Qtap → Config This page controls what traffic Qtap captures and where event metadata goes. #### Audit Log Destination **What it is:** Where anonymized event metadata (connection info, status codes, timing) gets sent. **Default:** Pulse (Qpoint Cloud event store) **Options:** * **Pulse**: Qpoint's managed ClickHouse-based event store. Required for Qplane dashboards, traffic analysis, and alerting to work. * **Stdout** * **Disabled** **When to change:** Rarely. Keep Pulse enabled unless you have a specific requirement to send events to your own infrastructure. **YAML mapping:** ```yaml services: event_stores: - id: qpoint_cloud type: pulse url: https://api-pulse.qpoint.io ``` #### Traffic Direction **What it is:** Which network connections Qtap monitors. **Default:** Egress (Only External) **Options:** * **Egress (Only External)**: Only outbound traffic to external/public IPs. Ignores internal/private IP ranges. * **Egress (All)**: All outbound traffic including internal services. * **Egress (Only Internal):** Only outbound traffic to internal IPs. {% hint style="warning" %} **Qplane Limitation:** Ingress and bidirectional (all) traffic capture are not supported through Qplane configuration. To capture ingress or all traffic, you must use [local YAML configuration](/getting-started/qtap/configuration/traffic-capture-settings) with standalone Qtap deployment. {% endhint %} **YAML mapping:** ```yaml tap: direction: egress-external # or egress, or egress-internal # Note: ingress and all are only available with local YAML config ``` #### Advanced Settings **Local Loopback Traffic** **What it is:** Capture traffic on loopback devices (127.0.0.1, localhost) **Default:** Disabled (toggle off) **When to enable:** * Debugging local development * Monitoring services that communicate via localhost * Testing Qtap configuration locally **Note:** Usually redundant. Most production traffic doesn't use loopback. **YAML mapping:** ```yaml tap: ignore_loopback: true # false to capture loopback ``` **Capture DNS** **What it is:** Log and capture DNS queries **Default:** Disabled (toggle off) **When to enable:** * Security investigations (DNS tunneling detection) * Troubleshooting DNS resolution issues * Understanding what domains your services resolve **Warning:** Very noisy. Generates a lot of events. Enable only when specifically needed. **YAML mapping:** ```yaml tap: audit_include_dns: false # true to capture DNS ``` *** ### Qtap Filters **Location:** Settings → Qtap → Filters Filters reduce noise by ignoring traffic from known processes you don't need to monitor. This improves performance and reduces data volume. #### Ignore Process Groups Pre-defined groups of processes you can toggle on/off: **Kubernetes** **What it ignores:** Standard Kubernetes processes (kube-proxy, kubelet, kube-dns, etc.) **When to enable:** Always, unless specifically debugging Kubernetes control plane issues. **Why:** Kubernetes processes generate constant internal traffic that's rarely relevant for application monitoring. **Container Runtimes** **What it ignores:** Container runtime processes (containerd, dockerd, runc, etc.) **When to enable:** Always, unless debugging container runtime issues. **Why:** Container runtime traffic is infrastructure-level and not application-relevant. **Google Kubernetes Engine (GKE)** **What it ignores:** Built-in GKE processes and system services. **When to enable:** Always when running on GKE. **Why:** GKE-specific processes create noise without providing application insights. **Amazon Elastic Kubernetes Service (EKS)** **What it ignores:** Built-in EKS processes and system services. **When to enable:** Always when running on EKS. **Why:** EKS-specific processes create noise without providing application insights. **Qpoint** **What it ignores:** Qpoint self-auditing (Qtap agent's own traffic) **When to enable:** Always. **Why:** Prevents Qtap from monitoring itself, which would create recursive capture loops. **Best Practice:** Enable all relevant filters for your environment. Start with all enabled, then disable selectively only if you need to investigate specific infrastructure issues. **YAML mapping:** ```yaml tap: filters: ignore_processes: groups: - kubernetes - container - qpoint - gke - eks ``` #### Custom Filters **What it is:** Manually specify executable paths to ignore. **Use cases:** * Internal monitoring tools that generate noise * Backup agents * Log shippers * CI/CD runners * Any process that creates traffic you don't need to monitor **How to add:** 1. Click "+ Add Filter" 2. Enter the full executable path (e.g., `/usr/bin/prometheus-node-exporter`) 3. Save **Finding executable paths:** Look at the Process column in your Traffic or Inventory pages to identify noisy processes. **YAML mapping:** ```yaml tap: filters: ignore_processes: executables: - /usr/bin/sshd - /usr/sbin/nginx ``` *** ### Environments **Location:** Settings → Deploy → Environments Environments manage registration tokens for connecting Qtap agents to Qplane. Each environment has its own token for tracking which agents belong to which deployment (dev, staging, prod). #### Default Environment Created automatically with your organization. Contains a pre-generated registration token. #### Viewing a Token Click "Manage" on an environment to see: **Name:** The environment identifier **Auth Token:** The registration token used to connect agents **Actions:** * **Copy button**: Copy token to clipboard #### Creating Additional Environments **Why create multiple environments:** * Separate dev, staging, and production agents * Track different clusters or regions * Isolate teams or projects * Facilitate troubleshooting by identifying which agents belong where **How to create:** 1. Click "+ Environment" 2. Enter a name (e.g., "Production", "Staging-US-West", "Team-Payments") 3. A new token is generated automatically 4. Copy the token 5. Deploy agents with this token using `--registration-token=` **Viewing agents by environment:** In Settings → Installations, you can see which environment each agent belongs to. **Best Practices:** * Use descriptive names: "Production-US-East-1" not "prod" * Create environments before deploying agents, not after * Store tokens securely (Kubernetes Secrets, AWS Secrets Manager, etc.) * Never commit tokens to version control * Document which token goes with which deployment **YAML impact:** Environments don't appear in YAML. The token is passed at agent startup: ```bash sudo qtap --registration-token= ``` *** ### Data Services **Location:** Settings → Deploy → Services Services define where Qtap sends different types of data: anonymized events, sensitive payloads, and data for classification. #### Object Stores **What they store:** Request/response headers and bodies (sensitive payloads) 📖 **See the** [**Object Storage Configuration Guide**](/getting-started/qplane/configuration/object-storage) **for complete setup instructions, understanding when data is stored, and security best practices.** **Default:** Qpoint Cloud (managed object store for preview and testing) **Important Security Note:** > Payloads may contain sensitive data. Configure an object store to ensure your data never leaves your network and is accessible only from those with proper access. **Qpoint Cloud (Default)** **Purpose:** Preview and testing only **When to use:** * Initial setup and testing * Proof of concept * Development environments **When NOT to use:** * Production environments * Any traffic containing real customer data * Compliance-sensitive deployments (GDPR, HIPAA, PCI-DSS) **Why:** Qpoint Cloud is a managed service for convenience, but your production data should stay in your infrastructure. **YAML mapping:** ```yaml services: object_stores: - id: qpoint_cloud type: qpoint url: https://warehouse.qpoint.io ``` **Adding Your Own Object Store** **Recommended for production.** Configure your own S3-compatible storage: **Supported:** * AWS S3 * Google Cloud Storage * MinIO * Any S3-compatible API **How to add:** 1. Click "+ Add Object Store" 2. Choose type (S3, GCS, etc.) 3. Configure: * Endpoint (e.g., `s3.amazonaws.com`) * Bucket name * Region * Access credentials (use environment variables recommended) * Access URL template **Example configuration:** ```yaml object_stores: - id: production_s3 type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint-data region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` **Security Best Practice:** Rotate access keys regularly and store them in a secrets manager. IAM roles and workload identity are not currently supported for S3 authentication. #### Event Stores **What they store:** Anonymized connection metadata (IPs, ports, status codes, timing) **Qpoint Cloud (Default)** **Purpose:** Powers Qpoint's dashboards, traffic analysis, and alerting **Description:** "Qpoint's managed cloud event store, powering events and analytics" **When to use:** Always, unless you have specific compliance requirements requiring all data to stay internal. **Why:** Qplane's Traffic, Inventory, Dashboard, and Alerting features depend on event data in Pulse. **Data sent:** * Source/destination IPs and ports * HTTP methods and status codes * Response times and sizes * Process and container names * Anonymized metadata only (no payloads) **YAML mapping:** ```yaml services: event_stores: - id: qpoint_cloud type: pulse url: https://api-pulse.qpoint.io ``` **Adding Custom Event Stores** **Use cases:** * Send events to your existing observability platform * Compliance requirements (data residency) * Custom analytics pipelines **How to add:** 1. Click "+ Add Event Store" 2. Configure endpoint and credentials **Note:** Even with custom event stores, keep Pulse enabled if you want to use Qplane's dashboards and alerting. #### Qscans **What they do:** Scan payloads for sensitive data classification (PII, credentials, secrets) **Default:** Qpoint Cloud (managed Qscan for preview and testing) **Important Note:** > Scan payloads and data for sensitive data like PII, PCI, PHI, etc. Run internally or leverage Qpoint's managed service. **Qpoint Cloud (Default)** **Purpose:** Preview and testing only **Description:** "Qpoint's managed cloud qscan service, for preview and testing only" **When to use:** * Testing the Scan Payloads & Data plugin * Proof of concept * Understanding what data types Qscan detects **When NOT to use:** * Production environments * Real customer data * Compliance-sensitive use cases **YAML mapping:** ```yaml services: qscan: type: client url: https://api-s.qpoint.io ``` **Adding Your Own Qscan** **Recommended for production.** Run Qscan in your environment: **Deployment:** * Docker container in your VPC * Kubernetes pod in your cluster * VM in your network **How to add:** 1. Deploy Qscan container in your environment 2. Click "+ Add Qscan" 3. Configure: * URL (e.g., `https://qscan.internal.company.com`) * Authentication token **Example configuration:** ```yaml services: qscan: type: client url: https://qscan.internal.company.com token: type: env value: QSCAN_TOKEN ``` **Using Qscan:** After configuring, add the "Scan Payloads & Data" plugin to stacks where you need sensitive data classification. *** ### Snapshot **Location:** Settings → Deploy → Snapshot The Snapshot page shows the complete auto-generated YAML configuration that Qplane sends to your connected agents. #### What You See **Title:** Auto-Generated Config **Content:** Full `qpoint.yaml` with: * Version * Services (event stores, object stores, qscan) * Stacks (with all plugins and configurations) * Tap settings (direction, filters, routing) **Copy button:** Click to copy the entire YAML to clipboard #### Why It's Useful **Verify UI Changes:** After making changes in Qplane (creating stacks, modifying plugins, adjusting filters), check the Snapshot to see the resulting YAML before it deploys to agents. **Version Control:** Copy the Snapshot YAML and commit to Git to track configuration changes over time. **Bootstrap New Environments:** Download Snapshot YAML to bootstrap a local deployment or test environment. **Debugging:** Compare the Snapshot with what an agent is actually running to identify configuration drift or propagation issues. **Learning:** Understand how UI actions translate to YAML structure. Great for learning the relationship between Qplane and Qtap configuration. **Migration:** If you need to migrate from Qplane to local YAML deployment, use the Snapshot as your starting point. #### Example Snapshot ```yaml version: 2 services: event_stores: - id: qpoint_cloud type: pulse url: https://api-pulse.qpoint.io token: type: text value: eyJhbGci... object_stores: - id: qpoint_cloud type: qpoint url: https://warehouse.qpoint.io token: type: text value: eyJhbGci... qscan: type: client url: https://api-s.qpoint.io stacks: basic_reporting_and_error_detection: plugins: - type: report_usage - type: detect_errors config: rules: - name: App Error trigger_status_codes: ['500'] only_categories: [app] report_as_issue: true record_req_headers: true record_req_body: true record_res_headers: true record_res_body: true # ... additional error rules tap: direction: egress-external ignore_loopback: true audit_include_dns: false filters: ignore_processes: groups: - kubernetes - container - qpoint http: stack: basic_reporting_and_error_detection ``` #### What's NOT in the Snapshot These features exist only in Qplane UI and don't appear in YAML: * Alerting rules and integrations * Team members and permissions * Views (filtered dashboards) * SSO configuration * Environment tokens # Qscan QScan is Qpoint's PII detection service for HTTP traffic captured by Qtap. It uses machine learning models to automatically identify sensitive data -- such as names, email addresses, credit card numbers, and social security numbers -- flowing through your APIs. ### How It Works QScan operates as an asynchronous scanning pipeline: 1. **Qtap** captures HTTP request and response payloads and stores them as artifacts in S3-compatible object storage 2. **Pulse** schedules scan jobs for new artifacts 3. **QScan** polls Pulse for pending jobs, pulls artifacts from S3, scans them using ML models, and reports results back to Pulse 4. **Pulse** surfaces PII findings in your dashboards {% hint style="info" %} QScan must have network access to both your S3 storage endpoint and the Pulse API. It does not need to run alongside Qtap -- it can be deployed anywhere with the required connectivity. {% endhint %} ### Detection Models QScan combines multiple detection engines for comprehensive coverage: * **Piiranha** -- transformer-based NER model optimized for PII detection * **Presidio** -- Microsoft's rule-based and ML-powered PII analyzer * **Flair NER** -- sequence labeling model for named entity recognition Results from all models are merged and deduplicated to minimize false negatives. ### Key Features * **Multi-model detection**: Three complementary engines for high recall across PII types * **GPU acceleration**: Optional NVIDIA GPU support for faster inference * **Horizontal scaling**: Deploy multiple instances with configurable poller and scanner counts * **Prometheus metrics**: Built-in metrics endpoint for monitoring scan throughput and health * **S3-compatible storage**: Works with AWS S3, Google Cloud Storage, MinIO, and any S3-compatible endpoint * **Automatic credential handling**: When using Qpoint's managed storage, the registration token is used for S3 authentication automatically # Getting Started This guide walks you through deploying your first QScan instance and verifying it is operational. ## Prerequisites Before deploying QScan, ensure you have: * A **registration token** from [app.qpoint.io](https://app.qpoint.io) (Settings -> API Tokens) * **S3-compatible object storage** configured and accessible (AWS S3, GCS, MinIO, or Qpoint's managed storage) * **Network access** to the Pulse API (`api-pulse.qpoint.io` on TCP 443) and your S3 endpoint * A **container runtime** (Docker, Kubernetes, or Cloud Run) ## Quick Start with Docker Pull and run the QScan container with your registration token: {% code overflow="wrap" %} ```bash docker run -d \ --name qscan \ -e REGISTRATION_TOKEN=your-registration-token \ -e METRICS_PORT=8080 \ -e LOG_LEVEL=info \ -p 8080:8080 \ --memory=12g \ --cpus=2 \ us-docker.pkg.dev/qpoint-edge/public/qscan:latest ``` {% endcode %} {% hint style="info" %} When using Qpoint's managed S3 storage (the default), you do not need to configure S3 credentials separately. The registration token is used as the S3 access key automatically. {% endhint %} If you are using your own S3-compatible storage, add the S3 configuration: {% code overflow="wrap" %} ```bash docker run -d \ --name qscan \ -e REGISTRATION_TOKEN=your-registration-token \ -e S3_ENDPOINT_URL=https://s3.amazonaws.com \ -e S3_BUCKET_NAME=your-bucket \ -e S3_REGION_NAME=us-east-1 \ -e AWS_ACCESS_KEY_ID=your-access-key \ -e AWS_SECRET_ACCESS_KEY=your-secret-key \ -e METRICS_PORT=8080 \ -e LOG_LEVEL=info \ -p 8080:8080 \ --memory=12g \ --cpus=2 \ us-docker.pkg.dev/qpoint-edge/public/qscan:latest ``` {% endcode %} ## Verify Operation **Check the metrics endpoint:** ```bash curl http://localhost:8080/ ``` A successful response indicates the QScan process is running and the metrics server is healthy. **Check the logs:** ```bash docker logs qscan ``` You should see log messages indicating that QScan has loaded its ML models and begun polling Pulse for scan jobs. Initial model loading takes 1-2 minutes depending on your hardware. ## Next Steps * [System Requirements](/getting-started/qscan/system-requirements) -- hardware and network specifications * [Installation](/getting-started/qscan/installation) -- detailed deployment guides for Docker, Kubernetes, and Cloud Run * [Configuration](/getting-started/qscan/configuration) -- environment variables and tuning options # System Requirements QScan is available in two variants: CPU-only and GPU-accelerated. Choose based on your throughput requirements and infrastructure. ## CPU Variant | Resource | Minimum | Recommended | | -------- | ------- | ----------- | | vCPUs | 2 | 6 | | Memory | 12 GB | 24 GB | | Disk | 10 GB | 20 GB | The ML models consume approximately 8.4 GB of memory at runtime. The recommended configuration supports running 2 pollers and 2 scanners concurrently. ## GPU Variant | Resource | Minimum | Recommended | | -------- | --------------------------- | ---------------------- | | vCPUs | 4 | 4 | | Memory | 16 GB | 16 GB | | GPU | 1x NVIDIA (CUDA-compatible) | 1x NVIDIA L4 or better | | Disk | 10 GB | 20 GB | GPU acceleration significantly improves inference throughput. Any CUDA-compatible NVIDIA GPU with sufficient VRAM is supported. ## Network Requirements **Outbound connectivity (required):** * **Pulse API**: `api-pulse.qpoint.io` (TCP 443/HTTPS) * **S3 storage**: Your configured S3 endpoint (e.g., `s3.amazonaws.com`, a MinIO instance, or `s3.warehouse.qpoint.io` for Qpoint-managed storage) {% hint style="info" %} **No inbound connections required.** QScan only makes outbound connections to poll for jobs and retrieve artifacts. No ports need to be opened for inbound traffic. {% endhint %} ## Container Runtime QScan is distributed as a container image and requires one of the following: * Docker Engine 20.10+ * Kubernetes 1.24+ * Google Cloud Run * Any OCI-compatible container runtime ## Image Registry The QScan container image is hosted at: ``` us-docker.pkg.dev/qpoint-edge/public/qscan ``` # Installation QScan can be deployed on any infrastructure that supports container workloads. Choose the installation method that matches your environment: * [Docker Container](/getting-started/qscan/installation/docker-container) -- standalone deployment using Docker or Docker Compose * [Kubernetes](/getting-started/qscan/installation/kubernetes) -- deployment manifest for Kubernetes clusters * [Cloud Run](/getting-started/qscan/installation/cloud-run) -- Google Cloud Run worker pool configuration All methods use the same container image and environment variables. The primary differences are in resource management and secret handling. # Docker Container ## Docker Run Deploy QScan with a single `docker run` command. At minimum, you need to provide a registration token: {% code overflow="wrap" %} ```bash docker run -d \ --name qscan \ --restart always \ -e REGISTRATION_TOKEN=your-registration-token \ -e METRICS_PORT=8080 \ -e LOG_LEVEL=info \ -e NUM_POLLERS=1 \ -e NUM_SCANNERS=1 \ -p 8080:8080 \ --memory=12g \ --cpus=2 \ us-docker.pkg.dev/qpoint-edge/public/qscan:latest ``` {% endcode %} For self-managed S3 storage, add the S3 environment variables: {% code overflow="wrap" %} ```bash docker run -d \ --name qscan \ --restart always \ -e REGISTRATION_TOKEN=your-registration-token \ -e S3_ENDPOINT_URL=http://your-s3-endpoint:3900 \ -e S3_BUCKET_NAME=qpoint \ -e S3_REGION_NAME=us-east-1 \ -e AWS_ACCESS_KEY_ID=your-access-key \ -e AWS_SECRET_ACCESS_KEY=your-secret-key \ -e METRICS_PORT=8080 \ -e LOG_LEVEL=info \ -e NUM_POLLERS=1 \ -e NUM_SCANNERS=1 \ -p 8080:8080 \ --memory=12g \ --cpus=2 \ us-docker.pkg.dev/qpoint-edge/public/qscan:latest ``` {% endcode %} ## Docker Compose The following example deploys QScan alongside a local S3-compatible store (such as Garage or MinIO): ```yaml services: qscan: image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest container_name: qscan restart: always deploy: resources: limits: cpus: '2.0' memory: 12G reservations: cpus: '0.5' memory: 256M environment: - REGISTRATION_TOKEN=${QPOINT_REGISTRATION_TOKEN} - LOG_LEVEL=info - NUM_POLLERS=1 - NUM_SCANNERS=1 - METRICS_PORT=8080 - S3_ENDPOINT_URL=http://your-s3-host:3900 - S3_BUCKET_NAME=qpoint - S3_REGION_NAME=us-east-1 - AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY} - AWS_SECRET_ACCESS_KEY=${S3_SECRET_KEY} ports: - "8080:8080" ``` ## Resource Limits QScan loads approximately 8.4 GB of ML models into memory at startup. Set container memory limits accordingly: | Configuration | Memory Limit | CPU Limit | | ----------------------------------- | ------------ | --------- | | Minimum (1 poller, 1 scanner) | 12 GB | 2 vCPUs | | Recommended (2 pollers, 2 scanners) | 24 GB | 6 vCPUs | {% hint style="warning" %} Setting memory limits below 12 GB may cause the container to be killed by the OOM killer during model loading or scanning. {% endhint %} # Kubernetes ## CPU Deployment Create a Kubernetes Secret for your registration token and any S3 credentials: ```bash kubectl create secret generic qscan-secrets \ --from-literal=REGISTRATION_TOKEN=your-registration-token \ --from-literal=AWS_ACCESS_KEY_ID=your-access-key \ --from-literal=AWS_SECRET_ACCESS_KEY=your-secret-key ``` Deploy QScan using the following manifest: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: qscan labels: app: qscan spec: replicas: 1 selector: matchLabels: app: qscan template: metadata: labels: app: qscan spec: containers: - name: qscan image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest env: - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: name: qscan-secrets key: REGISTRATION_TOKEN - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: qscan-secrets key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: qscan-secrets key: AWS_SECRET_ACCESS_KEY - name: S3_ENDPOINT_URL value: "http://your-s3-endpoint:3900" - name: S3_BUCKET_NAME value: "qpoint" - name: S3_REGION_NAME value: "us-east-1" - name: NUM_POLLERS value: "2" - name: NUM_SCANNERS value: "2" - name: LOG_LEVEL value: "info" - name: METRICS_PORT value: "8080" ports: - containerPort: 8080 name: metrics resources: requests: cpu: "2" memory: "12Gi" limits: cpu: "6" memory: "24Gi" livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 10 startupProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 30 failureThreshold: 5 ``` ## GPU Deployment For GPU-accelerated scanning, schedule the pod on a node with an NVIDIA GPU and adjust the resource limits: ```yaml resources: requests: cpu: "4" memory: "16Gi" limits: cpu: "4" memory: "16Gi" nvidia.com/gpu: "1" ``` {% hint style="info" %} GPU nodes must have the NVIDIA device plugin installed. On managed Kubernetes services, this typically means using a GPU-enabled node pool (e.g., GKE GPU node pools, EKS with `p3` or `g5` instances, AKS with NC-series VMs). {% endhint %} ## Scaling To scale horizontally, increase the number of replicas. Each replica operates independently, polling Pulse for scan jobs: ```bash kubectl scale deployment qscan --replicas=3 ``` You can also adjust `NUM_POLLERS` and `NUM_SCANNERS` per replica to control concurrency within each instance. See the [Configuration](/getting-started/qscan/configuration) guide for details. # Cloud Run QScan can be deployed as a Google Cloud Run Worker Pool. Worker pools are long-running instances suited for background processing workloads like QScan. ## Prerequisites * Google Cloud project with Cloud Run API enabled * `gcloud` CLI with beta components installed * Secrets stored in Google Secret Manager ## CPU Configuration Create a `worker.yaml` file: ```yaml apiVersion: run.googleapis.com/v1 kind: WorkerPool metadata: annotations: run.googleapis.com/launch-stage: BETA run.googleapis.com/scalingMode: manual run.googleapis.com/manualInstanceCount: "1" name: qscan-worker spec: template: metadata: annotations: run.googleapis.com/execution-environment: gen2 run.googleapis.com/cpu-throttling: "false" spec: containers: - name: qscan-worker image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest env: - name: NUM_POLLERS value: "2" - name: NUM_SCANNERS value: "2" - name: LOG_LEVEL value: "info" - name: LOG_ENCODING value: "json" - name: METRICS_PORT value: "8080" - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: key: latest name: qscan-registration-token livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 10 resources: limits: cpu: 6000m memory: 24Gi startupProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 30 failureThreshold: 5 ``` Deploy with: {% code overflow="wrap" %} ```bash gcloud beta run worker-pools replace worker.yaml --region=us-central1 ``` {% endcode %} ## GPU Configuration For GPU-accelerated scanning, modify the resource limits: ```yaml resources: limits: cpu: 4000m memory: 16Gi nvidia.com/gpu: "1" ``` {% hint style="info" %} GPU support in Cloud Run requires a region that offers GPU instances. Check the [Cloud Run GPU documentation](https://cloud.google.com/run/docs/configuring/services/gpu) for available regions and GPU types. {% endhint %} ## Secret Manager Store sensitive values in Google Secret Manager and reference them in your worker configuration: ```bash echo -n "your-registration-token" | \ gcloud secrets create qscan-registration-token --data-file=- echo -n "your-s3-access-key" | \ gcloud secrets create qscan-s3-access-key --data-file=- ``` Reference secrets in the worker YAML using `secretKeyRef`: ```yaml - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: key: latest name: qscan-registration-token ``` Ensure the Cloud Run service account has the `roles/secretmanager.secretAccessor` role. ## Updating the Image To update the QScan image without modifying the full configuration: {% code overflow="wrap" %} ```bash gcloud beta run worker-pools update qscan-worker \ --image=us-docker.pkg.dev/qpoint-edge/public/qscan:latest \ --region=us-central1 ``` {% endcode %} # Configuration QScan is configured entirely through environment variables. There is no configuration file to manage. ## Environment Variables ### Required | Variable | Description | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `REGISTRATION_TOKEN` | Authentication token from [app.qpoint.io](https://app.qpoint.io) (Settings -> API Tokens). Used for Pulse API authentication and, when using Qpoint-managed S3 storage, as the S3 access key. | ### Pulse Connectivity | Variable | Default | Description | | ---------------- | ----------------------------- | ------------------------------------------------------------------------ | | `PULSE_ENDPOINT` | `https://api-pulse.qpoint.io` | Pulse API endpoint URL. Override only for on-premises Pulse deployments. | ### Worker Pool | Variable | Default | Description | | ----------------------- | ------- | ----------------------------------------------------------------------- | | `NUM_POLLERS` | `1` | Number of concurrent threads polling Pulse for scan jobs. | | `NUM_SCANNERS` | `1` | Number of concurrent threads running PII detection models. | | `MEMORY_LIMIT_FRACTION` | `0.9` | Fraction of available memory QScan is allowed to use before throttling. | ### Metrics | Variable | Default | Description | | -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `METRICS_PORT` | `9090` | Port for the Prometheus metrics and health check HTTP endpoint. We recommend setting this to `8080` in container deployments for compatibility with Cloud Run and common ingress defaults. | ### S3 Storage | Variable | Default | Description | | ----------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `S3_ENDPOINT_URL` | `https://s3.warehouse.qpoint.io` | S3-compatible endpoint URL. Set this when using your own storage (AWS S3, GCS, MinIO). | | `S3_BUCKET_NAME` | `assets` | Name of the S3 bucket where Qtap stores artifacts. | | `S3_REGION_NAME` | `us-east-1` | S3 region. | | `AWS_ACCESS_KEY_ID` | *(derived from REGISTRATION\_TOKEN when using default endpoint)* | S3 access key. When using Qpoint-managed storage, this is set automatically from the registration token. | | `AWS_SECRET_ACCESS_KEY` | `unset` | S3 secret key. Required when using your own S3 storage. | ### Logging | Variable | Default | Description | | -------------- | ----------- | --------------------------------------------------------------- | | `LOG_LEVEL` | *(not set)* | Log verbosity: `debug`, `info`, `warning`, `error`. | | `LOG_ENCODING` | *(not set)* | Log format: `json` for structured logging, omit for plain text. | *** ## S3 Storage Configuration QScan supports any S3-compatible object storage: * **Qpoint-managed storage** (default) -- No additional configuration needed. The registration token is used as the S3 access key automatically. * **AWS S3** -- Set `S3_ENDPOINT_URL` to your S3 endpoint (or omit to use the AWS default), and provide `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. * **Google Cloud Storage** -- Use the S3-compatible endpoint: `https://storage.googleapis.com`. Provide HMAC keys as the access and secret keys. * **MinIO / Garage / other S3-compatible** -- Set `S3_ENDPOINT_URL` to your endpoint and provide credentials. {% hint style="info" %} The S3 configuration in QScan must match the object store configured in your Qtap deployment. QScan reads artifacts that Qtap writes, so both must point to the same bucket and endpoint. {% endhint %} *** ## Qtap-Side Configuration To enable PII scanning, add the `qscan` plugin to a stack in your Qtap configuration (YAML mode) or via Qplane (Stacks and Plugins): ```yaml stacks: sensitive_data_scanning: plugins: - type: http_metrics - type: report_usage - type: access_logs config: mode: summary - type: qscan config: record_document: true cache_ttl: 24h cache_size: 4096 sample_baseline: 10 sample_rate: 0.1 monitors: - type: PERSON record_value: false - type: EMAIL_ADDRESS record_value: false - type: PHONE_NUMBER record_value: false - type: US_SSN record_value: false - type: CREDIT_CARD record_value: false - type: STREET_ADDRESS record_value: false - type: US_BANK_NUMBER record_value: false - type: US_DRIVER_LICENSE record_value: false - type: LOCATION record_value: false - type: ORGANIZATION record_value: false ``` ### Qtap Plugin Options | Option | Default | Description | | ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `record_document` | `false` | Whether to record the full document content alongside PII findings. | | `cache_ttl` | `24h` | How long to cache scan results for repeated requests to the same endpoint. | | `cache_size` | `4096` | Maximum number of entries in the scan result cache. | | `sample_baseline` | `10` | Number of initial requests to always scan before applying the sample rate. | | `sample_rate` | `0.1` | Fraction of requests to scan after the baseline (0.0 to 1.0). | | `qscan_cloud` | `false` | Set to `true` to use Qpoint's hosted QScan cloud service instead of a self-hosted QScan deployment. When enabled, Qtap sends artifacts to Qpoint's cloud for scanning. | | `monitors` | *(none)* | List of PII types to detect. Each entry specifies a `type` and whether to `record_value`. | ### Monitor Types Supported PII types for the `monitors` list: `PERSON`, `EMAIL_ADDRESS`, `PHONE_NUMBER`, `LOCATION`, `US_SSN`, `STREET_ADDRESS`, `CREDIT_CARD`, `US_BANK_NUMBER`, `US_DRIVER_LICENSE`, `ORGANIZATION` Setting `record_value: false` reports the detection without storing the actual sensitive value. *** ## Scaling Guidance ### Pollers vs Scanners * **Pollers** fetch scan jobs from Pulse. Increasing pollers helps when job queue depth is high. * **Scanners** run the ML models. Increasing scanners improves throughput but requires proportionally more memory and CPU. ### Recommended Configurations | Workload | Pollers | Scanners | Memory | CPU | | ------------------------------ | ------- | -------- | ----------- | --------------- | | Light (< 100 scans/hour) | 1 | 1 | 12 GB | 2 vCPUs | | Moderate (100-1000 scans/hour) | 2 | 2 | 24 GB | 6 vCPUs | | Heavy (> 1000 scans/hour) | 2 | 2 | 16 GB + GPU | 4 vCPUs + 1 GPU | For heavy workloads, horizontal scaling (multiple QScan instances) is more effective than increasing pollers and scanners on a single instance. Each scanner thread loads its own copy of the ML models, so memory usage scales linearly with `NUM_SCANNERS`. # Guides Step-by-step tutorials to help you get started and master Qpoint. These guides provide validated, tested examples that you can follow to deploy, configure, and optimize your traffic visibility.

Qplane Guides

  • Cloud-connected deployment
  • Centralized management
  • Dashboard-based monitoring
  • Multi-agent visibility
/pages/WUJEOPsP1DBIce7Ismt0/files/MXe7kK3NRfkDOPAIx8AF

Qtap Guides

  • Standalone deployment
  • YAML-based configuration
  • Local control
  • Maximum flexibility
/pages/YuHPQ5gusnaF8myc3yGv/files/BiXJGvw8gvFvzRblbrMV
## Qplane Guides * [**POC Kick Off Guide**](/guides/qplane-guides/poc-kick-off-guide) ⭐ **Start here!** - Fastest way to see value (10 minutes) * [Hello World - Your First 30 Minutes](/guides/qplane-guides/qplane-hello-world) - Detailed walkthrough after deployment ## Qtap Guides ### Getting Started Perfect for beginners, these guides provide a progressive path from basic setup to production deployment. * [**5-Minute Quickstart**](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) ⭐ **Start here!** - Simplest possible configuration * [Complete Guide: Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive 4-level tutorial (50 min) * [Ingress Traffic Capture](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) - Monitor incoming API requests (15 min) ### Web Server Integration Learn how to capture traffic from popular web servers and proxies. * [NGINX Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) - Reverse proxy monitoring (30 min) * [Caddy Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-caddy-traffic) - Modern web server (30 min) * [Traefik Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-traefik-traffic) - Cloud-native proxy (30 min) * [HAProxy Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-haproxy-traffic) - Load balancer analytics (30 min) * [Envoy Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-envoy-traffic) - Service mesh visibility (30 min) ### Observability & Integration Connect Qtap with your existing observability stack. * [Prometheus + Grafana Monitoring](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) - Metrics and dashboards (45 min) * [OpenTelemetry Integration](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) - OTLP export (30 min) * [Fluent Bit Batching](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) - High-volume batching ### Advanced Use Cases Specialized scenarios and advanced configurations. * [HTTPS Header Capture Without Proxies](/guides/qtap-guides/advanced-use-cases/transparent-https-header-capture-without-proxies) - TLS inspection details (30 min) * [CI Security Validation](/guides/qtap-guides/advanced-use-cases/ci-security-validation-with-qtap) - Build pipeline validation # Qtap Guides Standalone deployment guides for Qtap with local YAML-based configuration. Maximum control and flexibility for self-managed environments. ## What You'll Learn These guides cover: * **Local configuration** - Writing and managing qtap.yaml files * **Storage backends** - Configuring S3, MinIO, and other object stores * **Traffic processing** - Using plugins, stacks, and rulekit expressions * **Conditional capture** - Smart filtering to reduce noise and costs * **Production deployment** - Best practices for long-term operation ## Available Guides ### Getting Started #### [5-Minute Quickstart](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) The simplest possible Qtap setup - everything outputs to your terminal for immediate visibility. **Perfect for:** * Quick testing and experimentation * Development environments * Understanding Qtap basics before adding complexity **What's included:** * Minimal YAML configuration (15 lines) * Stdout-only output (no external services needed) * Test commands to verify it works **Time to complete:** 5 minutes **Skill level:** Beginner *** #### [Complete Guide - From Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) A progressive, hands-on tutorial that takes you from basic setup to production-ready configuration in four levels: **Level 1: Dead Simple - Verify It's Working** (5 minutes) * Basic qtap configuration * Verify HTTPS visibility * Understand output format **Level 2: Basic Filtering and Selective Capture** (10 minutes) * Filter out noisy processes * Apply different capture levels by domain * Use multiple stacks **Level 3: Conditional Capture with Rulekit** (15 minutes) * Use rulekit expressions for intelligent capture * Create reusable macros * Capture only errors and specific request types **Level 4: Production Storage with S3** (20 minutes) * Configure S3-compatible object storage * Keep sensitive data in your network * Implement cost-effective storage strategies **Time to complete:** 50 minutes total (or do one level at a time) **Skill level:** Beginner to Advanced *** #### [Ingress Traffic Capture with Python](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) Learn how to capture **incoming HTTP requests** to your applications using a Python FastAPI server example. **Perfect for:** * Monitoring API endpoints you expose * Debugging incoming client requests * Security auditing of inbound traffic * Understanding ingress vs egress capture **What's included:** * Complete FastAPI server example * Ingress-specific Qtap configuration * Step-by-step testing instructions * Loopback traffic handling **Key concepts:** * `direction: ingress` configuration * `ignore_loopback: false` for localhost capture * Full request/response visibility for incoming traffic **Time to complete:** 15 minutes **Skill level:** Beginner *** ### Web Server Integration #### [Capturing NGINX Traffic](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) Monitor HTTP traffic flowing through NGINX reverse proxy with process-aware visibility. **Time to complete:** 30 minutes **Skill level:** Beginner-Intermediate *** #### [Capturing Caddy Traffic](/guides/qtap-guides/web-server-integration/capturing-caddy-traffic) Observe HTTP traffic in Caddy web server environments. **Time to complete:** 30 minutes **Skill level:** Beginner-Intermediate *** #### [Capturing Traefik Traffic](/guides/qtap-guides/web-server-integration/capturing-traefik-traffic) Capture cloud-native proxy traffic with Traefik integration. **Time to complete:** 30 minutes **Skill level:** Beginner-Intermediate *** #### [Capturing HAProxy Traffic](/guides/qtap-guides/web-server-integration/capturing-haproxy-traffic) Load balancer analytics and visibility with HAProxy. **Time to complete:** 30 minutes **Skill level:** Intermediate *** #### [Capturing Envoy Traffic](/guides/qtap-guides/web-server-integration/capturing-envoy-traffic) Service mesh visibility with Envoy proxy. **Time to complete:** 30 minutes **Skill level:** Intermediate *** ### Observability & Integration #### [Monitoring Qtap with Prometheus and Grafana](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) Complete guide to setting up observability for Qtap using Prometheus metrics and Grafana dashboards. **What you'll learn:** * Exposing Qtap's Prometheus metrics endpoints * Configuring Prometheus to scrape metrics * Importing and customizing Grafana dashboards * Setting up alerts for error rates and latency * Optimizing metrics for production **What's included:** * Prometheus configuration examples (Docker Compose + Kubernetes) * Pre-built Grafana dashboard with RED metrics * Alert rules for errors, latency, and traffic spikes * PromQL queries for troubleshooting * Production optimization tips **Perfect for:** * Production deployments requiring observability * Teams using existing Prometheus/Grafana stacks * Performance monitoring and capacity planning * SRE teams managing Qtap at scale **Time to complete:** 45 minutes **Skill level:** Intermediate *** #### [Sending Qtap Events to OpenTelemetry](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) Send Qtap network observability data to any OpenTelemetry-compatible backend using the OTLP protocol. **What you'll learn:** * Exporting Qtap events as OpenTelemetry Logs * Configuring OTLP endpoints (gRPC and HTTP) * Deploying with OpenTelemetry Collector * Backend-specific configurations (Datadog, Honeycomb, New Relic, Grafana Cloud, Elastic) * Understanding log event structure and attributes * Querying and filtering events **What's included:** * Docker Compose setup with OTel Collector * Kubernetes deployment with OpenTelemetry Operator * Configuration examples for major observability platforms * Complete log structure documentation * Troubleshooting guide **Perfect for:** * Teams using existing OpenTelemetry infrastructure * Organizations standardizing on OTLP * Multi-platform observability strategies * Sending data to SaaS observability platforms **Key insight:** Qtap exports events as **OpenTelemetry Logs**, making it compatible with any log aggregation or observability platform that supports OTLP. Events include connection metadata and HTTP transaction summaries (artifact\_record events) with rich structured attributes. **Time to complete:** 30 minutes **Skill level:** Intermediate *** #### [Capturing All HTTP Traffic with Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) High-volume batching and processing with Fluent Bit integration. **Time to complete:** Variable **Skill level:** Intermediate-Advanced *** ### Advanced Use Cases #### [HTTPS Header Capture Without Proxies](/guides/qtap-guides/advanced-use-cases/transparent-https-header-capture-without-proxies) Deep dive into how Qtap uses eBPF to capture HTTPS traffic without proxies, certificates, or decryption. **What you'll learn:** * How eBPF hooks into TLS libraries (OpenSSL, GoTLS, etc.) * Capturing plaintext before encryption happens * Using rulekit expressions for conditional capture * Storing captured data in S3 * Analyzing captured traffic with jq **Perfect for:** * Understanding how eBPF enables TLS visibility * Security teams evaluating the technology * Advanced users who want granular control **Time to complete:** 30 minutes **Skill level:** Intermediate *** #### [CI Security Validation](/guides/qtap-guides/advanced-use-cases/ci-security-validation-with-qtap) Build pipeline validation and security checks with Qtap. **Time to complete:** Variable **Skill level:** Advanced *** ## Why Choose Qtap Standalone? **Maximum Control** * Full control over configuration via YAML * Version control your configs with git * No dependency on external control plane **Data Sovereignty** * All data stays in your network * Configure your own S3-compatible storage * No external API calls (except to your chosen storage) **Flexibility** * Run in air-gapped environments * Customize every aspect of traffic processing * Use with your existing tooling and workflows *** ## Next Steps **New to Qtap?** Start with the [5-Minute Quickstart](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) or the comprehensive [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide). **Integrating with web servers?** Browse our [Web Server Integration](https://github.com/qpoint-io/documentation/blob/main/guides/web-servers/README.md) guides for NGINX, Caddy, Traefik, HAProxy, and Envoy. **Need observability?** Check out [Prometheus/Grafana](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana), [OpenTelemetry](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry), or [Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) integration. **Want centralized management?** Check out the [Qplane Guides](/guides/qplane-guides) for cloud-connected deployment. # Getting Started # 5-Minute Quickstart Get Qtap running in 5 minutes with console output - perfect for a fast preview of Qtap's capabilities. ## Who This Is For **Use this guide if you want to:** * Preview Qtap's traffic capture quickly without complex setup * See immediate console output (no storage configuration needed) * Understand basic Qtap concepts before diving deeper * Test if Qtap works on your system **This is NOT for:** * Production deployments (use [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) instead) * Long-term traffic storage (add [S3 configuration](/guides/qtap-guides/getting-started/getting-started-complete-guide#level-4-add-s3-storage-for-production) later) * Multi-environment management (consider [Qplane](/guides/qplane-guides/poc-kick-off-guide)) **Time to complete:** 5 minutes **What you'll get:** Real-time visibility into HTTP/HTTPS traffic with console output *** This is a simple configuration to get started with Qtap without any external dependencies. Everything outputs to your console for immediate visibility. ### Quick Start Configuration Save this as `qtap-starter.yaml`: ```yaml version: 2 # Storage Configuration - Everything to console services: # Event metadata goes to stdout event_stores: - type: stdout # Object data (headers) goes to stdout object_stores: - type: stdout # Processing Stack - Simple HTTP capture stacks: starter_stack: plugins: # HTTP Capture plugin - outputs to console - type: http_capture config: level: headers # Capture headers (use 'full' for bodies too) format: text # Human-readable format (use 'json' for structured) # Traffic Capture Settings tap: direction: egress # Capture outgoing traffic ignore_loopback: true # Skip localhost traffic audit_include_dns: false # Skip DNS queries for cleaner output http: stack: starter_stack ``` ### Running the Starter Config #### Quick Test ```bash # Install/Update QTap curl -s https://get.qpoint.io/install | sudo sh # Run with the config (runs in foreground by default) sudo qtap --config=qtap-starter.yaml ``` ### Testing Your Configuration Once QTap is running, test it with some HTTP requests on the same host: ```bash # In another terminal, make test requests: # Test 1: Simple GET request curl https://httpbin.org/get # Test 2: POST with headers curl -X POST https://httpbin.org/post \ -H "X-Test-Header: test-value" \ -d "test=data" ``` You should see output in your QTap terminal showing the captured traffic. ### Configuration Variations #### Variation 1: Capture Everything (Including Bodies) ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug_stack: plugins: - type: http_capture config: level: full # Capture everything including bodies format: json # Structured output for parsing tap: direction: all # Capture both ingress and egress ignore_loopback: false # Include localhost traffic audit_include_dns: true # Include DNS queries http: stack: debug_stack ``` #### Variation 2: With Filtering Rules ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: filtered_stack: plugins: - type: http_capture config: level: summary # Default: just basic info format: text rules: # Capture headers for specific domains - name: "API calls" expr: http.req.host contains "api" level: headers # Capture everything for errors - name: "Error debugging" expr: http.res.status >= 400 level: full # Skip health checks entirely - name: "Ignore health" expr: http.req.path in ["/health", "/ping"] level: none tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: filtered_stack ``` #### Variation 3: Using access\_logs Plugin ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: logging_stack: plugins: # Access logs plugin - old - type: access_logs config: mode: details # Options: summary, details, full format: console # Human-readable format tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: logging_stack ``` ### Understanding the Output #### With `http_capture` plugin (text format): ``` [HTTP] GET https://httpbin.org/get Headers: User-Agent: curl/7.81.0 Accept: */* Status: 200 OK Duration: 145ms ``` #### With `http_capture` plugin (json format): ```json { "timestamp": "2024-10-15T10:23:45Z", "http": { "method": "GET", "host": "httpbin.org", "path": "/get", "status": 200, "headers": { "user-agent": "curl/7.81.0" } }, "direction": "egress" } ``` #### With `access_logs` plugin: ``` 2024-10-15T10:23:45Z [EGRESS] GET httpbin.org/get 200 145ms ``` ### Troubleshooting #### Quick Checklist: Not Working? Use this checklist to diagnose common issues: * [ ] **Qtap is running**: `sudo systemctl status qtap` or `docker ps | grep qtap` * [ ] **Correct config file**: Verify YAML syntax with `yamllint qtap-starter.yaml` * [ ] **Sufficient permissions**: Running as root/sudo with necessary capabilities * [ ] **eBPF support**: Kernel 5.10+ with `uname -r`, eBPF enabled * [ ] **Traffic being generated**: Make HTTP requests with `curl https://example.com` * [ ] **Correct direction**: Use `egress` for outbound (curl), `ingress` for inbound (web server) * [ ] **Not filtered**: Check if process is in `filters` section * [ ] **Logs show activity**: Check qtap logs for errors with `--log-level=debug` #### No Output Appearing **Symptom:** Qtap is running but no traffic is captured **Solutions:** 1. **Check for errors in debug mode:** ```bash sudo qtap --config=qtap-starter.yaml --log-level=debug ``` Look for messages like "attached to process" or "http capture" 2. **Verify traffic is being generated:** ```bash # Make sure this runs AFTER qtap is started curl -v https://httpbin.org/get ``` 3. **Check process attribution:** ```bash # See if qtap detected the curl process ps aux | grep curl ``` 4. **Verify eBPF hooks:** ```bash # Check if eBPF programs are loaded sudo bpftool prog list | grep qtap ``` **Common causes:** * Qtap started AFTER traffic was generated (restart qtap, then generate traffic) * Process doesn't use standard TLS libraries (OpenSSL, BoringSSL, GnuTLS) * Traffic is going through a proxy that qtap isn't monitoring #### Too Much Output **Symptom:** Overwhelming amount of captured traffic **Solution:** Add filters to ignore noisy processes: ```yaml tap: filters: groups: - kubernetes # Ignore k8s system traffic - qpoint # Ignore QTap itself custom: - exe: /usr/bin/prometheus strategy: exact - exe: /opt/datadog-agent strategy: prefix ``` **Common noisy processes:** * Monitoring agents (Prometheus, Datadog, New Relic) * Log shippers (Fluent Bit, Logstash) * Kubernetes system components * Container runtime health checks #### Wrong Traffic Direction **Symptom:** Expected traffic not appearing **Solution:** Choose the correct direction: * `egress`: Outgoing traffic from your system (e.g., curl making requests) * `ingress`: Incoming traffic to your services (e.g., web server receiving requests) * `all`: Both directions **Examples:** * Testing with curl → use `egress` * Monitoring NGINX web server → use `ingress` * Capturing both client and server traffic → use `all` #### Protocol Shows as "other" Instead of "http" **Symptom:** Traffic captured but not parsed as HTTP **Possible causes:** 1. **HTTP/3 (QUIC)**: Not yet supported - shows as "other" 2. **Non-standard TLS library**: Qtap supports OpenSSL, BoringSSL, GnuTLS 3. **Binary protocol**: Not HTTP traffic (database protocols, gRPC, etc.) **How to verify:** ```bash # Check qtap debug logs for TLS detection sudo qtap --config=qtap-starter.yaml --log-level=debug 2>&1 | grep -i "tls" ``` #### Still Having Issues? * **Check system requirements:** [System Requirements](/getting-started/qtap/system-requirements) * **Run preflight check:** `curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash` * **See detailed troubleshooting:** [Troubleshooting Guide](/troubleshooting) * **Review healthy capture patterns:** [What a Healthy Capture Looks Like](/appendix/healthy-capture-patterns) ### What's Next? Now that you've confirmed Qtap is capturing traffic, choose your next step based on your goals: #### **For Production Deployment** Follow the progressive tutorial with real-world examples: * [Complete Guide: Hello World to Production](/guides/qtap-guides/getting-started/getting-started-complete-guide) - 50-minute tutorial covering rules, filtering, S3 storage, and production deployment #### **For Specific Use Cases** * **Web servers (NGINX, Caddy, etc.):** [NGINX Traffic Capture](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) * **Ingress traffic:** [Ingress Traffic Capture](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) * **Production debugging:** [Production Debugging with HTTPS Visibility](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) #### **For Configuration Details** * **Add S3 storage:** [Storage Configuration](/getting-started/qtap/configuration/storage-configuration#s3-compatible-object-storage) * **Add filtering rules:** [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings#filters) * **Understand plugins:** [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) * **See more examples:** [Configuration Examples](/getting-started/qtap/configuration/configuration-examples) #### **Quick Improvements to This Setup** 1. **Add error-only capture** - Reduce volume by only capturing errors: ```yaml # Add to your config rulekit: macros: - name: is_error expr: http.res.status >= 400 stacks: starter_stack: plugins: - type: http_capture config: level: none # Don't capture by default format: text rules: - name: "Capture errors only" expr: is_error() level: full ``` 2. **Add process filtering** - Ignore noisy processes: ```yaml # Add to tap section tap: filters: custom: - exe: /usr/bin/prometheus strategy: exact ``` 3. **Switch to JSON** - For easier parsing and integration: ```yaml # Change format in http_capture plugin format: json # instead of text ``` ### Minimal Test Config The absolute minimum config to see if QTap works: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: test: plugins: - type: access_logs config: mode: details format: console tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: test ``` Save as `test.yaml` and run: ```bash sudo qtap --config=test.yaml ``` Then in another terminal on same host: ```bash curl https://example.com ``` You should immediately see the captured request in your QTap terminal. # Sending All Traffic to S3 Capture all HTTP/HTTPS traffic and store it in S3-compatible storage for persistent, secure access. ## Who This Is For **Use this guide if you want to:** * Store all captured traffic persistently in S3 * Keep sensitive HTTP data within your own infrastructure * Set up a simple, no-filtering capture pipeline * Get started with S3 storage before adding rules later **Prerequisites:** * S3-compatible storage ready (AWS S3, MinIO, Google Cloud Storage) * S3 credentials (access key and secret key) * S3 bucket created **Time to complete:** 10 minutes *** ## Configuration Save this as `qtap-s3.yaml`: ```yaml version: 2 # Storage Configuration services: # Connection metadata to console (for visibility) event_stores: - type: stdout # HTTP payloads to S3 (where sensitive data lives) object_stores: - type: s3 endpoint: s3.amazonaws.com # Your S3 endpoint bucket: my-qtap-data # Your bucket name region: us-east-1 # Your region access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false # Use HTTPS access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY # Processing Stack stacks: capture_all: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: json # (json|text) - Structured for storage # Traffic Capture Settings tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) - Skip localhost audit_include_dns: false # (true|false) - Skip DNS queries http: stack: capture_all ``` ## Running Qtap ### Set Your S3 Credentials ```bash export S3_ACCESS_KEY=your_access_key export S3_SECRET_KEY=your_secret_key ``` ### Start Qtap with Docker ```bash docker run -d \ --name qtap-s3 \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-s3.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ -e S3_ACCESS_KEY \ -e S3_SECRET_KEY \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" ``` ### Or with Linux Binary ```bash sudo S3_ACCESS_KEY=$S3_ACCESS_KEY S3_SECRET_KEY=$S3_SECRET_KEY \ qtap --config=qtap-s3.yaml ``` ## Testing Wait for qtap to initialize, then generate some traffic: ```bash sleep 6 curl https://httpbin.org/get ``` Check the logs for confirmation: ```bash docker logs qtap-s3 2>&1 | grep "httpbin" ``` Verify objects are in your S3 bucket: ```bash aws s3 ls s3://my-qtap-data/ ``` ## S3 Provider Examples ### AWS S3 ```yaml object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: my-qtap-data region: us-west-2 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` ### MinIO (Self-Hosted) ```yaml object_stores: - type: s3 endpoint: minio.internal:9000 bucket: qtap-data region: us-east-1 access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` ### Google Cloud Storage ```yaml object_stores: - type: s3 endpoint: storage.googleapis.com bucket: my-qtap-data region: us-central1 access_url: https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: GCS_ACCESS_KEY secret_key: type: env value: GCS_SECRET_KEY ``` ## Understanding the Output With this configuration: * **Console output** shows connection metadata (timestamps, endpoints, duration) * **S3 bucket** stores full HTTP payloads (headers, bodies, sensitive data) Each captured transaction creates an object in S3 with a unique digest identifier. The `access_url` template lets you construct URLs to retrieve stored objects. ## Capture Levels Adjust the `level` setting based on your needs: | Level | What's Captured | Use Case | | --------- | --------------------------------- | ------------------------------------ | | `none` | Nothing | Use with rules for selective capture | | `summary` | Method, URL, status, duration | Lightweight monitoring | | `headers` | Summary + all headers | Header inspection | | `full` | Headers + request/response bodies | Complete visibility | ## What's Next? * **Add filtering rules**: [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) * **Capture only errors**: Set `level: none` as default, then add rules to capture `http.res.status >= 400` at `level: full` * **Kubernetes deployment**: [Helm Chart](/getting-started/qtap/installation/helm-chart) with S3 secrets * **Full configuration reference**: [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) # Complete Guide: Hello World to Production Progressive 4-level tutorial that takes you from "hello world" to production-ready Qtap deployment. Perfect for teams wanting full control with YAML configuration. ## Who This Is For **Use this guide if you want to:** * Learn Qtap systematically from basics to advanced features * Deploy Qtap for long-term production monitoring * Understand how all Qtap features work together * Build self-managed infrastructure with version-controlled configs **Choose this guide for:** * Production deployments with S3 storage * Rule-based selective capture (capture errors only, specific endpoints, etc.) * Air-gapped or isolated environments * GitOps workflows with infrastructure-as-code **Choose something else if you:** * Need visibility RIGHT NOW for debugging → [Production Debugging](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) (30 seconds) * Want a quick preview → [5-Minute Quickstart](/guides/qtap-guides/getting-started/qtap-starter-configuration-stdout-only) (5 minutes) * Prefer managed dashboards → [Qplane POC Guide](/guides/qplane-guides/poc-kick-off-guide) (10 minutes) **Time to complete:** 50 minutes (can pause between levels) *** This guide takes you from a simple "hello world" setup to a production-ready configuration, with every example validated and tested. Follow along at your own pace - each level builds on the previous one. **What you'll learn:** * Level 1: Dead simple setup - verify qtap is working (10 min) * Level 2: Basic filtering and selective capture (10 min) * Level 3: Conditional capture with rulekit expressions (15 min) * Level 4: Production storage with S3 (15 min) **Prerequisites:** * Docker installed and running * Basic familiarity with YAML * curl or similar HTTP client *** ## Level 1: Dead Simple - Verify It's Working **Goal**: Get qtap running and see it capture HTTPS traffic in under 5 minutes. **What you'll learn:** * Basic qtap configuration * How to verify qtap sees inside HTTPS * Understanding qtap output ### Step 1: Create Your First Config Create a file called `qtap-level1.yaml`: ```yaml version: 2 # Where to send captured data (stdout = your terminal) services: event_stores: - type: stdout object_stores: - type: stdout # What to do with captured traffic stacks: basic_stack: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: text # (json|text) - Human-readable output # What traffic to capture tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) - Outbound traffic only ignore_loopback: true # (true|false) - Skip localhost traffic audit_include_dns: false # (true|false) - Don't capture DNS queries http: stack: basic_stack # Use the stack we defined above ``` **What this does:** * Captures all outbound HTTP/HTTPS traffic * Shows full request/response details in your terminal * No filters - captures everything (except localhost) ### Step 2: Start Qtap ```bash docker run -d \ --name qtap-level1 \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-level1.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" ``` **Wait for qtap to initialize** (important!): ```bash sleep 6 ``` ### Step 3: Generate Test Traffic ```bash docker run --rm curlimages/curl -s https://httpbin.org/get ``` ### Step 4: See What Qtap Captured ```bash docker logs qtap-level1 2>&1 | grep -A 50 "httpbin.org" ``` **What you should see:** ``` HTTP Transaction ================ Metadata: Transaction Time: 2025-10-14T17:10:57Z Duration: 3356ms Direction: egress-external Process ID: 230253 Process: /usr/bin/curl Bytes Sent: 41 Bytes Received: 395 Request: Method: GET URL: https://httpbin.org/get Headers: :path: /get :scheme: https User-Agent: curl/8.12.1 Accept: */* :authority: httpbin.org :method: GET Response: Status: 200 Content-Type: application/json Headers: Content-Length: 255 Server: gunicorn/19.9.0 Access-Control-Allow-Origin: * Access-Control-Allow-Credentials: true :status: 200 Date: Tue, 14 Oct 2025 17:10:57 GMT Content-Type: application/json Body: { "args": {}, "headers": { "Accept": "*/*", "Host": "httpbin.org", "User-Agent": "curl/8.12.1", "X-Amzn-Trace-Id": "Root=1-68ee841e-1663d2ae3f156ba80e3a3a36" }, "origin": "73.71.138.108", "url": "https://httpbin.org/get" } ``` **🎉 Success indicators:** * ✅ Clean formatted output (no internal logging clutter) * ✅ You see the full URL despite HTTPS encryption * ✅ All request and response headers are visible (including HTTP/2 pseudo-headers like `:path`, `:authority`) * ✅ The response body (JSON) is captured in full * ✅ Rich metadata: process info, duration, bytes sent/received ### Understanding the Output Let's break down what qtap showed you: 1. **Metadata Section**: * **Transaction Time** - When the request occurred * **Duration** - How long the request took (3356ms) * **Direction** - `egress-external` (outbound to external IP) * **Process** - `/usr/bin/curl` - Qtap knows which process made the request * **Bytes Sent/Received** - Network traffic volume 2. **Request Section**: * **Method & URL** - GET request to * **Headers** - Including HTTP/2 pseudo-headers (`:path`, `:scheme`, `:authority`, `:method`) * Standard headers like `User-Agent` and `Accept` 3. **Response Section**: * **Status** - 200 OK * **Headers** - All response headers including server, content-type, etc. * **Body** - Complete JSON response payload **How does qtap see inside HTTPS?** Qtap uses eBPF to hook into system calls *before* TLS encryption happens. It doesn't break TLS or act as a proxy - it just observes what your application sends before OpenSSL encrypts it. ### Troubleshooting **Don't see any output?** 1. Make sure qtap was running BEFORE you generated traffic 2. Wait 6 seconds after starting qtap before testing 3. Check qtap is running: `docker ps | grep qtap-level1` **See connection info but no HTTP details?** * Look for `"l7Protocol": "other"` - means HTTP parsing failed * Check if you're looking at the right traffic (search for your test domain) **Clean up when done**: ```bash docker rm -f qtap-level1 ``` *** ## What's Next? ✅ **Level 1 Complete!** You now know: * How to configure qtap basics * How to verify qtap is capturing traffic * That qtap can see inside HTTPS **Next up**: Level 2 - Basic Filtering and Selective Capture * Filter out noisy processes * Apply different capture levels by domain * Use multiple stacks for different traffic types *** ## Level 2: Basic Filtering and Selective Capture **Goal**: Control what traffic gets captured and apply different capture levels to different traffic. **What you'll learn:** * Filter out noisy processes (curl, wget) * Apply different capture levels by domain * Use different capture levels for different endpoints ### Why Filtering Matters In real environments, you'll see a LOT of traffic. Level 1 captures everything, which means: * Kubernetes health checks flood your logs * Package managers create noise * Your own debugging with curl shows up * Monitoring agents add clutter Level 2 teaches you to be selective. ### Step 1: Create a Filtered Config Create `qtap-level2.yaml`: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: # Detailed stack for important APIs detailed_stack: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Full capture format: text # (json|text) # Lightweight stack for everything else lightweight_stack: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Just basic info format: text # (json|text) tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) audit_include_dns: false # (true|false) # Default: use lightweight stack http: stack: lightweight_stack # Filter out noisy processes filters: groups: - qpoint # Don't capture qtap's own traffic custom: - exe: /usr/bin/curl strategy: exact # Filter out manual curl commands # Apply specific stacks to specific domains endpoints: - domain: 'httpbin.org' http: stack: detailed_stack # Full capture for httpbin - domain: 'api.github.com' http: stack: detailed_stack # Full capture for GitHub API ``` **What's new here:** * **Two stacks**: Detailed (full) vs lightweight (summary only) * **Process filters**: Ignore curl (filter out manual testing) * **Domain-specific stacks**: httpbin.org uses detailed\_stack for full capture * **Default lightweight**: Everything else is summary only ### Step 2: Start Level 2 ```bash docker run -d \ --name qtap-level2 \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-level2.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" sleep 6 ``` ### Step 3: Test the Filters **Test 1: Filtered traffic (should NOT appear)** ```bash docker run --rm curlimages/curl -s https://example.com/get ``` Check logs: ```bash docker logs qtap-level2 2>&1 | grep "example.com" ``` **Expected**: Nothing! curl is filtered. **Test 2: Domain-specific capture (should appear with FULL details)** ```bash docker run --rm alpine sh -c "wget -O- https://httpbin.org/get 2>/dev/null" ``` Check logs: ```bash docker logs qtap-level2 2>&1 | grep -A 20 "httpbin.org" ``` **Expected**: Full HTTP details including body because: 1. httpbin.org matches our endpoint rule 2. Endpoint rule applies `detailed_stack` for full capture 3. wget (/bin/busybox) is NOT filtered, so traffic is captured ### Understanding the Results **What you should see:** 1. **Curl traffic**: Filtered out ```bash # This returns nothing docker logs qtap-level2 | grep -c "curl" # Output: 0 ``` 2. **httpbin.org via wget**: CAPTURED with full details * Method, URL, headers, body all visible * Because it matches the endpoint rule 3. **Other domains**: Would show summary only (if we tested them) * Basic info: method, URL, status, duration * No headers, no bodies ### Why This Matters This configuration gives you: * **Less noise**: Filters out development tools * **Selective detail**: Full capture only where you need it * **Better performance**: Summary-only for bulk traffic * **Cost savings**: Less data stored/transmitted ### Advanced: Using Prefix Filters Want to filter an entire directory? ```yaml filters: custom: - exe: /usr/bin/ strategy: prefix # Filters ALL /usr/bin/* processes ``` This would block `/usr/bin/curl`, `/usr/bin/wget`, `/usr/bin/python3`, etc. ### Clean Up ```bash docker rm -f qtap-level2 ``` *** ## What's Next? ✅ **Level 2 Complete!** You now know: * How to filter noisy processes * How to apply different capture levels by domain * How to use multiple stacks for different traffic types **Next up**: Level 3 - Conditional Capture with Rulekit * Use rulekit expressions for intelligent capture * Create reusable macros * Capture only errors and specific request types * Filter by container and pod labels *** ## Level 3: Conditional Capture with Rulekit **Goal**: Use rulekit expressions for intelligent, conditional traffic capture. **What you'll learn:** * Capture only errors and POST requests * Use different capture levels based on conditions * Create reusable macros for complex logic * Filter by container and pod labels ### Why Conditional Capture Matters In high-traffic environments, capturing everything is expensive and noisy. You want: * **Selective detail**: Full capture only when needed * **Error focus**: Always capture failures * **Method-based filtering**: Capture mutating operations (POST, PUT, DELETE) * **Smart sampling**: Reduce volume without losing critical data ### Step 1: Create a Rulekit-Based Config Create `qtap-level3.yaml`: ```yaml version: 2 # Define reusable macros rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 - name: is_post expr: http.req.method == "POST" - name: is_api_call expr: http.req.path matches /^\/api\// services: event_stores: - type: stdout object_stores: - type: stdout stacks: # Smart stack: Conditional capture levels smart_stack: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Default: basic info only format: text # (json|text) rules: # Full capture for errors - name: "Capture all errors with full details" expr: is_error() level: full # Full capture for POST requests - name: "Capture POST requests" expr: is_post() level: full # Details for API calls (headers, no body) - name: "Capture API calls with headers" expr: is_api_call() level: headers # Error-only stack: Only capture failures error_only_stack: plugins: - type: http_capture config: level: none # (none|summary|headers|full) - Default: capture nothing format: text # (json|text) rules: # Only capture 4xx and 5xx - name: "Client errors" expr: http.res.status >= 400 && http.res.status < 500 level: full - name: "Server errors" expr: http.res.status >= 500 level: full tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) audit_include_dns: false # (true|false) http: stack: smart_stack # Default: smart conditional capture filters: groups: - qpoint custom: - exe: /usr/bin/curl strategy: exact # (exact|prefix|regex) # Apply different stacks to different domains endpoints: # GitHub API: error-only capture - domain: 'api.github.com' http: stack: error_only_stack # httpbin: smart capture with all rules - domain: 'httpbin.org' http: stack: smart_stack ``` ### Step 2: Understanding Rulekit **Macros:** * Reusable expressions defined once, used many times * Called like functions: `is_error()`, `is_slow()` * Make configs cleaner and easier to maintain **Capture Levels in Rules:** * `none`: Skip capture entirely * `summary`: Basic info (method, URL, status, duration, process) * `details`: Include headers (no bodies) * `full`: Everything (headers + bodies) **Available Fields:** * Request: `http.req.method`, `http.req.path`, `http.req.host`, `http.req.url` * Response: `http.res.status` * Headers: `http.req.headers.`, `http.res.headers.` * Source: `src.container.name`, `src.container.labels.`, `src.pod.name`, `src.pod.labels.` ### Step 3: Start Level 3 ```bash docker run -d \ --name qtap-level3 \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-level3.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" sleep 6 ``` ### Step 4: Test Conditional Capture **Test 1: Success (should show SUMMARY - basic info only):** ```bash docker run --rm alpine sh -c "wget -O- http://httpbin.org/get 2>/dev/null" ``` Check logs - should see basic metadata (method, URL, status) but no headers or body: ```bash docker logs qtap-level3 2>&1 | grep -A 20 "HTTP Transaction" ``` **Test 2: Error (should show FULL details):** ```bash docker run --rm alpine sh -c "wget -O- http://httpbin.org/status/404 2>/dev/null" || true ``` Check logs - should see full capture with all details: ```bash docker logs qtap-level3 2>&1 | grep -A 20 "404" ``` **Test 3: API path (should show DETAILS with headers):** ```bash docker run --rm alpine sh -c "wget -O- http://httpbin.org/api/test 2>/dev/null" || true ``` ### Understanding the Results **What you should see:** 1. **Success requests**: Summary level (default) * Method, URL, status code * Duration, process info * No headers, no bodies 2. **Error requests (404, 500)**: Full capture * Complete request and response headers * Full request/response bodies * Because they match `is_error()` macro 3. **API paths**: Details level * Headers included * No bodies (saves space) * Because path matches `/^\/api\//` ### Advanced Rulekit Patterns **Container-based filtering:** ```yaml rules: - name: "Debug specific container" expr: src.container.name == "my-app" level: full ``` **Regex matching:** ```yaml rules: - name: "External APIs" expr: http.req.host matches /\.(googleapis\.com|amazonaws\.com)$/ level: headers ``` **Complex conditions:** ```yaml rules: - name: "Production API errors" expr: is_api_call() && is_error() && src.pod.labels.env == "production" level: full ``` ### Clean Up ```bash docker rm -f qtap-level3 ``` *** ## Level 4: Production Storage with S3 **Goal**: Store captured traffic in S3 for long-term retention and compliance. **What you'll learn:** * Configure S3-compatible object storage * Use MinIO for local testing * Combine stdout (debugging) with S3 (persistence) * Production storage best practices ### Why S3 Storage Matters **The primary benefit of S3 storage is keeping sensitive data within your network boundary.** When you use S3-compatible storage (MinIO, AWS S3, GCS), captured HTTP traffic containing sensitive information (API keys, tokens, PII, etc.) never leaves your infrastructure. This is critical for: * **Data sovereignty**: Keep sensitive data in your own network/region * **Security**: Prevent data exfiltration - traffic never goes to external services * **Compliance**: Meet regulatory requirements (GDPR, HIPAA, SOC2) * **Control**: You own and control access to all captured data Additionally, S3 provides: * **Forensics**: Investigate security incidents weeks/months later * **Analytics**: Analyze traffic patterns over time * **Debugging**: Review full request/response data when issues occur * **Durability**: Long-term retention with lifecycle policies stdout is great for development, but production requires secure, persistent storage within your control. ### Step 1: Set Up MinIO (Local S3) For this guide, we'll use MinIO - an S3-compatible storage you can run locally: ```bash # Start MinIO docker run -d \ --name minio \ -p 9000:9000 \ -p 9001:9001 \ -e MINIO_ROOT_USER=minioadmin \ -e MINIO_ROOT_PASSWORD=minioadmin \ quay.io/minio/minio server /data --console-address ":9001" sleep 3 # Create bucket docker run --rm \ --network=host \ --entrypoint /bin/sh \ quay.io/minio/minio -c " mc alias set local http://localhost:9000 minioadmin minioadmin && mc mb local/qtap-captures && echo 'Bucket created successfully!' " ``` ### Step 2: Create S3-Enabled Config Create `qtap-level4-s3.yaml`: ```yaml version: 2 # Reusable macros rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 services: # Events: Still use stdout for real-time monitoring event_stores: - type: stdout # Objects: Store in S3 for persistence object_stores: - type: s3 endpoint: localhost:9000 bucket: qtap-captures region: us-east-1 insecure: true # Only for local MinIO testing access_key: type: text value: minioadmin secret_key: type: text value: minioadmin stacks: production_stack: plugins: - type: http_capture config: level: none # (none|summary|headers|full) - Default: don't capture format: json # (json|text) rules: # Only capture errors (saves S3 costs) - name: "Capture errors" expr: is_error() level: full # (none|summary|headers|full) tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) audit_include_dns: false # (true|false) http: stack: production_stack filters: groups: - qpoint custom: - exe: /usr/bin/curl strategy: exact # (exact|prefix|regex) ``` **What's different:** * **S3 object\_store**: Captured HTTP transactions go to MinIO * **stdout event\_store**: Connection metadata still goes to console * **Error-only capture**: Only store failures (reduces costs) ### Step 3: Start Qtap with S3 ```bash docker run -d \ --name qtap-level4-s3 \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-level4-s3.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" sleep 6 ``` ### Step 4: Test S3 Storage **Generate an error (will be stored in S3):** ```bash docker run --rm alpine sh -c "wget -O- http://httpbin.org/status/500 2>/dev/null" || true sleep 2 ``` **Check qtap logs for S3 upload confirmation:** ```bash docker logs qtap-level4-s3 2>&1 | grep -i "s3\|upload\|object" ``` **List objects in MinIO:** ```bash docker run --rm \ --network=host \ --entrypoint /bin/sh \ quay.io/minio/minio -c " mc alias set local http://localhost:9000 minioadmin minioadmin && mc ls local/qtap-captures/ " ``` You should see files with SHA digest names - these are your captured HTTP transactions! ### Step 5: Retrieve a Captured Transaction **Get the digest from qtap logs:** ```bash DIGEST=$(docker logs qtap-level4-s3 2>&1 | grep '"digest":' | head -1 | grep -o '"digest":"[^"]*"' | cut -d'"' -f4) echo "Digest: $DIGEST" ``` **Download the object from MinIO:** ```bash docker run --rm \ --network=host \ --entrypoint /bin/sh \ quay.io/minio/minio -c " mc alias set local http://localhost:9000 minioadmin minioadmin && mc cat local/qtap-captures/$DIGEST " ``` You'll see the full HTTP transaction - request headers, response headers, and body! ### Understanding S3 Configuration **S3 Parameters:** | Parameter | Purpose | Example | | ------------ | ------------------------ | ---------------------------------------- | | `endpoint` | S3 server address | `s3.amazonaws.com` or `localhost:9000` | | `bucket` | Where to store objects | `qtap-captures` | | `region` | S3 region | `us-east-1` | | `insecure` | Allow HTTP (MinIO only!) | `true` for local, `false` for production | | `access_key` | S3 credentials | From env or text | | `secret_key` | S3 credentials | From env or text | **Production S3 Config (AWS):** ```yaml object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: my-company-qtap region: us-west-2 insecure: false # Always false in production! access_key: type: env # Use environment variables value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` Then run qtap with: ```bash docker run ... \ -e AWS_ACCESS_KEY_ID=your_key \ -e AWS_SECRET_ACCESS_KEY=your_secret \ ... ``` ### Production Best Practices **Storage Strategy:** * ✅ Use S3 for object storage (HTTP transactions) * ✅ Use stdout or logging service for events (metadata) * ✅ Only capture what you need (errors, slow requests) * ✅ Set S3 lifecycle rules to auto-delete old data **Security:** * ✅ Always use `insecure: false` in production * ✅ Use environment variables for credentials (never hardcode) * ✅ Enable S3 bucket encryption * ✅ Restrict S3 bucket access with IAM policies **Cost Optimization:** * ✅ Use `level: none` by default, capture via rules only * ✅ Capture errors at `full` level (need details to debug) * ✅ Capture success at `summary` or `details` level * ✅ Use S3 lifecycle policies to transition to cheaper storage tiers ### S3 Lifecycle Example For AWS S3, set up lifecycle rules: ```json { "Rules": [ { "Id": "TransitionOldCaptures", "Status": "Enabled", "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER" } ], "Expiration": { "Days": 365 } } ] } ``` This: * Keeps data in standard storage for 30 days * Moves to infrequent access after 30 days (cheaper) * Archives to Glacier after 90 days (very cheap) * Deletes after 1 year (compliance) ### Clean Up ```bash # Stop qtap docker rm -f qtap-level4-s3 # Stop MinIO (optional - keeps your test data) docker rm -f minio ``` *** ## Congratulations! 🎉 You've completed the Ultimate Qtap Setup Guide. You now know how to: **Level 1 - Basics:** * ✅ Configure qtap to capture HTTP/HTTPS traffic * ✅ Verify qtap can see inside TLS connections * ✅ Understand qtap output format **Level 2 - Filtering:** * ✅ Filter processes to reduce noise * ✅ Apply different capture levels by domain * ✅ Use different capture levels for different endpoints **Level 3 - Conditional Capture:** * ✅ Use rulekit expressions for intelligent capture * ✅ Create reusable macros * ✅ Capture only errors and specific request types * ✅ Filter by container and pod labels **Level 4 - Production Storage:** * ✅ Configure S3-compatible object storage * ✅ Combine stdout and S3 for debugging + persistence * ✅ Implement cost-effective storage strategies * ✅ Follow security best practices *** ## Next Steps ### Production Readiness Ready to deploy qtap in production? Here's your checklist: **Configuration Reference:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - Complete S3 setup, credentials management, and event stores * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - All available plugins and advanced options * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Filters, endpoints, and direction settings * [Rulekit Documentation](https://github.com/qpoint-io/rulekit) - Advanced expression syntax and examples **Alternative: Cloud Management** Want centralized management with visual dashboards? * [Qplane Overview](/getting-started/qplane) - Cloud control plane for managing multiple agents * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Quick start with Qplane (10 minutes) **Get Help:** * [Troubleshooting](/troubleshooting) - Common issues and solutions * Report issues on [GitHub](https://github.com/qpoint-io/qtap/issues) *** *All examples in this guide have been validated and tested. Every configuration is guaranteed to work.* # Ingress Traffic Capture This guide shows you how to use Qtap to capture **ingress (incoming) HTTP traffic** to your applications. We'll use a simple Python FastAPI web server as an example, but the same principles apply to any HTTP server. ## What You'll Learn * Configure Qtap to capture incoming HTTP requests * See request headers, bodies, and metadata from client requests * Understand the difference between ingress and egress capture * Monitor API endpoints your application exposes ## Use Cases **Why capture ingress traffic?** * Debug API requests from clients * Monitor what data clients are sending to your endpoints * Audit incoming traffic for security analysis * Understand client behavior and usage patterns * Troubleshoot issues with incoming requests *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Python 3.7+ installed * Qtap installed (see [Installation](/getting-started/qtap/installation)) * Root/sudo access *** ## Part 1: The Python Web Server First, let's create a simple FastAPI web server that we'll monitor. ### Install FastAPI and Uvicorn ```bash pip install fastapi uvicorn ``` ### Create the Web Server Save this as `api_server.py`: ```python from fastapi import FastAPI, Header from typing import Optional import uvicorn app = FastAPI() @app.get("/") async def root(): """Simple health check endpoint""" return {"message": "API server is running", "status": "healthy"} @app.get("/api/users/{user_id}") async def get_user(user_id: int, x_request_id: Optional[str] = Header(None)): """Get user by ID with optional request tracking header""" return { "user_id": user_id, "name": f"User {user_id}", "email": f"user{user_id}@example.com", "request_id": x_request_id } @app.post("/api/users") async def create_user(user_data: dict): """Create a new user""" return { "message": "User created successfully", "data": user_data, "id": 12345 } @app.get("/api/slow") async def slow_endpoint(): """Simulate a slow endpoint""" import time time.sleep(2) return {"message": "This took 2 seconds"} if __name__ == "__main__": # Listen on all interfaces so other containers/machines can reach it # This is required for true ingress capture from network sources uvicorn.run(app, host="0.0.0.0", port=8000) ``` ### Start the Server In one terminal, start the Python server: ```bash python api_server.py ``` You should see: ``` INFO: Started server process [12345] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ``` The server is now listening on all network interfaces, making it accessible from other containers or machines on your network. *** ## Part 2: Qtap Configuration for Ingress Now let's configure Qtap to capture incoming requests to our Python server. ### Create the Qtap Config Save this as `qtap-ingress.yaml`: ```yaml version: 2 # Storage Configuration - Output to console services: # Event metadata (connection info, timing) event_stores: - type: stdout # HTTP request/response data object_stores: - type: stdout # Processing Stack - Capture incoming HTTP traffic stacks: ingress_stack: plugins: # HTTP Capture plugin - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything including request/response bodies format: text # (json|text) - Human-readable format # Traffic Capture Settings - INGRESS configuration tap: direction: ingress # (egress|ingress|all) - Only capture INCOMING traffic ignore_loopback: true # (true|false) - True since we're capturing network traffic (not localhost) audit_include_dns: false # (true|false) - Skip DNS for cleaner output http: stack: ingress_stack # Use our ingress processing stack ``` **Key Configuration Points:** * **`direction: ingress`** - Only captures incoming HTTP requests from the network * **`ignore_loopback: true`** - We're capturing real network traffic, not localhost * **`level: full`** - Captures complete request/response including bodies *** ## Part 3: Running Qtap and Testing ### Step 1: Start Qtap (Before Making Requests!) In a **second terminal**, start Qtap with the ingress config: ```bash # Using Docker docker run -d \ --name qtap-ingress \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-ingress.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" # Wait for Qtap to initialize (CRITICAL - must wait before traffic!) sleep 6 ``` Or using the Linux binary: ```bash sudo qtap \ --config=qtap-ingress.yaml \ --log-level=info \ --log-encoding=console ``` ### Step 2: Find Your Host's Network IP Address Find the IP address of the machine running your Python server: ```bash # On Linux - find your primary network interface ip addr show | grep "inet " | grep -v 127.0.0.1 # Example output: # inet 192.168.1.100/24 brd 192.168.1.255 scope global eth0 ``` The IP address before the `/` is what you'll use (e.g., `192.168.1.100`). {% hint style="info" %} **Note:** You can test from the **same machine** using its network IP (not localhost), or from a **different machine** on the same network. Both demonstrate true ingress traffic. {% endhint %} ### Step 3: Allow Firewall Access (If Needed) If you have a firewall enabled, allow traffic on port 8000: ```bash # On Ubuntu/Debian with ufw sudo ufw allow 8000/tcp sudo ufw status # Or temporarily disable for testing # sudo ufw disable ``` ### Step 4: Generate Test Traffic from Another Host From **another machine on your network** (or the same machine using its network IP): ```bash # Replace 192.168.1.100 with your server's IP address # Test 1: Simple GET request (TRUE INGRESS!) curl http://192.168.1.100:8000/ # Test 2: GET with path parameter curl http://192.168.1.100:8000/api/users/42 # Test 3: GET with custom header curl http://192.168.1.100:8000/api/users/99 \ -H "X-Request-ID: test-12345" # Test 4: POST with JSON body curl -X POST http://192.168.1.100:8000/api/users \ -H "Content-Type: application/json" \ -d '{"name": "Alice Smith", "email": "alice@example.com", "role": "admin"}' # Test 5: Slow endpoint (tests latency tracking) curl http://192.168.1.100:8000/api/slow ``` {% hint style="success" %} **Why not use localhost?** Using the network IP (even from the same machine) creates **true ingress traffic** from the network perspective. The kernel treats this as actual incoming network traffic, not loopback. {% endhint %} **Alternative: Test from the same machine** If you don't have another machine available, you can still test from the same host using its network IP: ```bash # On the SAME machine where Python and Qtap are running # Use your network IP, NOT 127.0.0.1 curl http://192.168.1.100:8000/ # Replace with your actual IP # This creates network traffic (goes through the network stack) # Even though source and destination are the same physical machine ``` ### Step 5: View Captured Traffic Check the Qtap logs to see the captured ingress traffic: ```bash # If using Docker docker logs qtap-ingress # Look for output showing the Python process and incoming requests ``` **What you should see:** ``` === HTTP Transaction === Source Process: python (PID: 12345) Direction: INGRESS ← (incoming from network) Source IP: 192.168.1.50 (client machine or same host via network IP) Destination IP: 192.168.1.100:8000 (your Python server) Method: POST URL: http://192.168.1.100:8000/api/users Status: 200 OK Duration: 15ms --- Request Headers --- Host: 192.168.1.100:8000 User-Agent: curl/7.81.0 Accept: */* Content-Type: application/json Content-Length: 67 --- Request Body --- {"name": "Alice Smith", "email": "alice@example.com", "role": "admin"} --- Response Headers --- Content-Type: application/json Content-Length: 89 --- Response Body --- {"message":"User created successfully","data":{"name":"Alice Smith","email":"alice@example.com","role":"admin"},"id":12345} ======================== ``` **Key indicators that it's working:** * ✅ `"exe": "/usr/bin/python3"` - Python process identified * ✅ `Direction: INGRESS` - True incoming traffic from network * ✅ `Source IP: 192.168.1.50` - **NOT 127.0.0.1** - This is real network traffic! * ✅ `Destination IP: 192.168.1.100:8000` - Your server's network IP * ✅ `"protocol": "http1"` - HTTP protocol parsed correctly * ✅ Full request/response bodies visible * ✅ Custom headers captured (`X-Request-ID`) * ✅ Latency/duration tracked ## Part 4: Configuration Variations ### Variation 1: Capture Both Ingress and Egress To capture both incoming requests AND outgoing requests your Python app makes: ```yaml tap: direction: all # Capture everything ignore_loopback: false # Include localhost http: stack: ingress_stack ``` ### Variation 2: Headers Only (No Bodies) To reduce output size, capture only headers: ```yaml stacks: headers_only: plugins: - type: http_capture config: level: headers # (summary|headers|full) - Headers includes headers but not bodies format: text ``` ### Variation 3: Filter Specific Endpoints with Rules Capture only POST requests or specific paths using Rulekit: ```yaml stacks: filtered_stack: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Only capture POST requests - name: "POST requests only" expr: http.req.method == "POST" level: full # Capture slow requests (> 1 second) - name: "Slow endpoints" expr: http.res.duration_ms > 1000 level: full ``` ### Variation 4: Production Setup with S3 Storage For production, send sensitive request/response data to your own S3 bucket: ```yaml services: event_stores: - type: stdout # Metadata to console object_stores: - type: s3 # Bodies to S3 (never leaves your environment) endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-qtap-ingress access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false ``` See [Level 4 of the Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide#level-4-production-storage-with-s3-20-minutes) for full S3 setup. *** ## Understanding the Output ### What Qtap Captures for Ingress **Event Metadata (anonymized):** * Source IP and port (client) * Destination IP and port (your server) * Process information (`python`, PID, container) * Timing (request start, duration) * Status code **Object Data (sensitive):** * HTTP method, path, query parameters * Request headers (including custom headers) * Request body (JSON, form data, etc.) * Response headers * Response body ### Direction Field Explained * **`INGRESS` (incoming)**: Traffic coming INTO your application * Client → Your Python server * Shows what clients are requesting * **`EGRESS` (outgoing)**: Traffic going OUT from your application * Your Python server → External API * Shows what your server is calling *** ## Troubleshooting ### Not Seeing Any Traffic? **Check 1: Is your Python server listening on `0.0.0.0`?** ```python # In api_server.py - must bind to 0.0.0.0, not 127.0.0.1 uvicorn.run(app, host="0.0.0.0", port=8000) ``` **Check 2: Is Qtap running BEFORE you made requests?** ```bash # Qtap must be running first, then generate traffic docker logs qtap-ingress # Check if Qtap started successfully ``` **Check 3: Is your Python server actually running?** ```bash # Test with your network IP (NOT localhost!) curl http://192.168.1.100:8000/ # Replace with your actual IP # Should return JSON response ``` **Check 4: Can clients reach your server?** ```bash # Check firewall status sudo ufw status # If active, ensure port 8000 is allowed sudo ufw allow 8000/tcp # Test network connectivity from client ping 192.168.1.100 # Replace with your server's IP ``` **Check 5: Check Qtap is hooking into Python correctly** ```bash docker logs qtap-ingress 2>&1 | grep -i python # Should see logs about attaching to Python process ``` ### Seeing `"l7Protocol": "other"` instead of `"http1"`? This means Qtap captured the connection but couldn't parse HTTP. Possible causes: * Python server using HTTPS (TLS) - Qtap should still see it via SSL hooks * Traffic is not HTTP * Qtap not fully initialized when request was made (wait 6+ seconds) ### Too Much Noise from System Traffic? Add process filters to exclude specific executables: ```yaml tap: direction: ingress ignore_loopback: true filters: custom: - exe: /usr/sbin/sshd # Exclude SSH daemon strategy: exact - exe: /lib/systemd/ # Exclude systemd processes strategy: prefix ``` *** ## Real-World Example: API Gateway Monitoring Here's a practical example for monitoring an API gateway: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 # Store sensitive request/response data securely config: endpoint: https://minio.internal.company.com bucket: api-gateway-audit # credentials from environment stacks: # Stack for all ingress traffic api_monitoring: plugins: - type: http_capture config: level: none # Default: don't capture format: json rules: # Capture all authentication requests - name: "Auth endpoints" expr: http.req.path matches /^\/api\/auth\// level: full # Capture all errors - name: "Error responses" expr: http.res.status >= 400 level: full # Capture slow requests (for performance debugging) - name: "Slow requests" expr: http.res.duration_ms > 500 level: headers # Headers only, no bodies tap: direction: ingress ignore_loopback: true # True for network traffic (production) http: stack: api_monitoring ``` *** ## Next Steps **Learn More About Ingress Capture:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Complete `tap` section reference * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - All plugin options * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial covering all Qtap features **Production Deployment:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - S3 setup for production * [Kubernetes Manifest](/getting-started/qtap/installation/kubernetes-manifest) - Deploy Qtap in K8s * [Helm Chart](/getting-started/qtap/installation/helm-chart) - Helm deployment **Alternative: Cloud Management:** * [Qplane](/getting-started/qplane) - Manage Qtap with visual dashboards * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Quick start with cloud control plane *** ## Cleanup ```bash # Stop Qtap docker rm -f qtap-ingress # Stop Python server (CTRL+C in the terminal) # Remove config file rm qtap-ingress.yaml api_server.py ``` *** *This guide uses validated configurations. All examples are tested and guaranteed to work.* # Web Server Integration # NGINX Traffic Capture This guide shows you how to use Qtap to capture HTTP traffic flowing through **NGINX**, one of the most popular web servers and reverse proxies. You'll learn how to observe both incoming client requests and outgoing upstream connections, all without proxies or code changes. ## What You'll Learn * Capture NGINX ingress traffic (client requests coming in) * Capture NGINX egress traffic (upstream service requests going out) * Monitor both sides of a reverse proxy simultaneously * Apply conditional capture rules for specific endpoints * Set up NGINX + Qtap in Docker for testing * Deploy production-ready configurations with S3 storage ## Use Cases **Why capture NGINX traffic?** * **Reverse Proxy Visibility**: See both client requests and upstream responses in one place * **Performance Analysis**: Measure latency between client→nginx and nginx→upstream * **API Gateway Monitoring**: Track all API calls flowing through your gateway * **Security Auditing**: Detect malicious requests or data exfiltration attempts * **Troubleshooting**: Debug issues with request/response transformations * **Compliance**: Audit all traffic for regulatory requirements * **Load Balancer Analytics**: Understand traffic distribution patterns *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Docker installed (for this guide's examples) * Root/sudo access * Basic understanding of NGINX configuration *** ## Part 1: Simple NGINX Web Server Let's start with a basic NGINX setup serving static content and reverse proxying to an upstream service. ### Step 1: Create NGINX Configuration Create a directory for our demo: ```bash mkdir nginx-qtap-demo cd nginx-qtap-demo ``` Create `nginx.conf`: ```nginx events { worker_connections 1024; } http { # Enable access logs for debugging access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # Upstream service (backend API) upstream backend_api { server httpbin.org:80; } server { listen 8080; server_name localhost; # Static content endpoint location / { return 200 'Hello from NGINX!\n'; add_header Content-Type text/plain; } # Health check endpoint location /health { return 200 'OK\n'; add_header Content-Type text/plain; } # Reverse proxy to upstream API location /api/ { # Remove /api prefix before forwarding rewrite ^/api/(.*)$ /$1 break; proxy_pass http://httpbin.org; proxy_set_header Host httpbin.org; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # Another upstream for testing location /example/ { proxy_pass https://example.com/; proxy_set_header Host example.com; } } } ``` ### Step 2: Create Qtap Configuration Create `qtap.yaml`: ```yaml version: 2 # Storage Configuration services: # Connection metadata (anonymized) event_stores: - type: stdout # HTTP request/response data (sensitive) object_stores: - type: stdout # Processing Stack stacks: nginx_capture: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: text # (json|text) - Human-readable format # Traffic Capture Settings tap: direction: all # (egress|ingress|all) - Capture BOTH client requests AND upstream calls ignore_loopback: false # (true|false) - Capture localhost (nginx often uses loopback) audit_include_dns: false # (true|false) - Skip DNS queries for cleaner output http: stack: nginx_capture # Use our nginx processing stack # Optional: Filter out health check noise filters: groups: - qpoint # Don't capture qtap's own traffic ``` **Key Configuration Points:** * **`direction: all`** - Captures both incoming (client→nginx) AND outgoing (nginx→upstream) traffic * **`ignore_loopback: false`** - Important! NGINX often communicates via localhost * **`level: full`** - Captures complete requests/responses including bodies ### Step 3: Create Docker Compose Setup Create `docker-compose.yaml`: ```yaml version: '3.8' services: # NGINX web server nginx: image: nginx:alpine container_name: nginx-demo ports: - "8080:8080" volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro networks: - demo-network # Qtap agent qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-nginx privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml networks: demo-network: driver: bridge ``` *** ## Part 2: Running and Testing ### Step 1: Start the Services ```bash # Start NGINX and Qtap docker compose up -d # Wait for Qtap to initialize (CRITICAL - must happen before traffic!) sleep 6 # Verify NGINX is running curl http://localhost:8080/ # Expected: "Hello from NGINX!" ``` ### Step 2: Generate Test Traffic ```bash # Test 1: Simple GET to NGINX (INGRESS only - nginx returns static response) curl http://localhost:8080/ # Test 2: Health check (INGRESS only) curl http://localhost:8080/health # Test 3: Reverse proxy to httpbin.org (INGRESS + EGRESS) # You'll see TWO captures: client→nginx AND nginx→httpbin.org curl http://localhost:8080/api/get # Test 4: POST with JSON body through reverse proxy curl -X POST http://localhost:8080/api/post \ -H "Content-Type: application/json" \ -H "X-Custom-Header: test-value" \ -d '{"username": "alice", "action": "login"}' # Test 5: GET to example.com through nginx curl http://localhost:8080/example/ # Test 6: Generate multiple requests to see traffic patterns for i in {1..5}; do curl -s http://localhost:8080/api/uuid sleep 1 done ``` ### Step 3: View Captured Traffic ```bash # View Qtap logs docker logs qtap-nginx # Filter for nginx process docker logs qtap-nginx 2>&1 | grep -A 30 "nginx" # Count captured transactions docker logs qtap-nginx 2>&1 | grep -c "HTTP Transaction" ``` **What you should see:** ``` === HTTP Transaction === Source Process: nginx (PID: 123, Container: nginx-demo) Direction: INGRESS ← (client to nginx) Method: POST URL: http://localhost:8080/api/post Status: 200 OK Duration: 12ms --- Request Headers --- Host: localhost:8080 User-Agent: curl/7.81.0 Content-Type: application/json X-Custom-Header: test-value --- Request Body --- {"username": "alice", "action": "login"} --- Response Headers --- Content-Type: application/json Content-Length: 523 --- Response Body --- { "args": {}, "data": "{\"username\": \"alice\", \"action\": \"login\"}", "headers": { "Host": "httpbin.org", "X-Real-Ip": "172.18.0.1", "X-Forwarded-For": "172.18.0.1" }, "json": { "username": "alice", "action": "login" }, "url": "http://httpbin.org/post" } ======================== === HTTP Transaction === Source Process: nginx (PID: 123, Container: nginx-demo) Direction: EGRESS → (nginx to upstream) Method: POST URL: http://httpbin.org/post Status: 200 OK Duration: 245ms --- Request Headers --- Host: httpbin.org X-Real-IP: 172.18.0.1 X-Forwarded-For: 172.18.0.1 Content-Type: application/json --- Request Body --- {"username": "alice", "action": "login"} ======================== ``` **Key indicators that it's working:** * ✅ `"exe": "/usr/sbin/nginx"` - NGINX process identified * ✅ `Direction: INGRESS` - Client to NGINX * ✅ `Direction: EGRESS` - NGINX to upstream * ✅ **Two transactions** for proxied requests (one ingress, one egress) * ✅ Custom headers visible (`X-Custom-Header`, `X-Real-IP`) * ✅ Full request/response bodies captured * ✅ Latency tracked for both hops *** ## Part 3: Advanced Configurations ### Configuration 1: Capture Only Errors Reduce volume by capturing only failed requests (4xx/5xx status codes): ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout # Define reusable macros rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 stacks: error_only: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: - name: "Capture errors only" expr: is_error() level: full # But capture errors fully tap: direction: all ignore_loopback: false http: stack: error_only ``` Test it: ```bash # This should NOT be captured (200 OK) curl http://localhost:8080/ # This SHOULD be captured (404) curl http://localhost:8080/nonexistent # This SHOULD be captured (500 - if you create an error endpoint) curl http://localhost:8080/api/status/500 ``` ### Configuration 2: Separate Ingress and Egress Stacks Apply different capture levels to ingress vs. egress traffic: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout # Lightweight stack for ingress (headers only) stacks: ingress_light: plugins: - type: http_capture config: level: headers # (summary|headers|full) - Headers only, no bodies format: json # Full capture for egress (to debug upstream issues) egress_full: plugins: - type: http_capture config: level: full # Everything including bodies format: json tap: direction: all ignore_loopback: false # Default stack for ingress http: stack: ingress_light # Override for specific upstream domains endpoints: - domain: 'httpbin.org' http: stack: egress_full # Full capture for httpbin.org calls ``` ### Configuration 3: Filter by API Endpoint Capture only specific API paths using Rulekit: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_api_endpoint expr: http.req.path matches /^\/api\// - name: is_auth_endpoint expr: http.req.path matches /^\/api\/auth\// - name: is_sensitive expr: is_auth_endpoint() || http.req.method == "POST" stacks: selective_capture: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Capture all authentication requests with full details - name: "Auth endpoints" expr: is_auth_endpoint() level: full # Capture API errors - name: "API errors" expr: is_api_endpoint() && http.res.status >= 400 level: full # Capture POST requests (likely mutations) - name: "POST requests" expr: http.req.method == "POST" level: headers # Headers only for volume control tap: direction: all ignore_loopback: false http: stack: selective_capture ``` ### Configuration 4: Production Setup with S3 For production, store sensitive data in your own S3 bucket: ```yaml version: 2 services: # Metadata to stdout (for monitoring) event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-nginx-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false stacks: production_capture: plugins: - type: http_capture config: level: full format: json rules: # Only capture errors in production - name: "Production errors" expr: http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: production_capture # Exclude health checks from capture filters: groups: - qpoint ``` Update `docker-compose.yaml` to pass S3 credentials: ```yaml qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 environment: - TINI_SUBREAPER=1 - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} # ... rest of config ``` See [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) for complete S3 setup. *** ## Part 4: Real-World Use Cases ### Use Case 1: API Gateway Monitoring Monitor all API traffic flowing through NGINX as an API gateway: ```yaml version: 2 services: event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-nginx-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_error expr: http.res.status >= 400 - name: is_slow expr: http.res.duration_ms > 1000 - name: is_large_payload expr: http.req.headers.content-length > 1000000 # > 1MB stacks: api_gateway: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Capture all errors - name: "API errors" expr: is_error() level: full # Capture slow requests - name: "Slow requests" expr: is_slow() level: headers # Headers only # Capture large payloads (potential abuse) - name: "Large payloads" expr: is_large_payload() level: summary # Metadata only # Capture authentication attempts - name: "Auth attempts" expr: http.req.path matches /^\/api\/v1\/auth\// level: full tap: direction: all ignore_loopback: false http: stack: api_gateway # Exclude internal health checks filters: custom: - exe: /usr/bin/health-checker strategy: exact ``` ### Use Case 2: Debugging Reverse Proxy Issues Capture both sides of a reverse proxy to debug transformation issues: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug_proxy: plugins: - type: http_capture config: level: full # Capture everything for debugging format: text # Human-readable for quick analysis tap: direction: all # CRITICAL: Capture both ingress and egress ignore_loopback: false http: stack: debug_proxy # Only capture traffic to/from specific upstream endpoints: - domain: 'backend-api.internal.company.com' http: stack: debug_proxy ``` This configuration lets you compare: * What the client sent to NGINX (ingress) * What NGINX forwarded to the upstream (egress) * What the upstream returned (egress response) * What NGINX sent back to the client (ingress response) ### Use Case 3: Load Balancer Analytics Track traffic distribution across multiple upstreams: **nginx.conf** (simplified): ```nginx upstream backend_pool { server backend1:8080; server backend2:8080; server backend3:8080; } server { listen 80; location / { proxy_pass http://backend_pool; } } ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-nginx-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false stacks: load_balancer: plugins: - type: http_capture config: level: summary # Just metadata for analytics format: json tap: direction: egress # Only capture nginx→upstream (see distribution) ignore_loopback: false http: stack: load_balancer ``` Analyze the logs to see which backend server received each request. *** ## Understanding the Output ### Dual Capture for Reverse Proxy When NGINX proxies a request, Qtap captures **two separate HTTP transactions**: **Transaction 1: INGRESS (Client → NGINX)** ``` Source Process: nginx Direction: INGRESS ← Source IP: 192.168.1.100 (client) Destination: localhost:8080 Method: GET URL: http://localhost:8080/api/users/42 ``` **Transaction 2: EGRESS (NGINX → Upstream)** ``` Source Process: nginx Direction: EGRESS → Destination: httpbin.org:80 Method: GET URL: http://httpbin.org/users/42 ``` This dual capture lets you: * Measure end-to-end latency vs. upstream latency * See how NGINX transforms requests (headers, paths, bodies) * Debug issues on either side of the proxy ### Capture Levels Explained * **`none`**: No capture (use with rules for conditional capture) * **`summary`**: Basic metadata (method, URL, status, duration) - no headers/bodies * **`details`**: Includes headers - no bodies * **`full`**: Everything (headers + bodies) For high-traffic NGINX servers, start with `summary` or `details` to control volume. *** ## Troubleshooting ### Not Seeing NGINX Traffic? **Check 1: Is Qtap running before you made requests?** ```bash # Qtap must be running BEFORE traffic is generated docker logs qtap-nginx | head -20 # Should see startup messages ``` **Check 2: Is ignore\_loopback set correctly?** ```yaml # If NGINX uses localhost/127.0.0.1, you MUST set: tap: ignore_loopback: false ``` **Check 3: Is NGINX actually processing requests?** ```bash # Check NGINX access logs docker exec nginx-demo cat /var/log/nginx/access.log ``` **Check 4: Verify Qtap is hooking NGINX** ```bash docker logs qtap-nginx 2>&1 | grep -i nginx # Should see logs about attaching to nginx process ``` ### Seeing `"l7Protocol": "other"`? This means Qtap captured the connection but couldn't parse HTTP: * NGINX might be using HTTPS internally (check TLS configuration) * Traffic might not be HTTP * Qtap may not have fully initialized (wait 6+ seconds after starting) ### Too Much Traffic Captured? **Option 1: Use conditional rules** ```yaml stacks: reduced_volume: plugins: - type: http_capture config: level: none rules: - name: "Errors only" expr: http.res.status >= 400 level: full ``` **Option 2: Filter specific paths** ```yaml rules: - name: "Skip health checks" expr: http.req.path != "/health" level: full ``` **Option 3: Capture summary only** ```yaml config: level: summary # Metadata only, no headers/bodies ``` ### Duplicate Transactions? If you see the same request captured multiple times, this is expected for reverse proxies: * One INGRESS capture (client → nginx) * One EGRESS capture (nginx → upstream) To capture only one direction: ```yaml tap: direction: ingress # or egress ``` *** ## Performance Considerations ### NGINX + Qtap Performance Impact Qtap operates **out-of-band** using eBPF, with minimal impact: * **CPU overhead**: \~1-3% for typical HTTP traffic * **Memory**: \~50-200MB depending on traffic volume * **Latency**: No additional latency (passive observation) **Best practices for high-traffic NGINX:** 1. Use `level: summary` or `details` (avoid `full` with large bodies) 2. Apply conditional rules to reduce captured volume 3. Filter out health checks and monitoring endpoints 4. Send data to S3 in batches (use Fluent Bit for buffering) 5. Set TTL policies on storage (90 days recommended) ### Scaling Recommendations | **Traffic Volume** | **Recommended Level** | **Storage** | | ------------------ | --------------------- | -------------------------------------- | | < 100 req/sec | `full` | stdout or S3 | | 100-1000 req/sec | `details` | S3 with batching | | 1000-10000 req/sec | `summary` | S3 + Fluent Bit | | > 10000 req/sec | conditional rules | S3 + Fluent Bit + aggressive filtering | *** ## Next Steps **Learn More About Qtap:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Complete `tap` configuration * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - All plugin options * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial covering all features **Production Deployment:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - S3 setup guide * [Capturing All HTTP Traffic with Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) - Batching and buffering * [Kubernetes Manifest](/getting-started/qtap/installation/kubernetes-manifest) - Deploy in K8s **Related Guides:** * [Ingress Traffic Capture with Python](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) - Similar concepts for other servers * [HTTPS Header Capture Without Proxies](/guides/qtap-guides/advanced-use-cases/transparent-https-header-capture-without-proxies) - TLS inspection details **Alternative: Cloud Management:** * [Qplane](/getting-started/qplane) - Manage Qtap with visual dashboards * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Quick start with cloud control plane *** ## Cleanup ```bash # Stop all services docker compose down # Remove containers and volumes docker compose down -v # Clean up files rm nginx.conf qtap.yaml docker-compose.yaml ``` *** *This guide uses validated configurations. All examples are tested and guaranteed to work with NGINX and Qtap.* # Caddy Traffic Capture This guide shows you how to use Qtap to capture HTTP traffic flowing through **Caddy**, a modern web server with automatic HTTPS. You'll learn how to observe both incoming client requests and outgoing upstream connections from your Caddy server, all without proxies or code changes. ## What You'll Learn * Capture Caddy ingress traffic (client requests) * Capture Caddy egress traffic (upstream service requests) * Monitor both sides of a reverse proxy simultaneously * Apply conditional capture rules for specific routes * Handle Caddy's automatic HTTPS with Qtap's TLS inspection * Set up Caddy + Qtap in Docker for testing * Deploy production-ready configurations ## Use Cases **Why capture Caddy traffic?** * **Reverse Proxy Visibility**: See both client requests and backend responses * **API Gateway Monitoring**: Track all API calls through your Caddy gateway * **Automatic HTTPS Inspection**: See inside TLS traffic without managing certificates * **Microservices Debugging**: Debug issues between services * **Performance Analysis**: Measure latency at each hop * **Security Auditing**: Monitor for suspicious traffic patterns * **Migration Planning**: Understand traffic patterns before infrastructure changes *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Docker installed (for this guide's examples) * Root/sudo access * Basic understanding of Caddy/Caddyfile syntax *** ## Part 1: Simple Caddy Web Server Let's start with a basic Caddy setup serving static content and reverse proxying to upstream services. ### Step 1: Create Caddy Configuration Create a directory for our demo: ```bash mkdir caddy-qtap-demo cd caddy-qtap-demo ``` Create `Caddyfile`: ``` { # Global options auto_https off # Disable for local testing (use HTTP) admin off # Disable admin API for simplicity } :8080 { # Static response endpoint respond / "Hello from Caddy!" 200 # Health check endpoint respond /health "OK" 200 # JSON API endpoint handle /api/status { header Content-Type application/json respond `{"status": "healthy", "server": "caddy"}` 200 } # Reverse proxy to httpbin.org handle_path /api/httpbin/* { reverse_proxy http://httpbin.org { header_up Host httpbin.org header_up X-Forwarded-Server {host} } } # Reverse proxy to example.com handle_path /example/* { reverse_proxy https://example.com { header_up Host example.com } } # File server for static content file_server /static/* { root /var/www } # Enable access logging log { output stdout format console } } ``` ### Step 2: Create Qtap Configuration Create `qtap.yaml`: ```yaml version: 2 # Storage Configuration services: # Connection metadata (anonymized) event_stores: - type: stdout # HTTP request/response data (sensitive) object_stores: - type: stdout # Processing Stack stacks: caddy_capture: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: text # (json|text) - Human-readable format # Traffic Capture Settings tap: direction: all # (egress|ingress|all) - Capture BOTH directions ignore_loopback: false # (true|false) - Capture localhost (caddy uses loopback) audit_include_dns: false # (true|false) - Skip DNS for cleaner output http: stack: caddy_capture # Use our caddy processing stack # Optional: Filter out noise filters: groups: - qpoint # Don't capture qtap's own traffic ``` **Key Configuration Points:** * **`direction: all`** - Captures both client→caddy AND caddy→upstream traffic * **`ignore_loopback: false`** - Important! Caddy often uses localhost * **`level: full`** - Captures complete requests/responses including bodies ### Step 3: Create Docker Compose Setup Create `docker-compose.yaml`: ```yaml version: '3.8' services: # Caddy web server caddy: image: caddy:latest container_name: caddy-demo ports: - "8082:8080" volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro networks: - demo-network # Qtap agent qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-caddy privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml networks: demo-network: driver: bridge ``` *** ## Part 2: Running and Testing ### Step 1: Start the Services ```bash # Start Caddy and Qtap docker compose up -d # Wait for Qtap to initialize (CRITICAL - must happen before traffic!) sleep 6 # Verify Caddy is running curl http://localhost:8082/ # Expected: "Hello from Caddy!" ``` ### Step 2: Generate Test Traffic ```bash # Test 1: Simple GET to Caddy (INGRESS only - static response) curl http://localhost:8082/ # Test 2: Health check curl http://localhost:8082/health # Test 3: JSON API endpoint curl http://localhost:8082/api/status # Test 4: Reverse proxy to httpbin.org (INGRESS + EGRESS) # You'll see TWO captures: client→caddy AND caddy→httpbin curl http://localhost:8082/api/httpbin/get # Test 5: POST with JSON through reverse proxy curl -X POST http://localhost:8082/api/httpbin/post \ -H "Content-Type: application/json" \ -H "X-Request-ID: test-12345" \ -d '{"username": "alice", "role": "admin"}' # Test 6: Reverse proxy to example.com (HTTPS upstream) curl http://localhost:8082/example/ # Test 7: Generate multiple requests to see patterns for i in {1..5}; do curl -s http://localhost:8082/api/status sleep 1 done ``` ### Step 3: View Captured Traffic ```bash # View Qtap logs docker logs qtap-caddy # Filter for caddy process docker logs qtap-caddy 2>&1 | grep -A 30 "caddy" # Count captured transactions docker logs qtap-caddy 2>&1 | grep -c "HTTP Transaction" ``` **What you should see:** ``` === HTTP Transaction === Source Process: caddy (PID: 456, Container: caddy-demo) Direction: INGRESS ← (client to caddy) Method: POST URL: http://localhost:8082/api/httpbin/post Status: 200 OK Duration: 15ms --- Request Headers --- Host: localhost:8082 User-Agent: curl/7.81.0 Content-Type: application/json X-Request-ID: test-12345 --- Request Body --- {"username": "alice", "role": "admin"} --- Response Headers --- Content-Type: application/json Content-Length: 512 --- Response Body --- { "args": {}, "data": "{\"username\": \"alice\", \"role\": \"admin\"}", "headers": { "Host": "httpbin.org", "X-Forwarded-Server": "localhost:8082" }, "json": { "username": "alice", "role": "admin" }, "url": "http://httpbin.org/post" } ======================== === HTTP Transaction === Source Process: caddy (PID: 456, Container: caddy-demo) Direction: EGRESS → (caddy to upstream) Method: POST URL: http://httpbin.org/post Status: 200 OK Duration: 320ms --- Request Headers --- Host: httpbin.org X-Forwarded-Server: localhost:8082 Content-Type: application/json --- Request Body --- {"username": "alice", "role": "admin"} ======================== ``` **Key indicators that it's working:** * ✅ `"exe": "/usr/bin/caddy"` or similar - Caddy process identified * ✅ `Direction: INGRESS` - Client to Caddy * ✅ `Direction: EGRESS` - Caddy to upstream * ✅ **Two transactions** for proxied requests (ingress + egress) * ✅ Custom headers visible (`X-Request-ID`, `X-Forwarded-Server`) * ✅ Full request/response bodies captured * ✅ Latency tracked for both hops *** ## Part 3: Advanced Configurations ### Configuration 1: Capture Only Errors Reduce volume by capturing only failed requests: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout # Define reusable macros rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 - name: is_server_error expr: http.res.status >= 500 && http.res.status < 600 stacks: error_only: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Capture all errors - name: "HTTP errors" expr: is_error() level: full # Capture server errors with extra detail - name: "Server errors" expr: is_server_error() level: full tap: direction: all ignore_loopback: false http: stack: error_only ``` Test it: ```bash # This should NOT be captured (200 OK) curl http://localhost:8082/ # This SHOULD be captured (404) curl http://localhost:8082/nonexistent # This SHOULD be captured (if you add an error endpoint) curl http://localhost:8082/api/httpbin/status/500 ``` ### Configuration 2: Route-Specific Capture Capture different levels for different Caddy routes using Rulekit: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_api_route expr: http.req.path matches /^\/api\// - name: is_health_check expr: http.req.path == "/health" - name: is_static expr: http.req.path matches /^\/static\// - name: is_proxy_route expr: http.req.path matches /^\/api\/httpbin\// stacks: selective_capture: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Skip health checks entirely - name: "Skip health" expr: is_health_check() level: none # Capture API routes with full details - name: "API routes" expr: is_api_route() && !is_health_check() level: full # Capture static content metadata only - name: "Static content" expr: is_static() level: summary # Capture proxy errors in detail - name: "Proxy errors" expr: is_proxy_route() && http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: selective_capture ``` ### Configuration 3: HTTPS Upstream Monitoring When Caddy proxies to HTTPS upstreams, Qtap can still see the traffic: **Caddyfile**: ``` :8080 { # Proxy to HTTPS backend (Qtap will see decrypted traffic) reverse_proxy /secure/* { to https://api.github.com header_up Host api.github.com header_up User-Agent "Caddy-Proxy/1.0" } } ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: https_capture: plugins: - type: http_capture config: level: full format: text tap: direction: egress # Focus on caddy→upstream HTTPS calls ignore_loopback: false http: stack: https_capture # Only capture traffic to specific HTTPS endpoints endpoints: - domain: 'api.github.com' http: stack: https_capture ``` **Why this works**: Qtap hooks into Caddy's TLS library (typically Go's crypto/tls) before encryption happens, so it sees plaintext even for HTTPS upstreams. ### Configuration 4: Production Setup with S3 For production, store sensitive data securely: ```yaml version: 2 services: # Metadata to stdout (for monitoring) event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-caddy-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_error expr: http.res.status >= 400 stacks: production_capture: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Only capture errors in production - name: "Production errors" expr: is_error() level: full # Capture slow requests (> 2 seconds) - name: "Slow requests" expr: http.res.duration_ms > 2000 level: headers # Headers only, no bodies tap: direction: all ignore_loopback: false http: stack: production_capture ``` Update `docker-compose.yaml`: ```yaml qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 environment: - TINI_SUBREAPER=1 - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} # ... rest of config ``` *** ## Part 4: Real-World Use Cases ### Use Case 1: API Gateway with Authentication Monitor API gateway with focus on authentication and errors: **Caddyfile**: ``` :8080 { # Authentication endpoint reverse_proxy /api/auth/* { to http://auth-service:3000 } # Protected API endpoints reverse_proxy /api/v1/* { to http://backend-api:8000 } # Public endpoints respond /api/public/* `{"message": "public endpoint"}` 200 } ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-caddy-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_auth_endpoint expr: http.req.path matches /^\/api\/auth\// - name: is_error expr: http.res.status >= 400 - name: has_auth_header expr: http.req.headers.authorization != "" - name: is_unauthorized expr: http.res.status == 401 || http.res.status == 403 stacks: api_gateway: plugins: - type: http_capture config: level: none format: json rules: # Capture all authentication attempts - name: "Auth attempts" expr: is_auth_endpoint() level: full # Capture unauthorized requests - name: "Unauthorized access" expr: is_unauthorized() level: full # Capture API errors - name: "API errors" expr: is_error() && !is_auth_endpoint() level: headers # Headers only # Capture requests without auth header (potential security issue) - name: "Missing auth" expr: http.req.path matches /^\/api\/v1\// && !has_auth_header() level: summary tap: direction: all ignore_loopback: false http: stack: api_gateway ``` ### Use Case 2: Microservices Mesh Monitoring Monitor Caddy as a service mesh proxy: **Caddyfile**: ``` :8080 { # Service A reverse_proxy /service-a/* { to http://service-a:9000 header_up X-Mesh-Proxy Caddy } # Service B reverse_proxy /service-b/* { to http://service-b:9001 header_up X-Mesh-Proxy Caddy } # Service C (external) reverse_proxy /service-c/* { to https://external-api.example.com } } ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-caddy-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false stacks: mesh_monitoring: plugins: - type: http_capture config: level: summary # Just metadata for service mesh analytics format: json # Detailed stack for external services detailed_external: plugins: - type: http_capture config: level: full # Full capture for external service calls format: json tap: direction: all # Capture both ingress and egress ignore_loopback: false http: stack: mesh_monitoring # Apply different stacks to different services endpoints: - domain: 'external-api.example.com' http: stack: detailed_external # More detail for external calls ``` ### Use Case 3: Static Site with CDN Backend Monitor Caddy serving static sites with CDN backend: **Caddyfile**: ``` :8080 { # Static file server file_server / { root /var/www/html } # Proxy to CDN for media reverse_proxy /media/* { to https://cdn.example.com header_up Host cdn.example.com } } ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_media_request expr: http.req.path matches /^\/media\// - name: is_large_file expr: http.res.headers.content-length > 1000000 # > 1MB stacks: static_site: plugins: - type: http_capture config: level: none format: json rules: # Capture CDN errors - name: "CDN errors" expr: is_media_request() && http.res.status >= 400 level: full # Capture large file transfers (metadata only) - name: "Large files" expr: is_large_file() level: summary # Skip successful static content - name: "Skip successful static" expr: '!is_media_request() && http.res.status < 400' level: none tap: direction: all ignore_loopback: false http: stack: static_site ``` *** ## Understanding the Output ### Dual Capture for Reverse Proxy When Caddy proxies a request, Qtap captures **two HTTP transactions**: **Transaction 1: INGRESS (Client → Caddy)** ``` Source Process: caddy Direction: INGRESS ← Method: GET URL: http://localhost:8082/api/httpbin/users ``` **Transaction 2: EGRESS (Caddy → Upstream)** ``` Source Process: caddy Direction: EGRESS → Method: GET URL: http://httpbin.org/users ``` This lets you: * Measure total latency vs. backend latency * See how Caddy transforms requests (headers, paths) * Debug issues on either side of the proxy ### Caddy-Specific Details **Process Identification**: * Look for `exe` containing `caddy` (often `/usr/bin/caddy` or `/usr/local/bin/caddy`) * Container name: `caddy-demo` (in Docker) **Automatic HTTPS**: * When Caddy uses automatic HTTPS, Qtap still sees plaintext via eBPF TLS hooks * No certificate management needed * Works with Let's Encrypt, ZeroSSL, or custom CAs *** ## Troubleshooting ### Not Seeing Caddy Traffic? **Check 1: Is Qtap running before requests?** ```bash docker logs qtap-caddy | head -20 # Should see startup messages ``` **Check 2: Is ignore\_loopback correct?** ```yaml # If Caddy uses localhost, set: tap: ignore_loopback: false ``` **Check 3: Is Caddy processing requests?** ```bash # Check Caddy logs docker logs caddy-demo # Test Caddy directly curl http://localhost:8082/ ``` **Check 4: Verify Qtap hooks Caddy** ```bash docker logs qtap-caddy 2>&1 | grep -i caddy # Should see logs about attaching to caddy process ``` ### Seeing `"l7Protocol": "other"`? This means connection captured but HTTP not parsed: * Wait longer after starting Qtap (6+ seconds) * Check if Caddy is using HTTP/3 (QUIC) - not yet supported * Verify traffic is actually HTTP/HTTPS ### Caddy Using HTTP/3? Qtap currently supports HTTP/1.x and HTTP/2. If Caddy negotiates HTTP/3 (QUIC): **Disable HTTP/3 in Caddyfile**: ``` { servers { protocols h1 h2 # Only HTTP/1 and HTTP/2 } } ``` ### Too Much Traffic? **Option 1: Conditional capture** ```yaml stacks: reduced: plugins: - type: http_capture config: level: none rules: - name: "Errors only" expr: http.res.status >= 400 level: full ``` **Option 2: Filter specific routes** ```yaml rules: - name: "Skip health" expr: http.req.path != "/health" level: full ``` **Option 3: Summary level only** ```yaml config: level: summary # Metadata only ``` *** ## Performance Considerations ### Caddy + Qtap Performance Qtap operates **out-of-band** with minimal overhead: * **CPU**: \~1-3% for typical traffic * **Memory**: \~50-200MB depending on volume * **Latency**: Zero additional latency (passive observation) **Best practices for high-traffic Caddy:** 1. Use `level: summary` or `details` for high volume 2. Apply conditional rules to capture selectively 3. Filter health checks and monitoring endpoints 4. Send to S3 with batching (use Fluent Bit) 5. Set TTL policies on storage (90 days recommended) ### Scaling Recommendations | **Traffic Volume** | **Recommended Level** | **Storage** | | ------------------ | --------------------- | -------------------------------------- | | < 100 req/sec | `full` | stdout or S3 | | 100-1000 req/sec | `details` | S3 with batching | | 1000-10000 req/sec | `summary` | S3 + Fluent Bit | | > 10000 req/sec | conditional rules | S3 + Fluent Bit + aggressive filtering | *** ## Caddy vs NGINX: Key Differences **Process Name**: * Caddy: `/usr/bin/caddy` * NGINX: `/usr/sbin/nginx` **Configuration**: * Caddy: Caddyfile (simpler, more human-readable) * NGINX: nginx.conf (more complex, more options) **HTTPS**: * Caddy: Automatic by default (Qtap still works!) * NGINX: Manual configuration **Language**: * Caddy: Written in Go (uses Go's crypto/tls) * NGINX: Written in C (uses OpenSSL) Both work perfectly with Qtap's eBPF-based capture. *** ## Next Steps **Learn More About Qtap:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Complete `tap` configuration * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - All plugin options * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial **Production Deployment:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - S3 setup guide * [Capturing All HTTP Traffic with Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) - Batching for scale * [Kubernetes Manifest](/getting-started/qtap/installation/kubernetes-manifest) - Deploy in K8s **Related Guides:** * [Capturing NGINX Traffic](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) - Similar guide for NGINX * [Ingress Traffic Capture with Python](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) - Application server capture * [HTTPS Header Capture Without Proxies](/guides/qtap-guides/advanced-use-cases/transparent-https-header-capture-without-proxies) - TLS inspection details **Alternative: Cloud Management:** * [Qplane](/getting-started/qplane) - Manage Qtap with visual dashboards * [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) - Quick start *** ## Cleanup ```bash # Stop all services docker compose down # Remove containers and volumes docker compose down -v # Clean up files rm Caddyfile qtap.yaml docker-compose.yaml ``` *** *This guide uses validated configurations. All examples are tested and guaranteed to work with Caddy and Qtap.* # Traefik Traffic Capture This guide shows you how to use Qtap to capture HTTP traffic flowing through **Traefik**, a modern cloud-native reverse proxy and load balancer. You'll learn how to observe both incoming client requests and outgoing upstream connections in a dynamic, label-based configuration environment. ## What You'll Learn * Capture Traefik ingress traffic (client requests) * Capture Traefik egress traffic (backend service requests) * Monitor both sides of a reverse proxy simultaneously * Use Traefik's label-based configuration with Qtap * Leverage Traefik's automatic service discovery * Handle dynamic backend routing * Set up Traefik + Qtap in Docker for testing * Deploy production-ready configurations ## Use Cases **Why capture Traefik traffic?** * **Dynamic Service Discovery**: Monitor auto-discovered services in Docker/Kubernetes * **API Gateway Monitoring**: Track all API calls through your edge proxy * **Container Traffic Visibility**: See communication between microservices * **Load Balancer Analytics**: Understand traffic distribution across backends * **Automatic HTTPS Inspection**: See inside TLS traffic without certificate management * **Debugging Service Routing**: Verify Traefik routes traffic correctly * **Performance Analysis**: Measure latency at each routing hop *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Docker installed (for this guide's examples) * Root/sudo access * Basic understanding of Traefik and Docker labels *** ## Part 1: Traefik with Multiple Backends Traefik is unique because it configures routes via **Docker labels** instead of config files. Let's set up Traefik with multiple backend services. ### Step 1: Create Project Directory ```bash mkdir traefik-qtap-demo cd traefik-qtap-demo ``` ### Step 2: Create Traefik Configuration Create `traefik.yaml`: ```yaml # Traefik static configuration api: dashboard: true insecure: true # For testing only - dashboard on :8080 entryPoints: web: address: ":80" providers: docker: endpoint: "unix:///var/run/docker.sock" exposedByDefault: false # Only expose services with traefik.enable=true log: level: INFO format: common ``` ### Step 3: Create Backend Services We'll create two simple backend services to demonstrate routing. Create `backend-service.py`: ```python #!/usr/bin/env python3 from http.server import HTTPServer, BaseHTTPRequestHandler import json import sys import os class Handler(BaseHTTPRequestHandler): def do_GET(self): service_name = os.getenv('SERVICE_NAME', 'unknown') response = { "service": service_name, "path": self.path, "message": f"Hello from {service_name}!" } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def do_POST(self): content_length = int(self.headers.get('Content-Length', 0)) body = self.rfile.read(content_length).decode() if content_length > 0 else "" service_name = os.getenv('SERVICE_NAME', 'unknown') response = { "service": service_name, "method": "POST", "received": body, "message": f"POST received by {service_name}" } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def log_message(self, format, *args): # Suppress default logging pass if __name__ == '__main__': port = int(sys.argv[1]) if len(sys.argv) > 1 else 8000 server = HTTPServer(('0.0.0.0', port), Handler) print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}") server.serve_forever() ``` ### Step 4: Create Qtap Configuration Create `qtap.yaml`: ```yaml version: 2 # Storage Configuration services: # Connection metadata (anonymized) event_stores: - type: stdout # HTTP request/response data (sensitive) object_stores: - type: stdout # Processing Stack stacks: traefik_capture: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: text # (json|text) - Human-readable format # Traffic Capture Settings tap: direction: all # (egress|ingress|all) - Capture BOTH directions ignore_loopback: false # (true|false) - Capture localhost (traefik uses loopback) audit_include_dns: false # (true|false) - Skip DNS for cleaner output http: stack: traefik_capture # Use our traefik processing stack # Optional: Filter out noise filters: groups: - qpoint # Don't capture qtap's own traffic ``` ### Step 5: Create Docker Compose Setup Create `docker-compose.yaml`: ```yaml version: '3.8' services: # Traefik reverse proxy traefik: image: traefik:v3.0 container_name: traefik-demo ports: - "8083:80" # HTTP entrypoint - "8084:8080" # Dashboard volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - ./traefik.yaml:/etc/traefik/traefik.yaml:ro networks: - demo-network # Backend Service A service-a: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8000"] container_name: service-a environment: - SERVICE_NAME=service-a labels: - "traefik.enable=true" - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)" - "traefik.http.routers.service-a.entrypoints=web" - "traefik.http.services.service-a.loadbalancer.server.port=8000" - "traefik.http.middlewares.service-a-stripprefix.stripprefix.prefixes=/api/service-a" - "traefik.http.routers.service-a.middlewares=service-a-stripprefix" networks: - demo-network # Backend Service B service-b: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8000"] container_name: service-b environment: - SERVICE_NAME=service-b labels: - "traefik.enable=true" - "traefik.http.routers.service-b.rule=PathPrefix(`/api/service-b`)" - "traefik.http.routers.service-b.entrypoints=web" - "traefik.http.services.service-b.loadbalancer.server.port=8000" - "traefik.http.middlewares.service-b-stripprefix.stripprefix.prefixes=/api/service-b" - "traefik.http.routers.service-b.middlewares=service-b-stripprefix" networks: - demo-network # HTTPBin mock service (internal upstream) httpbin: image: kennethreitz/httpbin container_name: httpbin expose: - "80" labels: - "traefik.enable=true" - "traefik.http.routers.httpbin.rule=PathPrefix(`/api/httpbin`)" - "traefik.http.routers.httpbin.entrypoints=web" - "traefik.http.services.httpbin.loadbalancer.server.port=80" - "traefik.http.middlewares.httpbin-stripprefix.stripprefix.prefixes=/api/httpbin" - "traefik.http.routers.httpbin.middlewares=httpbin-stripprefix" networks: - demo-network # httpbin is bundled locally so you can exercise the full flow without relying on external connectivity. Traefik forwards `/api/httpbin/*` # to the container on port 80, and you still see the canonical httpbin responses in your curls and Qtap captures. # Qtap agent qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-traefik privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml networks: demo-network: driver: bridge ``` **Key Traefik Concepts:** * **Labels**: Configure routing via Docker labels (not config files) * **Routers**: Define how to match incoming requests (`PathPrefix`, `Host`, etc.) * **Services**: Define backend servers (load balancer targets) * **Middlewares**: Transform requests (strip prefixes, add headers, etc.) * **Automatic Discovery**: Traefik watches Docker for new containers *** ## Part 2: Running and Testing ### Step 1: Start the Services ```bash # Start all services docker compose up -d # Wait for Qtap to initialize (CRITICAL!) sleep 6 # Check Traefik dashboard (optional) # Open http://localhost:8084 in browser ``` ### Step 2: Generate Test Traffic ```bash # Test 1: Route to Service A (INGRESS + EGRESS) curl http://localhost:8083/api/service-a/ # Test 2: Route to Service B curl http://localhost:8083/api/service-b/ # Test 3: POST to Service A curl -X POST http://localhost:8083/api/service-a/users \ -H "Content-Type: application/json" \ -d '{"name": "Alice", "role": "admin"}' # Test 4: Route to the internal httpbin service curl http://localhost:8083/api/httpbin/get # Test 5: POST to httpbin via Traefik curl -X POST http://localhost:8083/api/httpbin/post \ -H "Content-Type: application/json" \ -d '{"test": "data"}' # Test 6: Generate load to see routing distribution for i in {1..10}; do curl -s http://localhost:8083/api/service-a/ curl -s http://localhost:8083/api/service-b/ done ``` ### Step 3: View Captured Traffic ```bash # View Qtap logs docker logs qtap-traefik # Filter for traefik process docker logs qtap-traefik 2>&1 | grep -A 30 "traefik" # Count transactions docker logs qtap-traefik 2>&1 | grep -c "HTTP Transaction" ``` **What you should see:** ``` === HTTP Transaction === Source Process: traefik (PID: 789, Container: traefik-demo) Direction: INGRESS ← (client to traefik) Method: POST URL: http://localhost:8083/api/service-a/users Status: 200 OK Duration: 8ms --- Request Headers --- Host: localhost:8083 User-Agent: curl/7.81.0 Content-Type: application/json --- Request Body --- {"name": "Alice", "role": "admin"} --- Response Headers --- Content-Type: application/json --- Response Body --- {"service":"service-a","method":"POST","received":"{\"name\": \"Alice\", \"role\": \"admin\"}","message":"POST received by service-a"} ======================== === HTTP Transaction === Source Process: traefik (PID: 789, Container: traefik-demo) Direction: EGRESS → (traefik to backend) Method: POST URL: http://service-a:8000/users Status: 200 OK Duration: 5ms --- Request Headers --- X-Forwarded-For: 172.18.0.1 X-Forwarded-Proto: http --- Request Body --- {"name": "Alice", "role": "admin"} ======================== ``` **Key indicators:** * ✅ `"exe"` contains `traefik` - Process identified * ✅ `Direction: INGRESS` - Client → Traefik * ✅ `Direction: EGRESS` - Traefik → Backend service * ✅ **Two transactions** per proxied request * ✅ Path transformation visible (prefix stripped) * ✅ Headers added by Traefik (`X-Forwarded-*`) *** ## Part 3: Advanced Configurations ### Configuration 1: Capture Only Specific Services Use Rulekit to capture only traffic to specific backend services: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_service_a expr: http.req.path matches /^\/api\/service-a\// - name: is_external expr: http.req.path matches /^\/api\/httpbin\// - name: is_error expr: http.res.status >= 400 stacks: selective_capture: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: # Capture all traffic to service-a - name: "Service A traffic" expr: is_service_a() level: full # Capture only errors from external services - name: "External errors" expr: is_external() && is_error() level: full # Capture slow requests anywhere - name: "Slow requests" expr: http.res.duration_ms > 1000 level: headers # Headers only tap: direction: all ignore_loopback: false http: stack: selective_capture ``` ### Configuration 2: Monitor Service Discovery Capture traffic as Traefik discovers and routes to new services: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: has_service_header expr: http.req.headers.x-service-name != "" stacks: discovery_monitoring: plugins: - type: http_capture config: level: summary # Just metadata for service analytics format: json tap: direction: egress # Focus on traefik→backend traffic ignore_loopback: false http: stack: discovery_monitoring ``` This captures metadata about which backends Traefik routes to, useful for understanding service discovery behavior. ### Configuration 3: API Gateway with Rate Limiting Detection Monitor API gateway patterns and detect potential rate limiting: ```yaml version: 2 services: event_stores: - type: stdout # Sensitive data to S3 (never leaves your infrastructure) object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-Traefik-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_rate_limited expr: http.res.status == 429 - name: is_auth_failure expr: http.res.status == 401 || http.res.status == 403 - name: is_api_path expr: http.req.path matches /^\/api\// stacks: api_gateway: plugins: - type: http_capture config: level: none format: json rules: # Capture rate limiting events - name: "Rate limited" expr: is_rate_limited() level: full # Capture authentication failures - name: "Auth failures" expr: is_auth_failure() level: full # Capture all API errors - name: "API errors" expr: is_api_path() && http.res.status >= 400 level: headers tap: direction: all ignore_loopback: false http: stack: api_gateway ``` ### Configuration 4: Production Setup with S3 ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-traefik-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false stacks: production_capture: plugins: - type: http_capture config: level: none format: json rules: # Only capture errors - name: "Production errors" expr: http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: production_capture ``` *** ## Part 4: Real-World Use Cases ### Use Case 1: Microservices Mesh Monitoring Monitor all service-to-service communication through Traefik: **docker-compose.yaml** (add more services): ```yaml service-c: build: ... labels: - "traefik.enable=true" - "traefik.http.routers.service-c.rule=PathPrefix(`/api/service-c`)" # ... more labels service-d: build: ... labels: - "traefik.enable=true" - "traefik.http.routers.service-d.rule=PathPrefix(`/api/service-d`)" # ... more labels ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-Traefik-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_internal_service expr: http.req.path matches /^\/api\/service-[a-z]\// stacks: mesh_monitoring: plugins: - type: http_capture config: level: summary # Metadata for analytics format: json rules: # Capture all internal service calls - name: "Service mesh traffic" expr: is_internal_service() level: summary # But capture errors in full - name: "Service mesh errors" expr: is_internal_service() && http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: mesh_monitoring ``` ### Use Case 2: Canary Deployment Monitoring Monitor traffic distribution during canary deployments: **docker-compose.yaml**: ```yaml service-a-v1: # ... existing service-a config labels: - "traefik.enable=true" - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)" - "traefik.http.services.service-a.loadbalancer.server.port=8000" - "traefik.http.services.service-a.loadbalancer.sticky.cookie=true" service-a-v2: # ... same as v1 but with SERVICE_NAME=service-a-v2 environment: - SERVICE_NAME=service-a-v2 labels: - "traefik.enable=true" - "traefik.http.routers.service-a.rule=PathPrefix(`/api/service-a`)" - "traefik.http.services.service-a.loadbalancer.server.port=8000" ``` Traefik will load balance between v1 and v2. Qtap captures which version served each request. **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: canary_monitoring: plugins: - type: http_capture config: level: summary # Capture metadata to see distribution format: json tap: direction: egress # Focus on traefik→backend to see which version ignore_loopback: false http: stack: canary_monitoring ``` Analyze logs to see v1 vs v2 traffic distribution. ### Use Case 3: Multi-Tenant API Gateway Route different tenants to different backends: **docker-compose.yaml**: ```yaml tenant-a-backend: labels: - "traefik.enable=true" - "traefik.http.routers.tenant-a.rule=Host(`tenant-a.example.com`)" tenant-b-backend: labels: - "traefik.enable=true" - "traefik.http.routers.tenant-b.rule=Host(`tenant-b.example.com`)" ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: tenant_a expr: http.req.headers.host matches /^tenant-a\./ - name: tenant_b expr: http.req.headers.host matches /^tenant-b\./ stacks: multi_tenant: plugins: - type: http_capture config: level: none format: json rules: # Capture all tenant A traffic for audit - name: "Tenant A traffic" expr: tenant_a() level: full # Capture only errors for tenant B - name: "Tenant B errors" expr: tenant_b() && http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: multi_tenant ``` *** ## Understanding Traefik + Qtap ### Dual Capture for Dynamic Routing When Traefik routes a request, Qtap captures **two transactions**: **Transaction 1: INGRESS (Client → Traefik)** ``` Source Process: traefik Direction: INGRESS ← URL: http://localhost:8083/api/service-a/users ``` **Transaction 2: EGRESS (Traefik → Backend)** ``` Source Process: traefik Direction: EGRESS → URL: http://service-a:8000/users ``` Notice: * Path transformation: `/api/service-a/users` → `/users` (middleware stripped prefix) * Container resolution: `service-a:8000` (Docker DNS) * Headers added: `X-Forwarded-*` ### Traefik-Specific Features **Process Identification**: * Look for `exe` containing `traefik` * Typically `/usr/local/bin/traefik` **Label-Based Configuration**: * Unlike NGINX/Caddy, routing is defined via container labels * Qtap sees the **result** of routing decisions * Changes to labels automatically discovered (no restart needed) **Automatic Service Discovery**: * Traefik watches Docker events * New containers auto-routed * Qtap captures new service traffic immediately *** ## Troubleshooting ### Not Seeing Traefik Traffic? **Check 1: Is Traefik routing correctly?** ```bash # Check Traefik dashboard curl http://localhost:8084/api/rawdata # Or check Traefik logs docker logs traefik-demo ``` **Check 2: Are services registered with Traefik?** ```bash # Verify labels are correct docker inspect service-a | grep -A 10 Labels ``` **Check 3: Is Qtap running before requests?** ```bash docker logs qtap-traefik | head -20 ``` **Check 4: Is ignore\_loopback correct?** ```yaml tap: ignore_loopback: false # MUST be false for Docker networking ``` ### Seeing `"l7Protocol": "other"`? * Wait longer after starting Qtap (6+ seconds) * Check if Traefik is using HTTP/3 (not yet supported by Qtap) * Verify traffic is actually HTTP ### Labels Not Working? **Common label mistakes:** ```yaml # ❌ Wrong - missing quotes - traefik.http.routers.myapp.rule=PathPrefix(/api) # ✅ Correct - quoted value - "traefik.http.routers.myapp.rule=PathPrefix(`/api`)" # ❌ Wrong - missing enable - "traefik.http.routers.myapp.rule=PathPrefix(`/api`)" # ✅ Correct - enable=true required - "traefik.enable=true" - "traefik.http.routers.myapp.rule=PathPrefix(`/api`)" ``` ### Too Much Traffic? Apply conditional capture: ```yaml stacks: reduced: plugins: - type: http_capture config: level: none rules: - name: "Errors only" expr: http.res.status >= 400 level: full ``` *** ## Performance Considerations ### Traefik + Qtap Performance * **CPU**: \~1-3% overhead for typical traffic * **Memory**: \~50-200MB for Qtap * **Latency**: Zero additional latency (passive observation) **Best practices for high-traffic Traefik:** 1. Use `level: summary` for high volume 2. Apply rules to capture selectively 3. Filter health checks and monitoring endpoints 4. Send to S3 with batching (Fluent Bit) 5. Set TTL policies on storage ### Scaling Recommendations | **Traffic Volume** | **Recommended Level** | **Notes** | | ------------------ | --------------------- | ------------------------- | | < 100 req/sec | `full` | Capture everything | | 100-1000 req/sec | `details` | Headers only | | 1000-10000 req/sec | `summary` | Metadata only | | > 10000 req/sec | conditional | Errors/slow requests only | *** ## Traefik vs NGINX/Caddy **Configuration**: * **Traefik**: Docker labels, dynamic discovery * **NGINX**: Static config files * **Caddy**: Static config files (but simpler) **Use Cases**: * **Traefik**: Containerized apps, Kubernetes, dynamic environments * **NGINX**: Traditional deployments, high performance * **Caddy**: Simplicity, automatic HTTPS **Qtap Compatibility**: * All three work perfectly with Qtap * Traefik's dynamic routing is fully observable * Same capture quality across all proxies *** ## Next Steps **Learn More About Qtap:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Complete configuration * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - All plugin options * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) - Progressive tutorial **Production Deployment:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) - S3 setup guide * [Capturing All HTTP Traffic with Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) - Batching for scale * [Kubernetes Manifest](/getting-started/qtap/installation/kubernetes-manifest) - Deploy in K8s **Related Guides:** * [Capturing NGINX Traffic](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) - Traditional reverse proxy * [Capturing Caddy Traffic](/guides/qtap-guides/web-server-integration/capturing-caddy-traffic) - Modern web server * [Ingress Traffic Capture with Python](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) - Application servers **Alternative: Cloud Management:** * [Qplane](/getting-started/qplane) - Manage Qtap with visual dashboards *** ## Cleanup ```bash # Stop all services docker compose down # Remove images docker compose down --rmi local # Clean up files rm backend-service.py traefik.yaml qtap.yaml docker-compose.yaml ``` *** *This guide uses validated configurations. All examples are tested and guaranteed to work with Traefik and Qtap.* # HAProxy Traffic Capture This guide shows you how to use Qtap to capture HTTP traffic flowing through **HAProxy**, the industry-standard high-performance load balancer. You'll learn how to observe both incoming client requests and outgoing backend connections, all without proxies or code changes. ## What You'll Learn * Capture HAProxy ingress traffic (client requests) * Capture HAProxy egress traffic (backend server requests) * Monitor load balancing across multiple backends * Observe health checks and failover behavior * Apply conditional capture rules for specific backends * Set up HAProxy + Qtap in Docker for testing * Deploy production-ready configurations ## Use Cases **Why capture HAProxy traffic?** * **Load Balancer Analytics**: Understand traffic distribution across backend servers * **Health Check Monitoring**: Observe health check behavior and failover events * **Performance Analysis**: Measure latency and identify slow backends * **Debugging Load Balancing**: Verify sticky sessions and routing algorithms * **API Gateway Monitoring**: Track all API calls through your edge load balancer * **Compliance & Audit**: Record all traffic for regulatory requirements * **Troubleshooting**: Debug issues between client and backend servers *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Docker installed (for this guide's examples) * Root/sudo access * Basic understanding of HAProxy configuration *** ## Part 1: HAProxy Load Balancer Setup HAProxy uses its own configuration file format. Let's set up a load balancer with multiple backend servers. ### Step 1: Create Project Directory ```bash mkdir haproxy-qtap-demo cd haproxy-qtap-demo ``` ### Step 2: Create HAProxy Configuration Create `haproxy.cfg`: ``` global log stdout local0 maxconn 4096 defaults log global mode http option httplog option dontlognull timeout connect 5000ms timeout client 50000ms timeout server 50000ms # Frontend: Listen for incoming HTTP requests frontend http_front bind *:80 # ACLs for path-based routing acl is_api path_beg /api acl is_static path_beg /static acl is_health path /health # Route based on path use_backend api_servers if is_api use_backend static_servers if is_static use_backend health_check if is_health default_backend web_servers # Backend: API servers (load balanced) backend api_servers balance roundrobin option httpchk GET /health http-check expect status 200 # Backend servers server api1 backend-api-1:8001 check inter 2000ms server api2 backend-api-2:8002 check inter 2000ms # Backend: Web servers (load balanced) backend web_servers balance leastconn server web1 backend-web-1:8003 check server web2 backend-web-2:8004 check # Backend: Static file server backend static_servers server static1 backend-static:8005 check # Backend: Health check endpoint backend health_check server health localhost:8080 # Stats page (optional) listen stats bind *:8404 stats enable stats uri /stats stats refresh 30s ``` ### Step 3: Create Backend Service Create `backend-service.py`: ```python #!/usr/bin/env python3 from http.server import HTTPServer, BaseHTTPRequestHandler import json import sys import os class Handler(BaseHTTPRequestHandler): def do_GET(self): service_name = os.getenv('SERVICE_NAME', 'unknown') port = os.getenv('PORT', '8000') # Health check endpoint if self.path == '/health': self.send_response(200) self.send_header('Content-Type', 'text/plain') self.end_headers() self.wfile.write(b'OK') return response = { "service": service_name, "port": port, "path": self.path, "message": f"Hello from {service_name} on port {port}!" } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def do_POST(self): content_length = int(self.headers.get('Content-Length', 0)) body = self.rfile.read(content_length).decode() if content_length > 0 else "" service_name = os.getenv('SERVICE_NAME', 'unknown') port = os.getenv('PORT', '8000') response = { "service": service_name, "port": port, "method": "POST", "received": body, "message": f"POST received by {service_name}" } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def log_message(self, format, *args): # Suppress default logging pass if __name__ == '__main__': port = int(sys.argv[1]) if len(sys.argv) > 1 else 8000 server = HTTPServer(('0.0.0.0', port), Handler) print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}") server.serve_forever() ``` ### Step 4: Create Qtap Configuration Create `qtap.yaml`: ```yaml version: 2 # Storage Configuration services: # Connection metadata (anonymized) event_stores: - type: stdout # HTTP request/response data (sensitive) object_stores: - type: stdout # Processing Stack stacks: haproxy_capture: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture everything format: text # (json|text) - Human-readable format # Traffic Capture Settings tap: direction: all # (egress|ingress|all) - Capture BOTH directions ignore_loopback: false # (true|false) - Capture localhost (haproxy uses loopback) audit_include_dns: false # (true|false) - Skip DNS for cleaner output http: stack: haproxy_capture # Use our haproxy processing stack # Optional: Filter out noise filters: groups: - qpoint # Don't capture qtap's own traffic ``` ### Step 5: Create Docker Compose Setup Create `docker-compose.yaml`: ```yaml version: '3.8' services: # HAProxy load balancer haproxy: image: haproxy:2.9-alpine container_name: haproxy-demo ports: - "8085:80" # HTTP - "8086:8404" # Stats page volumes: - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro networks: - demo-network # Backend API Server 1 backend-api-1: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8001"] container_name: backend-api-1 environment: - SERVICE_NAME=backend-api-1 - PORT=8001 networks: - demo-network # Backend API Server 2 backend-api-2: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8002"] container_name: backend-api-2 environment: - SERVICE_NAME=backend-api-2 - PORT=8002 networks: - demo-network # Backend Web Server 1 backend-web-1: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8003"] container_name: backend-web-1 environment: - SERVICE_NAME=backend-web-1 - PORT=8003 networks: - demo-network # Backend Web Server 2 backend-web-2: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8004"] container_name: backend-web-2 environment: - SERVICE_NAME=backend-web-2 - PORT=8004 networks: - demo-network # Backend Static Server backend-static: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py", "8005"] container_name: backend-static environment: - SERVICE_NAME=backend-static - PORT=8005 networks: - demo-network # Qtap agent qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-haproxy privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml networks: demo-network: driver: bridge ``` **Key HAProxy Concepts:** * **Frontend**: Listens for incoming connections * **Backend**: Defines pool of servers to route to * **ACL (Access Control List)**: Rules for routing decisions * **Balance Algorithm**: `roundrobin`, `leastconn`, `source`, etc. * **Health Checks**: Automatic checking of backend server health *** ## Part 2: Running and Testing ### Step 1: Start the Services ```bash # Start all services docker compose up -d # Wait for Qtap to initialize (CRITICAL!) sleep 6 # Check HAProxy stats (optional) # Open http://localhost:8086/stats in browser ``` ### Step 2: Generate Test Traffic ```bash # Test 1: Route to web backend (round-robin) curl http://localhost:8085/ # Test 2: Multiple requests to see load balancing for i in {1..6}; do curl -s http://localhost:8085/ | jq -r '.service' done # Test 3: Route to API backend curl http://localhost:8085/api/users # Test 4: Multiple API requests to see distribution for i in {1..6}; do curl -s http://localhost:8085/api/data | jq -r '.service' done # Test 5: POST to API backend curl -X POST http://localhost:8085/api/create \ -H "Content-Type: application/json" \ -d '{"name": "Alice", "role": "admin"}' # Test 6: Static content route curl http://localhost:8085/static/image.png # Test 7: Health check curl http://localhost:8085/health ``` ### Step 3: View Captured Traffic ```bash # View Qtap logs docker logs qtap-haproxy # Filter for haproxy process docker logs qtap-haproxy 2>&1 | grep -A 30 "haproxy" # Count transactions docker logs qtap-haproxy 2>&1 | grep -c "HTTP Transaction" ``` **What you should see:** ``` === HTTP Transaction === Source Process: haproxy (PID: 987, Container: haproxy-demo) Direction: INGRESS ← (client to haproxy) Method: POST URL: http://localhost:8085/api/create Status: 200 OK Duration: 12ms --- Request Headers --- Host: localhost:8085 User-Agent: curl/7.81.0 Content-Type: application/json --- Request Body --- {"name": "Alice", "role": "admin"} --- Response Headers --- Content-Type: application/json --- Response Body --- {"service":"backend-api-1","port":"8001","method":"POST","received":"{\"name\": \"Alice\", \"role\": \"admin\"}","message":"POST received by backend-api-1"} ======================== === HTTP Transaction === Source Process: haproxy (PID: 987, Container: haproxy-demo) Direction: EGRESS → (haproxy to backend) Method: POST URL: http://backend-api-1:8001/api/create Status: 200 OK Duration: 8ms --- Request Body --- {"name": "Alice", "role": "admin"} ======================== ``` **Key indicators:** * ✅ `"exe"` contains `haproxy` - Process identified * ✅ `Direction: INGRESS` - Client → HAProxy * ✅ `Direction: EGRESS` - HAProxy → Backend server * ✅ **Two transactions** per request (ingress + egress) * ✅ Load distribution visible (different backend servers) * ✅ Backend server name in egress URL *** ## Part 3: Advanced Configurations ### Configuration 1: Monitor Load Balancing Distribution Capture only egress traffic to see which backend serves each request: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: load_balance_monitoring: plugins: - type: http_capture config: level: summary # Just metadata to see distribution format: json tap: direction: egress # Only capture haproxy→backend ignore_loopback: false http: stack: load_balance_monitoring ``` Analyze logs to see traffic distribution across backends. ### Configuration 2: Capture Health Check Failures Monitor health check behavior and backend failures: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_health_check expr: http.req.path == "/health" - name: is_error expr: http.res.status >= 400 stacks: health_monitoring: plugins: - type: http_capture config: level: none format: json rules: # Skip successful health checks (too noisy) - name: "Skip healthy" expr: is_health_check() && http.res.status == 200 level: none # Capture failed health checks - name: "Health check failures" expr: is_health_check() && is_error() level: full # Capture all backend errors - name: "Backend errors" expr: is_error() && !is_health_check() level: full tap: direction: egress # Focus on haproxy→backend ignore_loopback: false http: stack: health_monitoring ``` ### Configuration 3: Backend-Specific Capture Capture different levels for different backend pools: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_api_backend expr: http.req.path matches /^\/api\// - name: is_slow expr: http.res.duration_ms > 500 stacks: selective_capture: plugins: - type: http_capture config: level: none format: json rules: # Capture all API traffic in full - name: "API traffic" expr: is_api_backend() level: full # Capture slow requests (any backend) - name: "Slow requests" expr: is_slow() level: headers # Headers only # Capture errors anywhere - name: "Errors" expr: http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: selective_capture ``` ### Configuration 4: Production Setup with S3 ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-haproxy-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false rulekit: macros: - name: is_error expr: http.res.status >= 400 stacks: production_capture: plugins: - type: http_capture config: level: none format: json rules: # Only capture errors in production - name: "Production errors" expr: is_error() level: full tap: direction: all ignore_loopback: false http: stack: production_capture ``` *** ## Part 4: Real-World Use Cases ### Use Case 1: Debugging Sticky Sessions Monitor sticky session behavior (source IP-based persistence): **haproxy.cfg**: ``` backend api_servers balance source # Sticky sessions based on source IP hash-type consistent server api1 backend-api-1:8001 check server api2 backend-api-2:8002 check ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: sticky_session_monitoring: plugins: - type: http_capture config: level: summary # Metadata shows which backend format: json tap: direction: egress # Focus on haproxy→backend routing ignore_loopback: false http: stack: sticky_session_monitoring ``` Generate traffic from same IP and verify it goes to the same backend. ### Use Case 2: Blue/Green Deployment Monitoring Monitor traffic split during blue/green deployments: **haproxy.cfg**: ``` backend app_servers # 90% traffic to blue (stable) server blue1 blue-app-1:8001 check weight 90 server blue2 blue-app-2:8002 check weight 90 # 10% traffic to green (canary) server green1 green-app-1:8001 check weight 10 ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: deployment_monitoring: plugins: - type: http_capture config: level: summary format: json rules: # Capture all traffic to see distribution - name: "All traffic" expr: http.res.status >= 0 level: summary tap: direction: egress ignore_loopback: false http: stack: deployment_monitoring ``` Analyze logs to verify 90/10 split and monitor error rates per version. ### Use Case 3: API Rate Limiting Detection Monitor for rate limiting and throttling: **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_rate_limited expr: http.res.status == 429 || http.res.status == 503 - name: is_retry expr: http.req.headers.retry-after != "" stacks: rate_limit_monitoring: plugins: - type: http_capture config: level: none format: json rules: # Capture rate limiting events - name: "Rate limited" expr: is_rate_limited() level: full # Capture retry attempts - name: "Retries" expr: is_retry() level: headers tap: direction: all ignore_loopback: false http: stack: rate_limit_monitoring ``` ### Use Case 4: Multi-Datacenter Load Balancing Monitor traffic distribution across multiple datacenters: **haproxy.cfg**: ``` backend geo_distributed # Primary datacenter (low latency) server dc1-web1 dc1-web-1:8001 check server dc1-web2 dc1-web-2:8002 check # Backup datacenter (high latency backup) server dc2-web1 dc2-web-1:8001 check backup server dc2-web2 dc2-web-2:8002 check backup ``` **qtap.yaml**: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_backup_dc expr: http.req.url matches /dc2-/ stacks: datacenter_monitoring: plugins: - type: http_capture config: level: none format: json rules: # Always capture backup datacenter traffic (should be rare) - name: "Backup DC traffic" expr: is_backup_dc() level: full # Capture primary DC errors - name: "Primary DC errors" expr: '!is_backup_dc() && http.res.status >= 500' level: full tap: direction: egress ignore_loopback: false http: stack: datacenter_monitoring ``` *** ## Understanding HAProxy + Qtap ### Dual Capture for Load Balancing When HAProxy routes a request, Qtap captures **two transactions**: **Transaction 1: INGRESS (Client → HAProxy)** ``` Source Process: haproxy Direction: INGRESS ← URL: http://localhost:8085/api/users ``` **Transaction 2: EGRESS (HAProxy → Backend)** ``` Source Process: haproxy Direction: EGRESS → URL: http://backend-api-1:8001/api/users # Or backend-api-2 depending on load balancing ``` This lets you: * See which backend served each request * Measure HAProxy overhead (ingress duration - egress duration) * Verify load balancing algorithm behavior * Detect backend-specific issues ### HAProxy-Specific Features **Process Identification**: * Look for `exe` containing `haproxy` * Typically `/usr/local/sbin/haproxy` **Load Balancing Algorithms**: * **roundrobin**: Rotate through backends equally * **leastconn**: Send to backend with fewest connections * **source**: Sticky sessions based on source IP * **uri**: Route based on request URI Qtap shows which backend was chosen for each request. **Health Checks**: * HAProxy constantly health checks backends * Qtap captures these checks (can be filtered out) * Failed health checks visible in logs *** ## Troubleshooting ### Not Seeing HAProxy Traffic? **Check 1: Is HAProxy running?** ```bash docker logs haproxy-demo # Should see backend servers marked as UP ``` **Check 2: Is Qtap running before requests?** ```bash docker logs qtap-haproxy | head -20 ``` **Check 3: Are backends healthy?** ```bash # Check HAProxy stats curl http://localhost:8086/stats # Or check logs docker logs haproxy-demo | grep -i "check" ``` **Check 4: Is ignore\_loopback correct?** ```yaml tap: ignore_loopback: false # MUST be false ``` ### Seeing Only Health Checks? Health checks are noisy. Filter them out: ```yaml filters: custom: - exe: /usr/local/sbin/haproxy strategy: exact # Then use rules to capture only non-health-check traffic ``` Or in rules: ```yaml rules: - name: "Skip health checks" expr: http.req.path != "/health" level: full ``` ### Backend Server Down? If a backend is down, HAProxy won't route to it. Check logs: ```bash # Check which backends are UP docker logs haproxy-demo | grep "UP\|DOWN" # Restart a backend docker restart backend-api-1 ``` ### Too Much Traffic? Apply conditional capture: ```yaml config: level: none rules: - name: "Errors only" expr: http.res.status >= 400 level: full ``` *** ## Performance Considerations ### HAProxy + Qtap Performance * **CPU**: \~1-3% overhead * **Memory**: \~50-200MB for Qtap * **Latency**: Zero additional latency (passive observation) **HAProxy is extremely performance-sensitive**. Best practices: 1. Use `level: summary` for high volume 2. Filter health checks (very noisy) 3. Capture selectively with rules 4. Send to S3 with batching 5. Monitor Qtap resource usage ### Scaling Recommendations | **Traffic Volume** | **Recommended Level** | **Notes** | | -------------------- | --------------------- | --------------------------------- | | < 1000 req/sec | `full` | Capture everything | | 1000-10000 req/sec | `details` | Headers only | | 10000-100000 req/sec | `summary` | Metadata only | | > 100000 req/sec | conditional | Errors only, aggressive filtering | HAProxy can handle millions of connections. Qtap scales with it. *** ## HAProxy vs NGINX/Caddy/Traefik **Purpose**: * **HAProxy**: Dedicated load balancer (Layer 4 + Layer 7) * **NGINX**: Web server + reverse proxy + load balancer * **Caddy**: Web server + automatic HTTPS * **Traefik**: Cloud-native reverse proxy **Performance**: * **HAProxy**: Extreme performance, lowest latency * Others: Fast, but not HAProxy-level **Configuration**: * **HAProxy**: Own syntax, focused on load balancing * **NGINX**: nginx.conf * **Caddy**: Caddyfile * **Traefik**: Docker labels/YAML **Qtap Compatibility**: * All work perfectly with Qtap * Same capture quality across all *** ## Next Steps **Learn More About Qtap:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) * [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) **Production Deployment:** * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) * [Capturing All HTTP Traffic with Fluent Bit](/guides/qtap-guides/observability-and-integration/capturing-all-http-traffic-with-fluent-bit) * [Kubernetes Manifest](/getting-started/qtap/installation/kubernetes-manifest) **Related Guides:** * [Capturing NGINX Traffic](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) * [Capturing Caddy Traffic](/guides/qtap-guides/web-server-integration/capturing-caddy-traffic) * [Capturing Traefik Traffic](/guides/qtap-guides/web-server-integration/capturing-traefik-traffic) * [Ingress Traffic Capture with Python](/guides/qtap-guides/getting-started/ingress-traffic-capture-with-python) **Alternative: Cloud Management:** * [Qplane](/getting-started/qplane) - Manage Qtap with visual dashboards *** ## Cleanup ```bash # Stop all services docker compose down # Remove images docker compose down --rmi local # Clean up files rm backend-service.py haproxy.cfg qtap.yaml docker-compose.yaml ``` *** *This guide uses validated configurations. All examples are tested and guaranteed to work with HAProxy and Qtap.* # Envoy Traffic Capture This guide shows you how to use Qtap to capture HTTP traffic flowing through **Envoy**, the modern cloud-native proxy designed for service meshes. You'll learn how to observe both incoming and outgoing connections in a microservices architecture. ## What You'll Learn * Capture Envoy ingress traffic (incoming requests) * Capture Envoy egress traffic (upstream service requests) * Monitor service mesh traffic patterns * Configure Envoy listeners, routes, and clusters * Apply conditional capture rules * Set up Envoy + Qtap in Docker for testing * Deploy production-ready configurations ## Use Cases **Why capture Envoy traffic?** * **Service Mesh Visibility**: Monitor all service-to-service communication * **Istio/Consul Integration**: Observe traffic in service mesh deployments * **gRPC Traffic Inspection**: See gRPC calls between microservices * **Debugging Routing**: Verify Envoy routes traffic correctly * **Observability**: Deep insights into request/response patterns * **Security Auditing**: Monitor for suspicious traffic patterns * **Performance Analysis**: Measure latency and identify bottlenecks *** ## Prerequisites * Linux system with kernel 5.10+ and eBPF support * Docker installed (for this guide's examples) * Root/sudo access * Basic understanding of Envoy configuration concepts *** ## Part 1: Basic Envoy Proxy Setup Envoy uses YAML configuration with a specific structure: listeners, routes, and clusters. ### Step 1: Create Project Directory ```bash mkdir envoy-qtap-demo cd envoy-qtap-demo ``` ### Step 2: Create Envoy Configuration Create `envoy.yaml`: ```yaml static_resources: listeners: - name: listener_0 address: socket_address: address: 0.0.0.0 port_value: 10000 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: ingress_http access_log: - name: envoy.access_loggers.stdout typed_config: "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog http_filters: - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router route_config: name: local_route virtual_hosts: - name: backend domains: ["*"] routes: # Route /api/* to API service - match: prefix: "/api/" route: prefix_rewrite: "/" cluster: api_service # Route /web/* to web service - match: prefix: "/web/" route: prefix_rewrite: "/" cluster: web_service # Default route to httpbin - match: prefix: "/" route: cluster: httpbin_service clusters: - name: api_service type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: api_service endpoints: - lb_endpoints: - endpoint: address: socket_address: address: backend-api port_value: 8001 - name: web_service type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: web_service endpoints: - lb_endpoints: - endpoint: address: socket_address: address: backend-web port_value: 8002 - name: httpbin_service type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: httpbin_service endpoints: - lb_endpoints: - endpoint: address: socket_address: address: httpbin.org port_value: 80 admin: address: socket_address: address: 0.0.0.0 port_value: 9901 ``` ### Step 3: Create Backend Service Create `backend-service.py`: ```python #!/usr/bin/env python3 from http.server import HTTPServer, BaseHTTPRequestHandler import json import os class Handler(BaseHTTPRequestHandler): def do_GET(self): service_name = os.getenv('SERVICE_NAME', 'unknown') response = { "service": service_name, "path": self.path, "message": f"Hello from {service_name}!" } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def do_POST(self): content_length = int(self.headers.get('Content-Length', 0)) body = self.rfile.read(content_length).decode() if content_length > 0 else "" service_name = os.getenv('SERVICE_NAME', 'unknown') response = { "service": service_name, "method": "POST", "received": body } self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(response).encode()) def log_message(self, format, *args): pass if __name__ == '__main__': port = int(os.getenv('PORT', '8000')) server = HTTPServer(('0.0.0.0', port), Handler) print(f"Service {os.getenv('SERVICE_NAME', 'unknown')} listening on port {port}") server.serve_forever() ``` ### Step 4: Create Qtap Configuration Create `qtap.yaml`: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: envoy_capture: plugins: - type: http_capture config: level: full # (none|summary|headers|full) format: text # (json|text) tap: direction: all # (egress|ingress|all) ignore_loopback: false # (true|false) audit_include_dns: false # (true|false) http: stack: envoy_capture filters: groups: - qpoint ``` ### Step 5: Create Docker Compose Setup Create `docker-compose.yaml`: ```yaml version: '3.8' services: # Envoy proxy envoy: image: envoyproxy/envoy:v1.31-latest container_name: envoy-demo ports: - "8087:10000" # HTTP listener - "8088:9901" # Admin interface volumes: - ./envoy.yaml:/etc/envoy/envoy.yaml:ro command: ["-c", "/etc/envoy/envoy.yaml"] networks: - demo-network # Backend API service backend-api: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py"] container_name: backend-api environment: - SERVICE_NAME=backend-api - PORT=8001 networks: - demo-network # Backend web service backend-web: build: context: . dockerfile_inline: | FROM python:3.11-slim WORKDIR /app COPY backend-service.py /app/ RUN chmod +x /app/backend-service.py CMD ["python3", "/app/backend-service.py"] container_name: backend-web environment: - SERVICE_NAME=backend-web - PORT=8002 networks: - demo-network # Qtap agent qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-envoy privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml networks: demo-network: driver: bridge ``` **Key Envoy Concepts:** * **Listeners**: Accept incoming connections * **Routes**: Map requests to clusters * **Clusters**: Groups of upstream endpoints * **Filters**: Process requests (router, logging, etc.) *** ## Part 2: Running and Testing ### Step 1: Start the Services ```bash # Start all services docker compose up -d # Wait for Qtap to initialize (CRITICAL!) sleep 6 # Check Envoy admin interface (optional) curl http://localhost:8088/stats ``` ### Step 2: Generate Test Traffic ```bash # Test 1: Route to API service curl http://localhost:8087/api/users # Test 2: Route to web service curl http://localhost:8087/web/index # Test 3: Default route (httpbin) curl http://localhost:8087/get # Test 4: POST to API service curl -X POST http://localhost:8087/api/create \ -H "Content-Type: application/json" \ -d '{"name": "Alice"}' # Test 5: Multiple requests to see routing for i in {1..5}; do curl -s http://localhost:8087/api/data | jq . curl -s http://localhost:8087/web/page | jq . done ``` ### Step 3: View Captured Traffic ```bash # View Qtap logs docker logs qtap-envoy # Filter for envoy process docker logs qtap-envoy 2>&1 | grep -A 30 "envoy" # Check directions docker logs qtap-envoy 2>&1 | grep "Direction:" ``` **What you should see:** ``` === HTTP Transaction === Source Process: envoy (PID: 123, Container: envoy-demo) Direction: INGRESS ← (client to envoy) Method: POST URL: http://localhost:8087/api/create Status: 200 OK --- Request Headers --- Host: localhost:8087 Content-Type: application/json --- Request Body --- {"name": "Alice"} ======================== === HTTP Transaction === Source Process: envoy (PID: 123, Container: envoy-demo) Direction: EGRESS → (envoy to backend) Method: POST URL: http://backend-api:8001/create Status: 200 OK ======================== ``` **Key indicators:** * ✅ `"exe"` contains `envoy` * ✅ `Direction: INGRESS` - Client → Envoy * ✅ `Direction: EGRESS` - Envoy → Backend * ✅ Path transformation visible * ✅ Cluster routing working *** ## Part 3: Advanced Configurations ### Configuration 1: Service Mesh Monitoring Capture service-to-service traffic: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: mesh_monitoring: plugins: - type: http_capture config: level: summary # Metadata for analytics format: json tap: direction: all ignore_loopback: false http: stack: mesh_monitoring ``` ### Configuration 2: Selective Capture by Service ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_api_service expr: http.req.path matches /^\/api\// - name: is_error expr: http.res.status >= 400 stacks: selective: plugins: - type: http_capture config: level: none format: json rules: # Capture API traffic - name: "API traffic" expr: is_api_service() level: full # Capture errors - name: "Errors" expr: is_error() level: full tap: direction: all ignore_loopback: false http: stack: selective ``` ### Configuration 3: Production with S3 ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com region: us-east-1 bucket: my-company-envoy-traffic access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY insecure: false stacks: production: plugins: - type: http_capture config: level: none format: json rules: - name: "Errors only" expr: http.res.status >= 400 level: full tap: direction: all ignore_loopback: false http: stack: production ``` *** ## Real-World Use Cases ### Use Case 1: Istio Service Mesh Monitor Envoy sidecars in Istio: ```yaml # Qtap deployed as DaemonSet captures all Envoy traffic tap: direction: all http: stack: service_mesh_capture # Analyze service-to-service communication # See which services talk to each other # Measure latency between services ``` ### Use Case 2: API Gateway Envoy as edge proxy: ```yaml rulekit: macros: - name: is_external expr: http.req.headers.x-forwarded-for != "" stacks: api_gateway: plugins: - type: http_capture config: level: none rules: # Capture external traffic - name: "External requests" expr: is_external() level: full ``` *** ## Troubleshooting ### Not Seeing Envoy Traffic? **Check 1: Is Envoy running?** ```bash docker logs envoy-demo curl http://localhost:8088/stats ``` **Check 2: Qtap running before traffic?** ```bash docker logs qtap-envoy | head -20 ``` **Check 3: Envoy routing correctly?** ```bash # Check Envoy config curl http://localhost:8088/config_dump ``` ### Seeing `"l7Protocol": "other"`? * Envoy might be using HTTP/2 or HTTP/3 * Wait longer after starting Qtap * Check if TLS is involved *** ## Performance Considerations **Envoy + Qtap:** * CPU: \~1-3% overhead * Memory: \~50-200MB * Latency: Zero additional (passive) **Best practices:** 1. Use `summary` for high volume 2. Apply conditional rules 3. Filter admin traffic 4. Send to S3 with batching *** ## Next Steps **Learn More:** * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) **Related Guides:** * [Capturing NGINX Traffic](/guides/qtap-guides/web-server-integration/capturing-nginx-traffic) * [Capturing Traefik Traffic](/guides/qtap-guides/web-server-integration/capturing-traefik-traffic) * [Capturing HAProxy Traffic](/guides/qtap-guides/web-server-integration/capturing-haproxy-traffic) *** ## Cleanup ```bash docker compose down docker compose down --rmi local rm backend-service.py envoy.yaml qtap.yaml docker-compose.yaml ``` *** *This guide uses validated configurations for Envoy and Qtap.* # Observability & Integration # Prometheus + Grafana Monitoring This guide walks you through setting up comprehensive monitoring for qtap using Prometheus for metrics collection and Grafana for visualization. By the end, you'll have dashboards showing request rates, error rates, latency percentiles, and traffic patterns. ## What You'll Build * **Prometheus** scraping qtap metrics every 15 seconds * **Grafana dashboard** visualizing HTTP traffic patterns * **Alerts** for high error rates and latency spikes * **RED metrics** (Rate, Errors, Duration) for all observed traffic ## Prerequisites * Qtap installed and running (see [Getting Started](/getting-started/qtap/getting-started)) * Docker or Kubernetes environment * Basic familiarity with Prometheus and Grafana ## Step 0: Enable HTTP Metrics Plugin {% hint style="danger" %} **CRITICAL FIRST STEP**: Qtap requires the `http_metrics` plugin to expose HTTP-level Prometheus metrics. Without this plugin, you'll only see connection-level metrics. {% endhint %} ### Update Your Qtap Configuration Edit your qtap configuration file to include the `http_metrics` plugin: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: monitoring_stack: plugins: # HTTP capture plugin (optional - for logging/storage) - type: http_capture config: level: summary format: json # HTTP metrics plugin (REQUIRED for Prometheus metrics) - type: http_metrics tap: direction: egress http: stack: monitoring_stack ``` ### Restart Qtap After adding the plugin, restart qtap: ```bash # Docker docker restart qtap # Kubernetes kubectl rollout restart daemonset/qtap ``` ## Step 1: Verify Metrics Are Available First, confirm qtap is exposing HTTP metrics: ```bash # Check qtap is running docker ps | grep qtap # Access metrics endpoint curl http://localhost:10001/metrics | grep qtap_http_requests_total ``` You should see Prometheus-formatted metrics: ``` # HELP qtap_http_requests_total Total HTTP requests observed # TYPE qtap_http_requests_total counter qtap_http_requests_total{host="httpbin.org",method="GET",protocol="http2",status_code="200"} 5 ``` {% hint style="warning" %} **If you don't see `qtap_http_requests_total`**, the `http_metrics` plugin is not configured. Go back to Step 0. {% endhint %} If metrics exist but show zero, generate test traffic: ```bash # Generate test traffic curl https://httpbin.org/get # Check metrics again (wait 5 seconds for metrics to update) sleep 5 curl http://localhost:10001/metrics | grep qtap_http_requests_total ``` ## Step 2: Deploy Prometheus {% hint style="info" %} **Already have Prometheus running?** Skip the deployment sections below and jump to [adding the qtap scrape configuration](#add-qtap-scrape-config-to-existing-prometheus) to your existing `prometheus.yml`. {% endhint %} ### Option A: Docker Compose Create `prometheus.yml`: ```yaml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: # Application metrics - job_name: 'qtap' static_configs: - targets: ['host.docker.internal:10001'] metrics_path: '/metrics' # Agent health metrics - job_name: 'qtap-system' static_configs: - targets: ['host.docker.internal:10001'] metrics_path: '/system/metrics' scrape_interval: 30s ``` Create `docker-compose.yml`: ```yaml services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prometheus-data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' extra_hosts: - "host.docker.internal:host-gateway" # On Linux you may need this to reach the host; see note below grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=admin - GF_USERS_ALLOW_SIGN_UP=false volumes: prometheus-data: grafana-data: ``` Start the stack: ```bash docker compose up -d ``` ### Option B: Kubernetes with ServiceMonitor If using Prometheus Operator, create a ServiceMonitor: ```yaml apiVersion: v1 kind: Service metadata: name: qtap-metrics labels: app: qtap spec: selector: app: qtap ports: - name: metrics port: 10001 targetPort: 10001 --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: qtap labels: app: qtap spec: selector: matchLabels: app: qtap endpoints: - port: metrics path: /metrics interval: 15s - port: metrics path: /system/metrics interval: 30s ``` Apply it: ```bash kubectl apply -f qtap-servicemonitor.yaml ``` ### Option C: Kubernetes with Pod Annotations For standard Prometheus server, add annotations to your qtap DaemonSet: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: qtap spec: template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "10001" prometheus.io/path: "/metrics" spec: # ... qtap pod spec ``` ### Add Qtap Scrape Config to Existing Prometheus If you already have Prometheus running, add these scrape configs to your existing `prometheus.yml`: ```yaml scrape_configs: # ... your existing jobs ... # Qtap application metrics - job_name: 'qtap' # Docker for Mac/Windows exposes host.docker.internal automatically. # On Linux, add --add-host=host.docker.internal:host-gateway (Compose v2) # or replace with your bridge IP (commonly 172.17.0.1). static_configs: - targets: [':10001'] # Replace with your qtap host metrics_path: '/metrics' scrape_interval: 15s # Qtap system/health metrics - job_name: 'qtap-system' static_configs: - targets: [':10001'] metrics_path: '/system/metrics' scrape_interval: 30s ``` **For Kubernetes with Prometheus Operator:** If you already have Prometheus Operator installed, just apply the ServiceMonitor from Option B above - no need to modify Prometheus config files. Reload Prometheus configuration: ```bash # Docker docker exec prometheus kill -HUP 1 # Kubernetes (if using Prometheus Operator, reload is automatic) kubectl rollout restart deployment/prometheus-server -n monitoring ``` ## Step 3: Verify Prometheus Is Scraping Open Prometheus UI at `http://localhost:9090` and check: 1. **Status → Targets**: Verify `qtap` and `qtap-system` jobs are "UP" 2. **Run a query**: Try `qtap_http_requests_total` in the query box If targets show as DOWN: ```bash # Check network connectivity from Prometheus container # macOS/Windows docker exec prometheus wget -O- http://host.docker.internal:10001/metrics # Linux (if host.docker.internal is unavailable) # docker exec prometheus wget -O- http://172.17.0.1:10001/metrics # For Kubernetes, check service and endpoints kubectl get svc qtap-metrics kubectl get endpoints qtap-metrics ``` ## Step 4: Import Grafana Dashboard {% hint style="info" %} **Already have Grafana running?** Great! Skip ahead to [Add Prometheus Data Source](#add-prometheus-data-source) and use your existing Grafana instance. {% endhint %} ### Access Grafana Navigate to `http://localhost:3000` and log in: * **Username**: admin * **Password**: admin (or value from `GF_SECURITY_ADMIN_PASSWORD`) ### Add Prometheus Data Source {% hint style="info" %} **Already have Prometheus configured as a data source?** You can skip this section and go straight to [Import Qtap Dashboard](#import-qtap-dashboard). {% endhint %} 1. Navigate to **Configuration** → **Data Sources** 2. Click **Add data source** 3. Select **Prometheus** 4. Set URL: * Docker Compose: `http://prometheus:9090` * Kubernetes: `http://prometheus-server.monitoring.svc.cluster.local` 5. Click **Save & Test** ### Import Qtap Dashboard 1. Download the dashboard: [qtap-http-overview.json](https://github.com/qpoint-io/qtap/blob/main/examples/dashboards/qtap-http-overview.json) 2. In Grafana, navigate to **Dashboards** → **Import** 3. Click **Upload JSON file** and select the downloaded file 4. Select your Prometheus data source 5. Click **Import** {% hint style="info" %} The official Grafana dashboard may need label adjustments. Qtap v0 uses `host` labels, not `domain`. If panels are empty, edit queries to replace `domain` with `host`. {% endhint %} ### Dashboard Panels The dashboard includes: * **Request Rate**: Total requests per second over time * **Error Rate**: Percentage of 4xx/5xx responses * **Average Response Time**: Mean response duration * **Latency Percentiles**: p50, p95, p99 request duration * **Request/Response Sizes**: Average payload sizes * **Top Hosts**: Highest traffic hosts * **Status Code Distribution**: Breakdown by HTTP status * **Protocol Distribution**: HTTP/1.1 vs HTTP/2 traffic ## Step 5: Set Up Alerts Create `alerts.yml` for Prometheus: ```yaml groups: - name: qtap-alerts interval: 30s rules: # High error rate alert - alert: HighHTTPErrorRate expr: | ( sum(rate(qtap_http_requests_total{status_code=~"5.."}[5m])) / sum(rate(qtap_http_requests_total[5m])) ) > 0.05 for: 5m labels: severity: warning component: qtap annotations: summary: "High HTTP 5xx error rate detected" description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes" # High latency alert - alert: HighResponseLatency expr: | histogram_quantile(0.95, rate(qtap_http_requests_duration_ms_bucket[5m]) ) > 1000 for: 10m labels: severity: warning component: qtap annotations: summary: "High response latency detected" description: "95th percentile latency is {{ $value }}ms" # Traffic spike alert - alert: TrafficSpike expr: | rate(qtap_http_requests_total[5m]) > 2 * rate(qtap_http_requests_total[1h] offset 1h) for: 5m labels: severity: info component: qtap annotations: summary: "Unusual traffic spike detected" description: "Current request rate is {{ $value | humanize }}req/s" # Qtap agent health - alert: QtapAgentDown expr: up{job="qtap"} == 0 for: 2m labels: severity: critical component: qtap annotations: summary: "Qtap agent is down" description: "Qtap metrics endpoint is not responding" ``` Update `prometheus.yml` to load the rules: ```yaml global: scrape_interval: 15s rule_files: - /etc/prometheus/alerts.yml alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] # Optional: configure Alertmanager scrape_configs: # ... existing scrape configs ``` Reload Prometheus configuration: ```bash # Docker Compose docker compose restart prometheus # Kubernetes kubectl rollout restart deployment prometheus-server ``` ### Configure Grafana Alerts (Alternative) You can also set up alerts directly in Grafana: 1. Open a dashboard panel 2. Click **Edit** → **Alert** tab 3. Create alert conditions based on the queries 4. Configure notification channels (Slack, PagerDuty, email, etc.) ## Step 6: Test the Monitoring Setup Generate various types of traffic to see metrics flow through: ### Successful Requests ```bash for i in {1..50}; do curl -s https://httpbin.org/get > /dev/null sleep 0.1 done ``` ### Error Responses ```bash for i in {1..10}; do curl -s https://httpbin.org/status/500 > /dev/null sleep 0.2 done ``` ### High Latency ```bash for i in {1..5}; do curl -s https://httpbin.org/delay/2 > /dev/null done ``` ### View in Grafana 1. Open the qtap dashboard 2. Observe request rate increase 3. See error rate spike from 500 responses 4. Check latency percentiles from delayed requests ## Step 7: Optimize for Production ### Reduce Metric Cardinality High cardinality (many unique label combinations) can impact Prometheus performance. To reduce it: **Filter noisy processes** in qtap config: ```yaml filters: groups: - qpoint custom: - exe: /usr/bin/health-check strategy: exact - exe: /usr/local/bin/monitoring- strategy: prefix ``` **Focus on important domains**: ```yaml tap: endpoints: - domain: 'api.production.example.com' http: stack: monitored_stack ``` ### Use Recording Rules Pre-compute expensive queries with Prometheus recording rules: ```yaml groups: - name: qtap-recordings interval: 15s rules: # Pre-compute request rate by host - record: qtap:http_requests:rate5m expr: | sum by (host) (rate(qtap_http_requests_total[5m])) # Pre-compute error rate - record: qtap:http_errors:rate5m expr: | sum(rate(qtap_http_requests_total{status_code=~"5.."}[5m])) / sum(rate(qtap_http_requests_total[5m])) # Pre-compute p95 latency - record: qtap:http_latency:p95 expr: | histogram_quantile(0.95, sum(rate(qtap_http_requests_duration_ms_bucket[5m])) by (le) ) ``` Use these in your dashboards and alerts for better performance. ### Retention Settings Configure Prometheus retention based on your needs: ```yaml # docker-compose.yml command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention.time=30d' # Keep 30 days - '--storage.tsdb.retention.size=10GB' # Or max 10GB ``` ## Common Queries for Troubleshooting ### Find services with high error rates ```promql topk(5, sum by (host) (rate(qtap_http_requests_total{status_code=~"5.."}[5m])) / sum by (host) (rate(qtap_http_requests_total[5m])) ) ``` ### Identify slow endpoints ```promql topk(10, histogram_quantile(0.95, sum by (host, le) (rate(qtap_http_requests_duration_ms_bucket[5m])) ) ) ``` ### HTTP/2 vs HTTP/1 Traffic ```promql sum by (protocol) (rate(qtap_http_requests_total[5m])) ``` ### Average Request/Response Sizes ```promql # Average request size rate(qtap_http_requests_size_bytes_sum[5m]) / rate(qtap_http_requests_size_bytes_count[5m]) # Average response size rate(qtap_http_responses_size_bytes_sum[5m]) / rate(qtap_http_responses_size_bytes_count[5m]) ``` ### Compare traffic patterns over time ```promql sum(rate(qtap_http_requests_total[5m])) / sum(rate(qtap_http_requests_total[5m] offset 1d)) ``` ### Monitor overall system latency ```promql # Average end-to-end transaction time (combined request+response) rate(qtap_http_duration_ms_sum[5m]) / rate(qtap_http_duration_ms_count[5m]) # 95th percentile overall latency histogram_quantile(0.95, rate(qtap_http_duration_ms_bucket[5m]) ) ``` ### Check active connections ```promql # Total active connections sum(qtap_connection_active_total) # Active connections by destination topk(10, qtap_connection_active_total) ``` ### Track TLS/HTTPS usage ```promql # TLS handshake rate (HTTPS connection establishment) sum(rate(qtap_connection_tls_handshake_total[5m])) # Connections by TLS version sum by (version) (rate(qtap_connection_tls_handshake_total[5m])) ``` ## Troubleshooting ### No HTTP metrics appearing in Prometheus **Most Common Issue**: Missing `http_metrics` plugin. ```bash # Check if HTTP metrics exist curl http://localhost:10001/metrics | grep "qtap_http_requests_total" ``` If empty: 1. **Verify `http_metrics` plugin is in your qtap config:** ```yaml stacks: my_stack: plugins: - type: http_metrics # ← Must be present ``` 2. **Restart qtap after adding it:** ```bash docker restart qtap ``` 3. **Verify metrics now appear:** ```bash curl http://localhost:10001/metrics | grep "qtap_http_requests_total" ``` ### Only seeing connection metrics If you see `qtap_connection_*` metrics but no `qtap_http_*` metrics, the `http_metrics` plugin is not configured. See Step 0. ### Metrics show but dashboard is empty 1. **Check label names**: Qtap v0 uses `host`, not `domain`. Update queries: ```promql # Wrong sum by (domain) (rate(qtap_http_requests_total[5m])) # Correct sum by (host) (rate(qtap_http_requests_total[5m])) ``` 2. **Check data source**: Ensure Grafana is connected to the right Prometheus 3. **Check time range**: Extend the time range in Grafana 4. **Verify queries**: Test dashboard queries directly in Prometheus ### Prometheus can't reach qtap ```bash # Docker Compose (Mac/Windows) docker exec prometheus wget -O- http://host.docker.internal:10001/metrics # Docker Compose (Linux): # docker exec prometheus wget -O- http://172.17.0.1:10001/metrics # Kubernetes kubectl run curl-test --image=curlimages/curl --rm -it -- \ curl http://qtap-metrics.default.svc.cluster.local:10001/metrics ``` ### High cardinality warnings ``` level=warn msg="Metric has too many labels" ``` This means you have too many unique label combinations. Solutions: 1. Filter processes in qtap configuration 2. Use recording rules to aggregate 3. Limit domain capture with endpoints configuration 4. Consider using Prometheus remote write to long-term storage ### Missing metrics after qtap restart Prometheus counters reset when qtap restarts. This is normal. Use `rate()` function which handles counter resets automatically. ## Next Steps * **Customize dashboards**: Add panels for your specific use cases * **Set up Alertmanager**: Route alerts to Slack, PagerDuty, or email * **Create SLOs**: Define service level objectives based on qtap metrics * **Integrate with logs**: Correlate metrics with qtap captured payloads in your object store * **Multi-cluster monitoring**: Federate metrics from multiple qtap deployments ## Additional Resources * [Metrics Configuration Reference](/getting-started/qtap/configuration/metrics) * [Prometheus Metrics Reference](/appendix/prometheus-metrics) * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) * [Prometheus Documentation](https://prometheus.io/docs/) * [Grafana Tutorials](https://grafana.com/tutorials/) # OpenTelemetry Integration This guide shows you how to send qtap observability data to OpenTelemetry-compatible backends using OTLP (OpenTelemetry Protocol). ## Understanding Qtap's OpenTelemetry Integration Qtap exports two types of data as **OpenTelemetry Logs** via the OTLP protocol: | Data Type | Configuration | What It Contains | Use Case | | ----------- | ------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | **Events** | qtap.yaml `event_stores` | Connection metadata, HTTP transaction summaries (method, URL, status, duration, process attribution) | Analytics, monitoring, troubleshooting — know WHAT happened and WHICH process did it | | **Objects** | qtap.yaml `object_stores` | Full request/response headers and bodies | Compliance, security investigation, debugging — see the actual payload content | Both can be sent through the same OTel collector endpoint, giving you a **unified pipeline** for all captured data — no separate S3 setup required. {% hint style="info" %} **Events vs Objects:** Events are lightweight metadata about every connection and HTTP transaction. Objects are the full request/response content, captured selectively based on your [plugin configuration](/getting-started/qtap/configuration/traffic-processing-with-plugins). You can send both through OTel, or use OTel for events and S3 for objects — see [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) for all options. {% endhint %} ## Prerequisites * Qtap installed and running (see [Getting Started](/getting-started/qtap/getting-started)) * An OpenTelemetry-compatible backend or collector ## Quick Start (5 minutes) Send both events and artifacts to any OpenTelemetry-compatible backend. ### Step 1: Configure Qtap Create `qtap-config.yaml`: ```yaml version: 2 services: event_stores: - type: otel # Send logs to OpenTelemetry endpoint: "localhost:4317" # OTel Collector gRPC endpoint protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false # Set to true for production object_stores: - type: otel # Send artifacts through OTel (or use S3/stdout) otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false stacks: default: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: false # (true|false) audit_include_dns: false # (true|false) http: stack: default ``` ### Step 2: Deploy OpenTelemetry Collector {% hint style="info" %} **Already have an OpenTelemetry Collector?** Update the `endpoint` in Step 1 to point to your collector and skip to [Step 3](#step-3-start-qtap-logs-only). {% endhint %} Create `otel-collector-config.yaml`: ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 10s exporters: # Debug exporter - prints logs to console for testing debug: verbosity: detailed # Add your backend exporter here # otlphttp: # endpoint: "https://your-backend.com/v1/logs" # headers: # api-key: "${API_KEY}" service: pipelines: logs: # Logs pipeline for qtap event_stores receivers: [otlp] processors: [batch] exporters: [debug] # Add your backend exporter ``` Create `docker-compose.yaml`: ```yaml services: otel-collector: image: otel/opentelemetry-collector:latest container_name: otel-collector command: ["--config=/etc/otel-collector-config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP ``` Start the collector: ```bash docker compose up -d ``` ### Step 3: Start Qtap ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" ``` {% hint style="info" %} **Note**: No additional environment variables are needed — logs are configured entirely through qtap.yaml. {% endhint %} ### Step 4: Generate Traffic and Verify ```bash # Generate test traffic curl https://httpbin.org/get # Check OTel Collector logs docker logs otel-collector ``` **You should see two types of log events:** **1. Connection Event:** ``` LogRecord Body: Connection: [egress-external via tcp] hostname → httpbin.org Attributes: event.type: connection event.l7Protocol: http2 event.tlsVersion: 772 event.meta.connectionId: d3sggv07p3qrvfkj173g event.source.exe: /usr/bin/curl ``` **2. Artifact Record Event:** ``` LogRecord Body: Artifact stored: http_transaction Attributes: event.type: artifact_record event.summary.request_method: GET event.summary.request_host: httpbin.org event.summary.response_status: 200 event.summary.duration_ms: 2023 event.summary.connection_id: d3sggv07p3qrvfkj173g ``` **3. Object (Artifact Data):** When using OTel object stores, the full request/response content is also sent as log records: ``` LogRecord Body: Artifact: http_request (application/json, 1247 bytes) Attributes: artifact.type: http_request artifact.content_type: application/json artifact.digest: sha256:a1b2c3d4... artifact.size_bytes: 1247 artifact.data: connection.id: d3sggv07p3qrvfkj173g connection.endpoint_id: httpbin.org connection.request_id: d3sggv07p3qrvfkj174g ``` Objects include the complete HTTP headers and bodies as captured by the `http_capture` plugin. The `connection.id` links objects back to their corresponding connection and artifact record events. {% hint style="success" %} **Success!** Qtap is sending events and artifacts to OpenTelemetry. Use `connection.id` to correlate connection metadata, HTTP transaction summaries, and full request/response content. {% endhint %} ## Understanding the Data ### What Logs Contain **Connection Events** (`event.type: connection`): * TCP connection metadata * Protocol detection (http1, http2, tcp) * TLS version and inspection status * Bytes sent/received * Process information (exe path, user ID) **Artifact Record Events** (`event.type: artifact_record`): * HTTP method, URL, path, status code * User agent and content type * Duration in milliseconds * Process information * Link to stored artifact (OTel or S3 URL depending on object store type) ### What Objects Contain When using OTel object stores (`object_stores` type: `otel`), artifact data is sent as additional log records: * `artifact.type` — `http_request` or `http_response` * `artifact.content_type` — MIME type (e.g., `application/json`) * `artifact.data` — The raw bytes (full headers and body) * `artifact.size_bytes` — Size of the artifact * `artifact.digest` — Content hash for deduplication * `connection.id` — Links back to the connection and artifact record events * `artifact.summary.*` — Request/response metadata (method, host, status, etc.) {% hint style="info" %} **Events vs Objects:** Events (`event_stores`) are lightweight metadata sent for every connection and HTTP transaction. Objects (`object_stores`) are the full payload content, captured based on your `http_capture` plugin level setting (`summary`, `details`, or `full`). {% endhint %} **Example log query:** ``` event.type = "artifact_record" AND event.summary.response_status >= 500 ``` ## Kubernetes Deployment ### Using OpenTelemetry Operator {% hint style="info" %} **Already have OpenTelemetry Operator installed?** Skip to step 2. {% endhint %} 1. Install the OpenTelemetry Operator: ```bash kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml ``` 2. Deploy an OpenTelemetry Collector: ```yaml apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: qtap-collector namespace: monitoring spec: mode: daemonset config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: timeout: 10s exporters: otlphttp: endpoint: "https://your-backend.com/v1/logs" headers: api-key: "${API_KEY}" service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [otlphttp] ``` 3. Deploy qtap using Helm: First, create your qtap configuration file `qtap-config.yaml`: ```yaml version: 2 services: event_stores: - type: otel endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317" protocol: grpc service_name: "qtap" environment: "production" object_stores: - type: otel otel_endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317" protocol: grpc service_name: "qtap" environment: "production" stacks: default_stack: plugins: - type: http_capture config: level: summary format: json tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: default_stack ``` Install qtap with Helm: ```bash # Add Qpoint Helm repository helm repo add qpoint https://helm.qpoint.io/ helm repo update # Install qtap with config helm install qtap qpoint/qtap \ -n qpoint \ --create-namespace \ --set-file config=./qtap-config.yaml \ --set logLevel=warn ``` Verify the deployment: ```bash kubectl get pods -n qpoint kubectl logs -n qpoint -l app.kubernetes.io/name=qtap --tail=50 ``` ## Backend-Specific Configurations ### SaaS Platforms #### Datadog Datadog supports OTLP ingestion directly: ```yaml event_stores: - type: otel endpoint: "https://http-intake.logs.datadoghq.com/v2/logs" protocol: http service_name: "qtap" environment: "production" headers: DD-API-KEY: type: env value: DATADOG_API_KEY tls: enabled: true ``` #### Honeycomb ```yaml event_stores: - type: otel endpoint: "api.honeycomb.io:443" protocol: grpc service_name: "qtap" environment: "production" headers: x-honeycomb-team: type: env value: HONEYCOMB_API_KEY x-honeycomb-dataset: type: text value: "qtap-logs" tls: enabled: true ``` #### New Relic ```yaml event_stores: - type: otel endpoint: "otlp.nr-data.net:4317" protocol: grpc service_name: "qtap" environment: "production" headers: api-key: type: env value: NEW_RELIC_LICENSE_KEY tls: enabled: true ``` #### Grafana Cloud ```yaml event_stores: - type: otel endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443" protocol: grpc service_name: "qtap" environment: "production" headers: authorization: type: env value: GRAFANA_CLOUD_API_KEY # Format: "Basic base64(instanceID:apiKey)" tls: enabled: true ``` #### Elastic ```yaml event_stores: - type: otel endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443" protocol: http service_name: "qtap" environment: "production" headers: Authorization: type: env value: ELASTIC_APM_SECRET_TOKEN tls: enabled: true ``` ### Self-Hosted For self-hosted backends (Jaeger, Zipkin, Grafana Tempo, etc.), point qtap to your OpenTelemetry Collector, then configure the collector to export to your backend: ```yaml # Qtap config - same for all self-hosted backends event_stores: - type: otel endpoint: "otel-collector.monitoring.svc.cluster.local:4317" protocol: grpc ``` Then configure your OTel Collector with the appropriate exporter for your backend. ## Querying and Filtering ### Filter by Event Type ``` # Show only HTTP transaction summaries event.type = "artifact_record" # Show only TCP connections event.type = "connection" ``` ### Find Slow Requests ``` event.type = "artifact_record" AND event.summary.duration_ms > 1000 ``` ### Find Errors ``` # HTTP errors event.type = "artifact_record" AND event.summary.response_status >= 500 ``` ### Track Specific Endpoints ``` # Specific host event.type = "artifact_record" AND event.summary.request_host = "api.example.com" # Specific path pattern event.type = "artifact_record" AND event.summary.request_path LIKE "/api/users/%" ``` ### Monitor HTTP/2 vs HTTP/1 Traffic ``` # HTTP/2 connections (logs) event.type = "connection" AND event.l7Protocol = "http2" # HTTP/1 connections (logs) event.type = "connection" AND event.l7Protocol = "http1" ``` ## Troubleshooting ### No Logs Appearing **Check qtap logs for OTel connection:** ```bash docker logs qtap | grep -i "otel\|error" ``` **Common issues:** 1. **Wrong endpoint** - Verify the endpoint address and port * gRPC default: `4317` * HTTP default: `4318` * With `--network=host`: use `localhost:4317` 2. **TLS mismatch** - If backend requires TLS, set `tls.enabled: true` 3. **Authentication** - Verify headers/API keys are correct 4. **Collector not running** - Check `docker ps | grep otel` ### Connection Refused Errors ``` rpc error: code = Unavailable desc = name resolver error ``` **Solutions:** * Check if OTel Collector is running: `docker ps | grep otel` * Verify network connectivity between qtap and collector * Check firewall rules for port 4317/4318 ### Debugging with stdout Protocol For local debugging, use the `stdout` protocol to see OTLP data printed to qtap logs: ```yaml event_stores: - type: otel protocol: stdout # Prints OTLP data to console service_name: "qtap" environment: "development" ``` Then check qtap logs: ```bash docker logs qtap ``` ## Best Practices 1. **Use TLS in production** - Always enable `tls.enabled: true` for production 2. **Store artifacts securely** - Use OTel object stores for unified pipelines, or S3 for separate storage 3. **Filter appropriately** - Use qtap's filters to avoid capturing unnecessary traffic 4. **Set resource attributes** - Use `service_name` and `environment` for filtering 5. **Monitor qtap itself** - Set up alerts on qtap's Prometheus metrics 6. **Use batch processing** - OTel Collector's batch processor reduces API calls 7. **Use connection.id for correlation** - Link connection events with artifact records using `connection.id` ## Configuration Quick Reference **qtap.yaml:** ```yaml services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc ``` ## Next Steps * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) — OTel and S3 object storage options * [Self-Hosted ClickStack Observability](/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability) — Full setup with OTel events + objects * [Set up Prometheus Metrics](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) for qtap health monitoring * [Configure Traffic Filters](/getting-started/qtap/configuration/traffic-capture-settings) to reduce noise ## Additional Resources * [Storage Configuration Reference](/getting-started/qtap/configuration/storage-configuration) * [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/) * [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/) # Exporting HTTP Headers to OpenTelemetry Qtap can promote specific HTTP headers into OpenTelemetry attributes, making them available for filtering, grouping, and alerting in your observability backend. This is useful for identity headers, custom metadata, or any request/response header you want to query on. ## Why Promote Headers? When qtap captures HTTP traffic, it records standard metadata like method, URL, status code, and duration. But sometimes you need more — the identity of the authenticated user, the originating IP, or a custom tracking header. With header promotion, those values appear as first-class OTel attributes in your backend. Instead of digging through raw payloads, you can: * **Filter dashboards** by authenticated user email * **Group metrics** by upstream service or region * **Alert on specific header values** (e.g., missing auth headers) * **Correlate requests** across services using custom trace headers ## How It Works The `report_usage` plugin accepts a `headers` configuration that specifies which HTTP headers to extract. When qtap observes matching headers in captured traffic, it includes them as `customHeaders` on the reported event. In an OTel backend, these are flattened into dotted attributes. Headers are prefixed by direction: * `req.` — from the HTTP request * `res.` — from the HTTP response In your OTel backend, these appear as `customHeaders.req.` and `customHeaders.res.` due to struct flattening. ## Configuration ### Step 1: Configure an OTel Event Store Add an `otel` event store to your qtap config. If you already have one from the [OpenTelemetry Integration guide](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry), you can use your existing configuration. ```yaml services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false ``` ### Step 2: Add the report\_usage Plugin with Headers Add `report_usage` to your stack with the `headers` list. Each entry is a header name to extract from observed traffic. ```yaml stacks: default: plugins: - type: report_usage config: headers: - Cf-Access-Authenticated-User-Email - X-Forwarded-For - X-Request-Id - Content-Type ``` {% hint style="info" %} Header promotion works with any event store — including `stdout` for local debugging. You don't need an OTel backend to test the feature. {% endhint %} ### Step 3: Configure the Tap Point your tap at the stack containing the `report_usage` plugin: ```yaml tap: direction: egress ignore_loopback: true http: stack: default ``` ### Complete Example Here is a full `qtap.yaml` that captures HTTP traffic and promotes identity headers to OpenTelemetry: ```yaml version: 2 services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - type: stdout stacks: default: plugins: - type: http_metrics - type: access_logs config: mode: summary format: console - type: report_usage config: headers: - Cf-Access-Authenticated-User-Email - X-Forwarded-For - X-Request-Id - Content-Type tap: direction: egress ignore_loopback: true http: stack: default ``` ## What You Get in Your Backend When qtap observes traffic containing the configured headers, they appear as dotted OTLP attributes on log records: ``` customHeaders.req.Cf-Access-Authenticated-User-Email: "user@example.com" customHeaders.req.X-Forwarded-For: "203.0.113.50" customHeaders.req.X-Request-Id: "abc-123-def" customHeaders.res.Content-Type: "application/json" ``` These attributes are queryable in any OTel-compatible backend — filter dashboards, build alerts, or group by header value. ## Datadog Integration If you use Datadog as your observability backend, promoted headers appear as log attributes that you can use in Log Explorer, dashboards, and monitors. ### Viewing Headers in Datadog 1. Open **Log Explorer** in Datadog 2. Search for `@customHeaders.req.Cf-Access-Authenticated-User-Email:*` to find all logs with that header 3. Click a log entry to see the full attribute breakdown under `customHeaders` ### Creating Facets To use promoted headers in dashboard widgets and monitors, create a facet: 1. In Log Explorer, expand a log entry containing the header 2. Click the gear icon next to `@customHeaders.req.Cf-Access-Authenticated-User-Email` 3. Select **Create facet** 4. Give it a display name like "Authenticated User" Once faceted, you can: * **Group by** authenticated user in dashboard widgets * **Filter** logs to a specific user * **Create monitors** that alert when a specific user hits error rates ### Example Datadog Query ``` service:qtap @customHeaders.req.Cf-Access-Authenticated-User-Email:"anil@example.com" @event.summary.response_status:>=500 ``` This finds all 5xx errors for a specific authenticated user. ## Common Headers to Promote ### Identity & Authentication | Header | Description | | ------------------------------------ | ------------------------------------ | | `Cf-Access-Authenticated-User-Email` | Cloudflare Access authenticated user | | `X-Forwarded-For` | Client IP from reverse proxies | | `X-Forwarded-User` | Authenticated user from auth proxies | | `X-Real-Ip` | Original client IP | ### Request Tracking | Header | Description | | ------------------ | ------------------------- | | `X-Request-Id` | Unique request identifier | | `X-Correlation-Id` | Cross-service correlation | | `X-Amzn-Trace-Id` | AWS trace identifier | | `Traceparent` | W3C trace context | ### Application Metadata | Header | Description | | ----------------------- | --------------------- | | `Content-Type` | Response content type | | `X-Cache` | CDN cache hit/miss | | `X-Ratelimit-Remaining` | Rate limit status | {% hint style="warning" %} **Sensitive headers:** Be careful promoting headers like `Authorization`, `Cookie`, or `Set-Cookie`. Their values appear in plaintext in your OTel backend. Only promote sensitive headers if you understand the security implications and your backend has appropriate access controls. {% endhint %} ## Important Notes * **Case-insensitive matching** — Header names are matched case-insensitively. `Cf-Access-Authenticated-User-Email` and `cf-access-authenticated-user-email` both work. * **Only present headers are included** — If a configured header is not present in the actual request or response, it is simply omitted from the OTel output. No empty attributes are created. * **Direction prefixing** — Request headers get the `req.` prefix, response headers get `res.`. This lets you distinguish the same header name appearing in both directions. * **Works with any event store** — While this guide focuses on OTel, header promotion works with any event store including `stdout`. Use `stdout` for local debugging to verify headers are being captured correctly. * **Works with any OTel backend** — When using an OTel event store, promoted headers use standard OTLP attributes, so they work with Datadog, Grafana, Honeycomb, New Relic, and any other OTel-compatible backend. ## Next Steps * [OpenTelemetry Integration](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) — Full OTel setup guide * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) — All available plugins and configuration * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) — Control what traffic qtap observes # Self-Hosted Grafana Observability Stack This guide walks you through setting up a complete, self-hosted observability stack where Qtap events flow to Grafana via Loki and full HTTP payloads stay in your own S3-compatible object storage. Every byte of captured data — metadata and payloads — remains inside your infrastructure. ## Architecture ``` ┌──────────────────────────────────────────────────────────────────┐ │ YOUR INFRASTRUCTURE │ │ │ │ ┌───────┐ │ │ │ Qtap │──── Events (metadata) ───► OTel Collector (:4317) │ │ │ Agent │ │ │ │ │ │ ▼ │ │ │ │ Loki (:3100) │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ Grafana (:3000) │ │ │ │ ▲ │ │ │ │ │ click artifact URL │ │ │ │ ▼ │ │ │ │ nginx proxy (:3904) │ │ │ │ │ │ │ │ │ ▼ │ │ │ │── Objects (payloads) ──► Garage S3 (:3900) │ │ └───────┘ │ └──────────────────────────────────────────────────────────────────┘ ``` | Service | Role | Port | | ------------------ | -------------------------------------------------- | ------------- | | **Qtap** | eBPF agent — captures HTTP traffic at kernel level | Host network | | **OTel Collector** | Receives OTLP logs from Qtap, forwards to Loki | 4317 (gRPC) | | **Loki** | Log aggregation and storage | 3100 | | **Grafana** | Query, explore, and visualize events | 3000 | | **Garage** | S3-compatible object storage for HTTP payloads | 3900 (S3 API) | | **nginx** | Proxy for anonymous read access to stored objects | 3904 | {% hint style="info" %} This guide covers **events and object linking**. For sending events to any OpenTelemetry-compatible backend, see the [OpenTelemetry Integration](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) guide. {% endhint %} ## Prerequisites * Docker and Docker Compose * Linux kernel 5.10+ with eBPF support * `aws` CLI (for S3 verification) — [install guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ## Understanding the Two Data Paths Qtap produces two distinct outputs. Understanding the split is key to this architecture: | | Events | Objects | | --------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | **What** | Lightweight metadata — method, URL, status, duration, process | HTTP transaction objects — metadata at `summary` level; headers and bodies at `full` level | | **Sensitivity** | Low — safe to send anywhere | Varies — `summary` objects contain only metadata; `full` objects may contain API keys, tokens, PII | | **Storage** | Loki (via OTel Collector) | Garage S3 (your infrastructure) | | **Volume** | Every observed request | Every captured request (content varies by capture level) | **The link between them:** Qtap's `access_url` template embeds a clickable URL into each `artifact_record` event. When you find an interesting event in Grafana, you click the URL to fetch the complete HTTP transaction from your S3 storage. ``` access_url: http://localhost:3904/qpoint/{{DIGEST}} ^^^^^^^^ Replaced with SHA1 hash of the stored object ``` ## Configuration Files Create a project directory and add these files: ```bash mkdir -p grafana-stack/grafana/provisioning/datasources cd grafana-stack ``` ### OTel Collector {% code overflow="wrap" %} ```bash cat > otel-collector.yaml << 'EOF' receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: timeout: 10s exporters: otlphttp/loki: endpoint: http://localhost:3100/otlp tls: insecure: true service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [otlphttp/loki] EOF ``` {% endcode %} The collector listens for gRPC on port 4317 (where Qtap sends events) and forwards them to Loki's OTLP endpoint. Since the collector runs with `network_mode: host`, it reaches Loki at `localhost:3100`. ### Loki {% code overflow="wrap" %} ```bash cat > loki.yaml << 'EOF' auth_enabled: false server: http_listen_port: 3100 common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: "2024-01-01" store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h limits_config: retention_period: 168h allow_structured_metadata: true max_query_series: 100000 compactor: working_directory: /loki/compactor delete_request_store: filesystem compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 EOF ``` {% endcode %} Key settings: `allow_structured_metadata: true` lets Loki store Qtap's structured attributes (method, status, host, etc.) as queryable fields. Retention is set to 7 days (`168h`). ### Garage (S3-Compatible Object Storage) {% code overflow="wrap" %} ```bash cat > garage.toml << 'EOF' metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" db_engine = "sqlite" replication_factor = 1 rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" rpc_secret = "c052485a056fabf3c0832d98f63b14d58036f5189683fc39da199f43fde3f15e" [s3_api] s3_region = "us-east-1" api_bind_addr = "[::]:3900" [s3_web] bind_addr = "[::]:3902" root_domain = ".web.garage.localhost" [admin] api_bind_addr = "[::]:3903" admin_token = "demo-admin-token-for-local-use-only" EOF ``` {% endcode %} Garage provides the S3 API on port 3900 (where Qtap writes payloads), a web endpoint on 3902 (for anonymous reads), and an admin API on 3903 (for bucket management). ### Nginx Proxy {% code overflow="wrap" %} ```bash cat > nginx.conf << 'EOF' server { listen 3904; location /qpoint/ { rewrite ^/qpoint/(.*)$ /$1 break; proxy_pass http://garage:3902; proxy_set_header Host qpoint.web.garage.localhost; proxy_set_header X-Real-IP $remote_addr; } } EOF ``` {% endcode %} The nginx proxy translates requests like `http://localhost:3904/qpoint/` into Garage web requests with the correct virtual-host `Host` header. This avoids needing wildcard DNS for Garage's subdomain-based routing. ### Grafana Datasource {% code overflow="wrap" %} ```bash cat > grafana/provisioning/datasources/datasources.yaml << 'EOF' apiVersion: 1 datasources: - name: Loki type: loki uid: loki access: proxy url: http://loki:3100 isDefault: true editable: true EOF ``` {% endcode %} ### Qtap {% code overflow="wrap" %} ```bash cat > qtap.yaml << 'EOF' version: 2 services: event_stores: - type: otel endpoint: "localhost:4317" # OTel Collector gRPC protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - id: garage type: s3 endpoint: localhost:3900 # Garage S3 API bucket: qpoint region: us-east-1 access_url: http://localhost:3904/qpoint/{{DIGEST}} insecure: true access_key: type: env value: GARAGE_ACCESS_KEY secret_key: type: env value: GARAGE_SECRET_KEY rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 stacks: default_stack: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) rules: - name: "Full capture on errors" expr: is_error() level: full # Stores headers + bodies in S3 tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) audit_include_dns: false # (true|false) http: stack: default_stack filters: groups: - qpoint # Don't capture Qtap's own traffic EOF ``` {% endcode %} This configuration captures all egress HTTP traffic at `summary` level (metadata only — no headers or bodies), and automatically escalates to `full` capture for any 4xx or 5xx response. Full captures store complete request/response headers in Garage S3. Both levels emit `artifact_record` events with clickable URLs pointing to the stored objects. ## Docker Compose {% code overflow="wrap" %} ```bash cat > docker-compose.yaml << 'EOF' services: loki: image: grafana/loki:latest container_name: loki restart: always ports: - "3100:3100" volumes: - ./loki.yaml:/etc/loki/local-config.yaml:ro - loki-data:/loki command: -config.file=/etc/loki/local-config.yaml otel-collector: image: otel/opentelemetry-collector:latest container_name: otel-collector restart: always network_mode: host volumes: - ./otel-collector.yaml:/etc/otel-collector-config.yaml:ro command: ["--config=/etc/otel-collector-config.yaml"] garage: image: dxflrs/garage:v2.1.0 container_name: garage restart: always ports: - "3900:3900" - "3902:3902" - "3903:3903" volumes: - ./garage.toml:/etc/garage.toml:ro garage-web: image: nginx:alpine container_name: garage-web restart: always ports: - "3904:3904" volumes: - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro depends_on: - garage grafana: image: grafana/grafana:latest container_name: grafana restart: always ports: - "3000:3000" volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - grafana-data:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=admin - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Editor volumes: loki-data: grafana-data: EOF ``` {% endcode %} {% hint style="info" %} The OTel Collector uses `network_mode: host` so that Qtap (running on the host) can reach it at `localhost:4317`. Because it shares the host network stack, it also reaches Loki at `localhost:3100`. {% endhint %} ## Running the Stack ### Step 1: Start the Services ```bash docker compose up -d ``` Verify all containers are running: ```bash docker compose ps ``` ### Step 2: Initialize Garage Wait for Garage to be ready, then configure the cluster layout, create a bucket, and set up access credentials: {% code overflow="wrap" %} ```bash # Wait for Garage to start sleep 5 # Get the node ID NODE_ID=$(docker exec garage /garage status 2>/dev/null | grep -oE '[a-f0-9]{16}' | head -1) echo "Node ID: $NODE_ID" # Configure cluster layout docker exec garage /garage layout assign -z dc1 -c 1G "$NODE_ID" docker exec garage /garage layout apply --version 1 # Create the bucket docker exec garage /garage bucket create qpoint # Import S3 credentials docker exec garage /garage key import \ --yes \ -n qtap-key \ "GK0123456789abcdef01234567" \ "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" # Grant read/write access docker exec garage /garage bucket allow \ --read --write --owner qpoint --key qtap-key # Enable anonymous web reads (for the nginx proxy) docker exec garage /garage bucket website --allow qpoint ``` {% endcode %} {% hint style="warning" %} The credentials above are for local development only. For production, generate unique keys and manage them securely. {% endhint %} ### Step 3: Start Qtap {% code overflow="wrap" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ -e GARAGE_ACCESS_KEY=GK0123456789abcdef01234567 \ -e GARAGE_SECRET_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" ``` {% endcode %} ### Step 4: Wait for Initialization ```bash sleep 6 ``` Qtap needs a few seconds to load eBPF programs and start capturing. ### Step 5: Generate Test Traffic Generate both a successful request and an error to see both data paths: ```bash # Successful request — captured at summary level (metadata only, no headers/bodies) docker run --rm curlimages/curl -s https://httpbin.org/get > /dev/null # Error request — captured at full level (headers + bodies stored in S3) docker run --rm curlimages/curl -s https://httpbin.org/status/500 > /dev/null ``` ### Step 6: Verify Data Is Flowing **Check OTel Collector is receiving data:** {% code overflow="wrap" %} ```bash docker logs otel-collector 2>&1 | tail -5 ``` {% endcode %} You should see log export messages. **Check Loki has events:** {% code overflow="wrap" %} ```bash curl -s "http://localhost:3100/loki/api/v1/query_range?query=%7Bservice_name%3D%22qtap%22%7D&limit=5" | python3 -m json.tool | head -20 ``` {% endcode %} **Check S3 has objects (from the error request):** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID=GK0123456789abcdef01234567 \ AWS_SECRET_ACCESS_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ aws --endpoint-url http://localhost:3900 \ --region us-east-1 \ s3 ls s3://qpoint/ ``` {% endcode %} ## Exploring Data in Grafana Open in your browser (default credentials: `admin` / `admin`). ### Viewing Events 1. Navigate to **Explore** (compass icon in the left sidebar) 2. Select **Loki** as the datasource 3. Enter a LogQL query: ``` {service_name="qtap"} ``` 4. Click **Run query** You should see Qtap events — both `connection` events (TCP connection metadata) and `artifact_record` events (HTTP transaction summaries, with full headers and bodies for requests matching capture rules). ### Useful LogQL Queries | Query | Description | | --------------------------------------------------------------- | -------------------------------------------- | | `{service_name="qtap"}` | All Qtap events | | `{service_name="qtap"} \| json \| event_type="artifact_record"` | Only artifact records (objects stored in S3) | | `{service_name="qtap"} \| json \| response_status >= 400` | Error responses | | `{service_name="qtap"} \| json \| request_host="httpbin.org"` | Traffic to a specific host | | `{service_name="qtap"} \| json \| duration_ms > 1000` | Slow requests (> 1s) | | `{service_name="qtap"} \| json \| process_exe="/usr/bin/curl"` | Requests from curl | | `{service_name="qtap"} \| json \| direction="egress-external"` | External egress traffic | ### Expanding Log Entries Click on any log entry to expand it. You'll see structured attributes including: * `request_method`, `request_host`, `request_scheme` * `response_status`, `duration_ms` * `process_exe`, `direction` * For artifact records: `digest`, `url`, `type` ## Object Linking — From Events to Full Payloads This is the key capability of this stack: linking lightweight events in Grafana to complete HTTP transactions stored in your own S3. ### How It Works 1. Qtap captures a request — all capture levels store an object in Garage S3, keyed by its SHA1 digest. In our config, errors get `full` capture (with headers), while other traffic gets `summary` (metadata only) 2. The HTTP transaction object is stored in Garage S3 as JSON 3. Qtap emits an `artifact_record` event to the OTel Collector, which includes a `url` field pointing to the stored object 4. The event flows through to Loki and appears in Grafana 5. You click the URL to view the complete HTTP transaction ### Walkthrough **1. Find an error event in Grafana** In Explore, query for artifact records: ``` {service_name="qtap"} | json | event_type="artifact_record" ``` **2. Expand the log entry** Click on an artifact record event. Look for these fields: ```json { "type": "http_transaction", "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c", "url": "http://localhost:3904/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c", "summary": { "request_method": "GET", "request_host": "httpbin.org", "response_status": 500, "duration_ms": 580, "process_exe": "/usr/bin/curl", "direction": "egress-external" } } ``` **3. Click the URL** The `url` field is a direct link to the stored object. Click it (or open it in a new tab) to see the complete HTTP transaction: ```json { "metadata": { "process_exe": "/usr/bin/curl", "container_name": "relaxed_zhukovsky", "container_image": "curlimages/curl", "bytes_sent": 72, "bytes_received": 127, "connection_id": "d654d887p3qnd27vfhag", "endpoint_id": "httpbin.org" }, "request": { "method": "GET", "url": "https://httpbin.org/status/500", "scheme": "https", "authority": "httpbin.org", "protocol": "http2", "user_agent": "curl/8.18.0", "headers": { ":authority": "httpbin.org", ":method": "GET", ":path": "/status/500", "Accept": "*/*", "User-Agent": "curl/8.18.0" } }, "response": { "status": 500, "content_type": "text/html; charset=utf-8", "headers": { "Content-Type": "text/html; charset=utf-8", "Server": "gunicorn/19.9.0" } }, "duration_ms": 767, "direction": "egress-external" } ``` This is the full HTTP transaction — headers and metadata — stored entirely in your infrastructure. {% hint style="info" %} All capture levels generate `artifact_record` events and store objects in S3. The difference is content: `summary` objects contain only metadata (method, URL, status, duration), while `headers` and `full` objects include the complete request/response headers and bodies. The object linking walkthrough above is most useful for `full` captures where you can inspect the actual HTTP payload. {% endhint %} ### The `access_url` Template The link between events and objects is configured in the Qtap `object_stores` section: ```yaml access_url: http://localhost:3904/qpoint/{{DIGEST}} ``` `{{DIGEST}}` is replaced with the SHA1 hash of the stored object. The resulting URL is embedded in every `artifact_record` event. For production, replace `localhost` with the hostname or IP address that Grafana users can reach: ```yaml access_url: https://objects.internal.example.com/qpoint/{{DIGEST}} ``` ## Cleanup ```bash docker compose down -v docker rm -f qtap ``` ## Next Steps * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) — S3, MinIO, AWS, GCS options * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) — Rulekit rules, capture levels * [Prometheus + Grafana Monitoring](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) — Add metrics dashboards alongside logs * [OpenTelemetry Integration](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry) — Send events to any OTLP backend * [Qpoint Data Schema Reference](/appendix/qpoint-data-schema-reference) — Full event schema documentation # Self-Hosted ClickStack Observability **A fully self-hosted observability stack in a single container.** This guide walks you through setting up Qtap with [ClickStack](https://clickhouse.com/clickstack) — a self-contained observability stack that bundles ClickHouse, an OTel collector, and the HyperDX UI into a single Docker image. Events and artifacts flow through OpenTelemetry, with no S3 setup required. {% hint style="success" %} **Quick version:** Run ClickStack, point Qtap at it, see traffic in your browser. Total setup time: \~5 minutes. {% endhint %} *** ## What You'll Get | Component | What It Does | Port | | -------------- | ------------------------------------------------- | ---------------- | | **ClickStack** | ClickHouse + OTel collector + HyperDX UI | 8080, 4317, 4318 | | **Qtap** | eBPF agent capturing traffic and sending via OTel | — | Both events (connection metadata, HTTP transactions) and objects (request/response headers and bodies) are sent through the same OTel endpoint. No separate object storage to configure. *** ## Prerequisites * Docker installed * Linux host with kernel 5.8+ (for eBPF) * Root or `CAP_BPF` + `CAP_SYS_ADMIN` capabilities *** ## Step 1: Start ClickStack ```bash docker run -d --name clickstack \ -p 8080:8080 \ -p 4317:4317 \ -p 4318:4318 \ -e BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true \ clickhouse/clickstack-local:latest ``` Wait for it to be ready: ```bash docker logs -f clickstack ``` Once you see the startup complete, visit to access the HyperDX UI. *** ## Step 2: Create the Qtap Configuration Create a file called `qtap-clickstack.yaml`: ```yaml version: 2 services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false stacks: basic_reporting: plugins: - type: http_capture config: level: full format: json - type: report_usage tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: basic_reporting ``` This configuration: * Sends **events** (connection metadata, HTTP transaction summaries) to ClickStack via OTel * Sends **objects** (full request/response headers and bodies) to ClickStack via OTel * Captures **all egress HTTP traffic** at full detail level * Both stores share the same OTel endpoint — no S3 credentials needed {% hint style="info" %} This replaces many small S3 files with batched OTLP log records, which ClickHouse ingests efficiently. {% endhint %} *** ## Step 3: Start Qtap ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "$(pwd)/qtap-clickstack.yaml:/app/config/qtap.yaml" \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info \ --log-encoding=console \ --config="/app/config/qtap.yaml" ``` Verify Qtap is running and connected: ```bash docker logs qtap 2>&1 | tail -20 ``` You should see the eventstore and objectstore initialize successfully: ``` INFO eventstore.otel initialized INFO objectstore.otel initialized ``` *** ## Step 4: Generate Traffic and View Generate some traffic: ```bash curl https://httpbin.org/get curl https://api.github.com curl -X POST https://httpbin.org/post -d '{"test": "data"}' ``` Open in your browser. In the HyperDX UI, create a source with table name `otel_logs` to view captured events and artifacts. You'll see: * **Connection events** — TCP connections with protocol detection, TLS info, process attribution * **HTTP transaction summaries** — method, URL, status, duration, content type * **Full request/response data** — headers and bodies captured by the `http_capture` plugin *** ## How It Works ``` Your Applications │ │ (HTTP/HTTPS traffic) ▼ Qtap (eBPF) │ ├── Events ──→ OTel (gRPC :4317) ──→ ClickStack ──→ ClickHouse │ └── Objects ──→ OTel (gRPC :4317) ──→ ClickStack ──→ ClickHouse ``` Qtap captures traffic using eBPF and exports both event metadata and full artifacts as OpenTelemetry Logs. ClickStack receives them through its built-in OTel collector and stores everything in ClickHouse. *** ## Configuration Options ### Capture Level Control how much detail is captured per request: | Level | What's Captured | | --------- | ---------------------------------- | | `none` | Connection metadata only | | `summary` | HTTP method, URL, status, duration | | `details` | Summary + headers | | `full` | Summary + headers + bodies | Change the level in the `http_capture` plugin: ```yaml plugins: - type: http_capture config: level: summary # Less data, lower storage format: json ``` ### Traffic Direction Control which traffic is captured: | Direction | What's Captured | | ----------------- | -------------------------- | | `egress` | All outbound traffic | | `egress-external` | Only external destinations | | `egress-internal` | Only internal destinations | *** ## Cleanup ```bash docker rm -f qtap clickstack ``` *** ## Next Steps * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) — Compare OTel vs S3 object storage options * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) — Filter and transform captured traffic * [OTel Object Store Release Notes](/release-notes/otel-object-store) — Feature details * [Sample Configuration](https://github.com/qpoint-io/qtap/blob/main/examples/sample-otel-objectstore-clickstack.yaml) — Reference config on GitHub # Integrating Qtap with Your Observability Stack You already have an observability stack — Grafana + Loki, Datadog, Elastic, or similar. This guide shows how Qtap slots into what you already run, without provisioning new infrastructure. Qtap speaks OTLP for both events and artifacts. You can also use S3 for artifact storage if you prefer. If your stack can receive OTLP, you're set. {% hint style="info" %} **Choose your path:** * **Simplest setup:** Send both events and artifacts through OTel → [Self-Hosted ClickStack Observability](/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability) * **Full Grafana stack from scratch:** → [Self-Hosted Grafana Observability Stack](/guides/qtap-guides/observability-and-integration/self-hosted-grafana-observability-stack) * **Integrate with existing infrastructure:** Keep reading below {% endhint %} ## How Qtap Fits In ``` ┌──────────────────────────────────────────────────────────────────┐ │ YOUR INFRASTRUCTURE │ │ │ │ ┌───────┐ │ │ │ Qtap │──── Events (OTLP) ──────► OTel Collector │ │ │ Agent │ │ │ │ │ │ ▼ │ │ │ │ Your Log Backend │ │ │ │ (Loki, Datadog, Elastic, …) │ │ │ │ │ │ │ │── Objects ──► OTel (same collector) │ │ │ │ OR ► S3 Storage (MinIO, AWS S3, GCS, …) │ │ └───────┘ │ └──────────────────────────────────────────────────────────────────┘ ``` Qtap produces two data streams: events and objects. Both can flow through your OTel pipeline, or you can send objects to S3 storage separately. When using S3, the `access_url` in each `artifact_record` event links back to the stored object. ## Two Data Streams | | Events | Objects | | --------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | **What** | Lightweight metadata — method, URL, status, duration, process | HTTP transaction content — metadata at `summary` level; headers and bodies at `full` level | | **Sensitivity** | Low — safe to send to any backend | Varies — `summary` objects contain only metadata; `full` objects may contain API keys, tokens, PII | | **Protocol** | OTLP (gRPC or HTTP) | OTLP (gRPC or HTTP) **or** S3 API | | **Destination** | Any OTLP-compatible backend via OTel Collector | Same OTel Collector **or** any S3-compatible storage | | **Volume** | Every observed request | Every captured request (content varies by capture level) | **Choose your object storage approach:** * **OTel objects** — Simplest. Events and artifacts flow through the same collector. No S3 credentials needed. Best for ClickStack, self-contained setups, or when you don't need separate object storage. * **S3 objects** — Events go to OTel, artifacts go to S3. Each `artifact_record` event contains a clickable `url` linking to the stored object. Best when you need durable, queryable object storage or have existing S3 infrastructure. ## Configuring Qtap ### Option 1: Everything through OTel (simplest) ```yaml version: 2 services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" environment: "production" tls: enabled: false stacks: default_stack: plugins: - type: http_capture config: level: full format: json tap: direction: egress ignore_loopback: true audit_include_dns: false http: stack: default_stack ``` Both events and artifacts go to the same OTel endpoint. No S3 credentials, no bucket setup. ### Option 2: OTel events + S3 objects One complete `qtap.yaml` that connects to your existing OTel Collector and S3 storage: ```yaml version: 2 services: event_stores: - type: otel endpoint: "localhost:4317" # Your OTel Collector gRPC endpoint protocol: grpc # (grpc|http|stdout) service_name: "qtap" # Service name in your log backend environment: "production" # Environment tag for filtering tls: enabled: false # Set true for production / remote collectors object_stores: - id: s3 type: s3 endpoint: minio.internal:9000 # Your S3-compatible storage endpoint bucket: qpoint # Your bucket name region: us-east-1 # Your region access_url: https://objects.internal.example.com/qpoint/{{DIGEST}} insecure: false # Set true for HTTP-only endpoints access_key: type: env value: S3_ACCESS_KEY # Environment variable name secret_key: type: env value: S3_SECRET_KEY # Environment variable name rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 stacks: default_stack: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) format: json # (json|text) rules: - name: "Full capture on errors" expr: is_error() level: full # Stores headers + bodies in S3 tap: direction: egress # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # (true|false) audit_include_dns: false # (true|false) http: stack: default_stack filters: groups: - qpoint # Don't capture Qtap's own traffic ``` {% hint style="warning" %} **Replace these values with yours:** {% endhint %} | Placeholder | What to set | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `endpoint` (event\_stores) | Your OTel Collector address (e.g., `otel-collector:4317`) | | `endpoint` (object\_stores) | Your S3 storage address (e.g., `s3.amazonaws.com`) | | `bucket` | Your S3 bucket name | | `region` | Your S3 region | | `access_url` | URL template that your dashboard users can reach (see [Connecting to Your S3 Storage](#connecting-to-your-s3-storage)) | | `S3_ACCESS_KEY` / `S3_SECRET_KEY` | Environment variable names holding your S3 credentials | ## Connecting to Your OTel Collector If you already have an OTel Collector running, point Qtap's `event_stores` endpoint at it — no other changes needed. Qtap sends OTLP logs on the standard gRPC port (4317). If your collector doesn't yet forward to a log backend, add the relevant exporter. For example, to forward to Loki: ```yaml # Add to your existing OTel Collector config exporters: otlphttp/loki: endpoint: http://loki:3100/otlp tls: insecure: true service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [otlphttp/loki] # Add alongside your existing exporters ``` This works with any OTLP-compatible backend — replace the exporter with whatever your stack uses (Datadog, Elastic, Honeycomb, etc.). See [Backend-Specific Configurations](/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry#backend-specific-configurations) for detailed examples. ## Connecting to Your S3 Storage Qtap's S3 object store works with any S3-compatible storage. Set the `endpoint` to match your provider: | Provider | Endpoint Example | `access_url` Example | | ------------------------ | ------------------------ | -------------------------------------------------------- | | **MinIO** | `minio.internal:9000` | `https://minio.internal/{{BUCKET}}/{{DIGEST}}` | | **AWS S3** | `s3.amazonaws.com` | `https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}}` | | **Google Cloud Storage** | `storage.googleapis.com` | `https://storage.cloud.google.com/{{BUCKET}}/{{DIGEST}}` | | **Garage** | `garage.internal:3900` | `http://garage.internal:3902/{{BUCKET}}/{{DIGEST}}` | ### The `access_url` Template The `access_url` determines the clickable link embedded in every `artifact_record` event. It supports three template variables: * `{{DIGEST}}` — SHA1 hash of the stored object (always required) * `{{BUCKET}}` — The bucket name from your config * `{{ENDPOINT}}` — The S3 endpoint from your config The URL must be reachable by whoever is reading your logs. For internal dashboards, use an internal hostname. For shared access, consider: * **Presigned URLs** — Generate time-limited signed URLs for private buckets * **Public bucket** — Allow anonymous reads (development only) * **Reverse proxy** — Place nginx or similar in front of your S3 for access control ## What You'll See Qtap stores an HTTP transaction object in S3 and emits an `artifact_record` event for every captured request. At `summary` level, the object contains only metadata. At `headers` or `full` level, it includes the complete request/response headers and bodies. ### The `artifact_record` Event Search your log backend for `event_type="artifact_record"` to find stored objects: ```json { "type": "http_transaction", "digest": "35a712233f2a70e4842d83eb017f952ae09bf74c", "url": "https://objects.internal.example.com/qpoint/35a712233f2a70e4842d83eb017f952ae09bf74c", "summary": { "request_method": "GET", "request_host": "httpbin.org", "response_status": 500, "duration_ms": 580, "process_exe": "/usr/bin/curl", "direction": "egress-external" } } ``` ### The Stored Object Click the `url` to fetch the full HTTP transaction from your S3 storage: ```json { "metadata": { "process_exe": "/usr/bin/curl", "container_name": "relaxed_zhukovsky", "container_image": "curlimages/curl", "bytes_sent": 72, "bytes_received": 127, "connection_id": "d654d887p3qnd27vfhag", "endpoint_id": "httpbin.org" }, "request": { "method": "GET", "url": "https://httpbin.org/status/500", "scheme": "https", "authority": "httpbin.org", "protocol": "http2", "user_agent": "curl/8.18.0", "headers": { ":authority": "httpbin.org", ":method": "GET", ":path": "/status/500", "Accept": "*/*", "User-Agent": "curl/8.18.0" } }, "response": { "status": 500, "content_type": "text/html; charset=utf-8", "headers": { "Content-Type": "text/html; charset=utf-8", "Server": "gunicorn/19.9.0" } }, "duration_ms": 767, "direction": "egress-external" } ``` This works in any log UI — search for `artifact_record` events, click the URL, and you have the full request and response. {% hint style="info" %} All capture levels generate `artifact_record` events and store objects in S3. The difference is content: `summary` objects contain only metadata (method, URL, status), while `headers` and `full` objects include the complete request/response headers and bodies. {% endhint %} ## Common Configurations These snippets show just the `services:` block — the rest of the config (rulekit, stacks, tap) stays the same as the [full example above](#configuring-qtap). ### ClickStack (all OTel) ```yaml services: event_stores: - type: otel endpoint: "localhost:4317" protocol: grpc service_name: "qtap" tls: enabled: false object_stores: - type: otel otel_endpoint: "localhost:4317" protocol: grpc service_name: "qtap" tls: enabled: false ``` See [Self-Hosted ClickStack Observability](/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability) for the full walkthrough. ### Loki + MinIO ```yaml services: event_stores: - type: otel endpoint: "otel-collector:4317" protocol: grpc service_name: "qtap" tls: enabled: false object_stores: - type: s3 endpoint: minio.internal:9000 bucket: qpoint region: us-east-1 access_url: https://minio.internal/qpoint/{{DIGEST}} insecure: false access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` ### Datadog + AWS S3 ```yaml services: event_stores: - type: otel endpoint: "otel-collector:4317" protocol: grpc service_name: "qtap" tls: enabled: false object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: my-company-qpoint region: us-west-2 access_url: https://s3.amazonaws.com/my-company-qpoint/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY ``` ### Elastic + Google Cloud Storage ```yaml services: event_stores: - type: otel endpoint: "otel-collector:4317" protocol: grpc service_name: "qtap" tls: enabled: false object_stores: - type: s3 endpoint: storage.googleapis.com bucket: my-company-qpoint region: us-central1 access_url: https://storage.cloud.google.com/my-company-qpoint/{{DIGEST}} insecure: false access_key: type: env value: GCS_ACCESS_KEY secret_key: type: env value: GCS_SECRET_KEY ``` ## Verification Checklist 1. **Check OTel Collector** — Look for received log entries in your collector's output 2. **Query your log backend** — Search for `service_name="qtap"` events 3. **List S3 objects** — Verify objects appear in your bucket after generating traffic 4. **Test object linking** — Find an `artifact_record` event and open the URL to confirm the full HTTP transaction is accessible ## Next Steps * [Self-Hosted ClickStack Observability](/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability) — Simplest self-hosted setup (everything through OTel) * [Self-Hosted Grafana Observability Stack](/guides/qtap-guides/observability-and-integration/self-hosted-grafana-observability-stack) — Build the full Grafana stack from scratch * [Storage Configuration](/getting-started/qtap/configuration/storage-configuration) — OTel and S3 object storage options, credential management * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) — Rulekit rules, capture levels, conditional logic * [Qpoint Data Schema Reference](/appendix/qpoint-data-schema-reference) — Full event schema documentation # Fluent Bit Batching ## Capturing All HTTP Traffic with Fluent Bit ### Overview This guide shows you how to capture **every HTTP request and response** from Qtap and route it to storage backends using [Fluent Bit](https://fluentbit.io/). This pattern is ideal for comprehensive observability, troubleshooting, audit trails, and compliance requirements where you need complete traffic capture. #### Why Fluent Bit? While Qtap can write directly to S3-compatible object stores, **Fluent Bit provides critical advantages at scale**: * **Performance**: Batches and buffers writes to reduce API calls * **Reliability**: Built-in retry logic, buffering, and backpressure handling * **Flexibility**: Route traffic to multiple destinations (S3, CloudWatch, Elasticsearch, etc.) * **Filtering**: Apply additional filtering, enrichment, or transformation * **Cost optimization**: Batch uploads reduce S3 API costs significantly #### Architecture ``` ┌──────────┐ forward ┌──────────────┐ parse ┌──────────┐ │ Qtap │──────────▸│ Fluent Bit │─────────▸│ Filter & │ │ (eBPF) │ port │ (Batching) │ │ Route │ └──────────┘ 24224 └──────────────┘ └────┬─────┘ │ ┌──────────────┴──────────────┐ ▼ ▼ ┌──────────┐ ┌──────────┐ │ S3 │ │ Stdout / │ │ (MinIO, │ │ Other │ │ AWS, GCS)│ └──────────┘ └──────────┘ ``` **How it works:** 1. Qtap captures HTTP traffic using eBPF (TLS inspection, no proxies) 2. Qtap writes HTTP transaction objects as structured logs 3. Docker forwards logs to Fluent Bit using the fluentd log driver 4. Fluent Bit parses JSON, filters, and tags HTTP transactions 5. Fluent Bit batches, buffers, and routes to storage backends 6. Set TTL policies on storage (recommended: 90 days) ### Docker Deployment This setup uses Docker's fluentd log driver to forward logs directly to Fluent Bit over the network. This approach is simpler and more reliable than tailing log files. #### Step 1: Create Qtap Configuration Create `qtap-config.yaml`: ```yaml version: 2 services: event_stores: - type: stdout # Connection metadata object_stores: - type: stdout # Full HTTP payloads stacks: capture_all: plugins: - type: http_capture config: level: full # (none|summary|headers|full) - Capture headers + bodies format: json # (json|text) - JSON for Fluent Bit parsing tap: direction: egress # egress | egress-external | egress-internal | ingress | all ignore_loopback: true # Skip localhost traffic audit_include_dns: false # Skip DNS queries http: stack: capture_all ``` #### Step 2: Create Fluent Bit Configuration Create `fluent-bit.conf`: ```conf [SERVICE] Flush 5 Daemon Off Log_Level info Parsers_File parsers.conf # Input: Receive logs via forward protocol [INPUT] Name forward Listen 0.0.0.0 Port 24224 # Filter: Only keep HTTP transactions (lines with metadata field) [FILTER] Name grep Match docker.qtap # Matches the logging tag configured in docker-compose Regex log .*"metadata":.* # Filter: Parse JSON logs from Qtap (runs after grep so the log field still exists) [FILTER] Name parser Match docker.qtap Key_Name log Parser generic_json_parser Reserve_Data On Preserve_Key Off # Output: Write to stdout for testing [OUTPUT] Name stdout Match docker.qtap Format json_lines ``` {% hint style="info" %} **How the filters work:** * The **grep** filter runs **before** parsing so it can match the raw `log` field while it still exists * The **parser** filter then expands the JSON payload and removes the original `log` key * This sequencing ensures only HTTP transaction data reaches downstream outputs {% endhint %} Create `parsers.conf`: ```conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name generic_json_parser Format json ``` #### Step 3: Create Docker Compose Create `docker-compose.yaml`: ```yaml version: '3' services: fluent-bit: image: fluent/fluent-bit:latest container_name: fluent-bit network_mode: host volumes: - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf - ./parsers.conf:/fluent-bit/etc/parsers.conf command: ["fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.conf"] qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap depends_on: - fluent-bit privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - ./qtap-config.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 logging: driver: fluentd options: tag: docker.qtap # Must match the Fluent Bit Match pattern fluentd-address: 127.0.0.1:24224 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml ``` #### Step 4: Start and Validate ```bash # Start services docker compose up -d # Wait a few seconds for eBPF initialization sleep 5 # Generate test traffic docker run --rm curlimages/curl -s https://httpbin.org/get # View captured HTTP traffic docker logs fluent-bit | grep 'metadata' ``` **Expected output** (one line per HTTP transaction): ```json { "metadata": { "process_id": "108520", "process_exe": "/usr/bin/curl", "bytes_sent": 41, "bytes_received": 395, "connection_id": "d3shhpg7p3qm85psn2rg", "endpoint_id": "httpbin.org" }, "request": { "method": "GET", "url": "https://httpbin.org/get", "scheme": "https", "path": "/get", "authority": "httpbin.org", "protocol": "http2", "request_id": "d3shhpg7p3qm85psn2s0", "user_agent": "curl/8.12.1", "headers": { ":authority": "httpbin.org", ":method": "GET", ":path": "/get", ":scheme": "https", "Accept": "*/*", "User-Agent": "curl/8.12.1" } }, "response": { "status": 200, "content_type": "application/json", "headers": { ":status": "200", "Access-Control-Allow-Credentials": "true", "Access-Control-Allow-Origin": "*", "Content-Length": "255", "Content-Type": "application/json", "Date": "Wed, 22 Oct 2025 17:48:54 GMT", "Server": "gunicorn/19.9.0" }, "body": "ewogICJhcmdzIjoge30sIAogICJoZWFkZXJzIjogewogICAgIkFjY2VwdCI6ICIqLyoiLCAKICAgICJIb3N0IjogImh0dHBiaW4ub3JnIiwgCiAgICAiVXNlci1BZ2VudCI6ICJjdXJsLzguMTIuMSIsIAogICAgIlgtQW16bi1UcmFjZS1JZCI6ICJSb290PTEtNjhmOTE4ZTYtNmRiZGJlZDA0ZDllMjNhOTU1NjQ0YmEyIgogIH0sIAogICJvcmlnaW4iOiAiNzMuNzEuMTM4LjEwOCIsIAogICJ1cmwiOiAiaHR0cHM6Ly9odHRwYmluLm9yZy9nZXQiCn0K" }, "transaction_time": "2025-10-22T17:48:54.114375279Z", "duration_ms": 31205, "direction": "egress-external", "container_id": "49c2e0af4ffbbfff89be4766c6f84d7a3ad8e528d88fafeb3010facdb6374fb7", "container_name": "/qtap", "source": "stdout" } ``` {% hint style="info" %} Response bodies are base64 encoded in the `body` field. {% endhint %} ### Production Outputs #### AWS S3 Replace the stdout output in `fluent-bit.conf` with: ```conf [OUTPUT] Name s3 Match docker.qtap bucket your-bucket-name region us-east-1 total_file_size 50M upload_timeout 10m compression gzip s3_key_format /qtap/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz store_dir /tmp/fluent-bit/s3 use_put_object On ``` Add environment variables to the fluent-bit service in docker-compose: ```yaml environment: - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} ``` **Set lifecycle policy:** ```bash aws s3api put-bucket-lifecycle-configuration \ --bucket your-bucket-name \ --lifecycle-configuration '{ "Rules": [{ "Id": "DeleteAfter90Days", "Status": "Enabled", "Prefix": "qtap/", "Expiration": {"Days": 90} }] }' ``` #### MinIO (S3-Compatible) ```conf [OUTPUT] Name s3 Match docker.qtap endpoint http://minio:9000 bucket qtap-http-traffic region us-east-1 total_file_size 50M compression gzip s3_key_format /qtap/year=%Y/month=%m/day=%d/$UUID.gz use_put_object On ``` #### AWS CloudWatch Logs ```conf [OUTPUT] Name cloudwatch_logs Match docker.qtap region us-east-1 log_group_name /qpoint/http-traffic log_stream_prefix qtap- auto_create_group On ``` **Set retention policy:** ```bash aws logs put-retention-policy \ --log-group-name /qpoint/http-traffic \ --retention-in-days 90 ``` #### Multiple Destinations You can send data to multiple outputs simultaneously: ```conf # Send to S3 for long-term storage [OUTPUT] Name s3 Match docker.qtap bucket your-bucket-name region us-east-1 total_file_size 50M upload_timeout 10m compression gzip s3_key_format /qtap/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz store_dir /tmp/fluent-bit/s3 use_put_object On # Also send to stdout for debugging [OUTPUT] Name stdout Match docker.qtap Format json_lines ``` ### Filtering and Optimization #### Capture Only Errors (4xx/5xx) To reduce volume, capture only failed requests using Qtap's Rulekit: ```yaml stacks: error_only: plugins: - type: http_capture config: level: none # Don't capture by default format: json rules: - name: "Capture errors" expr: http.res.status >= 400 level: full # Capture errors fully ``` #### Filter by Domain Capture only specific domains: ```yaml tap: http: stack: lightweight_stack # Default: summary level endpoints: - domain: 'api.important.com' http: stack: capture_all # Full capture for this domain ``` #### Exclude Noisy Processes ```yaml tap: filters: groups: - qpoint # Exclude qtap's own traffic custom: - exe: /usr/bin/healthcheck strategy: exact - exe: /usr/sbin/ strategy: prefix # Exclude all /usr/sbin/ processes ``` #### Filter in Fluent Bit You can also filter within Fluent Bit using the grep filter: ```conf # Only capture requests to specific domain [FILTER] Name grep Match docker.qtap Regex request.url .*api\.important\.com.* # Only keep stdout log lines (tail input exposes `stream`, the forward driver exposes `source`) [FILTER] Name grep Match docker.* Regex stream stdout # Exclude health checks [FILTER] Name grep Match docker.qtap Exclude request.url .*/health$ ``` When you aggregate logs with Docker's forward log driver, Fluent Bit surfaces the channel as `source` instead of `stream`, so update the regex to `Regex source stdout` in that deployment model. ### Monitoring and Troubleshooting #### Verify Qtap is Capturing Traffic ```bash # Check Qtap logs for HTTP transactions docker logs qtap | grep '"metadata"' # Verify TLS detection docker logs qtap | grep '"is_tls":true' ``` **What to look for:** * `"exe": "/usr/bin/curl"` - Process identified correctly * `"protocol": "http2"` or `"http1"` - NOT "other" * `"is_tls": true` - TLS detected * `"tlsProbeTypesDetected": ["openssl"]` - TLS library hooked * Full HTTP details visible despite HTTPS #### Verify Fluent Bit is Processing ```bash # Check Fluent Bit logs for errors docker logs fluent-bit | grep -i "error\|warn" # Count processed HTTP objects (check for the metadata field) docker logs fluent-bit | grep -c '"metadata"' # Check S3 uploads (if using S3 output) docker logs fluent-bit | grep "s3" ``` #### Common Issues **No HTTP objects captured:** * Qtap must be running **before** traffic is generated * Wait 5-10 seconds after starting qtap for eBPF initialization * Check qtap logs for connection events * Verify processes are using supported TLS libraries (OpenSSL, BoringSSL, GnuTLS) **Fluent Bit not receiving logs:** * Check that Fluent Bit started before Qtap * Verify port 24224 is accessible from Qtap container * Test connectivity: `docker exec qtap nc -zv 127.0.0.1 24224` * Check Fluent Bit logs for connection messages * If you see `Error binding socket`, remove any leftover Fluent Bit containers that already bound to port 24224 and re-run `docker compose up -d` **No HTTP transactions in Fluent Bit output:** * Qtap writes both HTTP transaction JSON and operational messages to **stdout** * The grep filter `Regex log .*"metadata":.*` ensures only HTTP transactions are captured * Verify HTTP transactions reached Fluent Bit: `docker logs fluent-bit | grep '"metadata"'` * If you see records without `"metadata"`, those are operational logs that should be filtered out * Make sure the grep filter runs **before** the parser filter or leave `Preserve_Key On`; otherwise the `log` field will disappear before the match runs **High memory usage:** * Reduce Fluent Bit `Flush` interval (more frequent uploads) * Adjust `total_file_size` for S3 batching (smaller = more frequent uploads) * Add filtering to reduce captured volume * Enable compression for outputs **S3 upload failures:** * Verify AWS credentials are correct * Check IAM permissions (s3:PutObject required) * Ensure bucket exists and is in the correct region * Check network connectivity to S3 endpoint ### Alternative: File Tailing Approach If you have existing logging infrastructure or cannot use the forward protocol, you can tail Docker's log files directly: **fluent-bit.conf:** ```conf [INPUT] Name tail Path /var/lib/docker/containers/*/*.log Parser docker Tag docker.* Refresh_Interval 5 Read_from_Head true # Keep only stdout log lines (tail input exposes the field as `stream`) [FILTER] Name grep Match docker.* Regex stream stdout [FILTER] Name grep Match docker.* Regex log .*"metadata":.* # Parse the JSON payload after filtering [FILTER] Name parser Match docker.* Key_Name log Parser generic_json_parser Reserve_Data On Preserve_Key Off ``` **docker-compose.yaml changes:** ```yaml fluent-bit: volumes: - /var/lib/docker/containers:/var/lib/docker/containers:ro qtap: logging: driver: json-file options: max-size: "10m" max-file: "3" ``` Ensure your `parsers.conf` file includes both the built-in `docker` parser and the `generic_json_parser` definition so the tail input can decode Docker log metadata before emitting HTTP transactions. This approach requires more complex path management and may have permission issues. The forward protocol approach is recommended. ### Best Practices 1. **Start Small**: Begin with error-only capture or specific domains, then expand 2. **Set TTLs**: Always configure lifecycle policies (90 days recommended) 3. **Monitor Volume**: Track storage growth and adjust filtering as needed 4. **Secure Credentials**: Use `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables — never hardcode credentials in configuration files. Rotate keys regularly 5. **Compress**: Enable gzip compression for S3 uploads 6. **Batch Uploads**: Use appropriate `total_file_size` (50M default) 7. **Test Filtering**: Validate filters match expected objects 8. **Health Checks**: Monitor Fluent Bit metrics and error logs 9. **Backup Config**: Version control all configuration files 10. **Security**: Limit access to logs - they contain sensitive data ### Summary This guide demonstrated a validated deployment pattern for capturing all HTTP traffic with Fluent Bit using Docker: **Docker Deployment**: Qtap stdout → Forward protocol → Fluent Bit → S3/CloudWatch This pattern provides: * ✅ Complete HTTP capture (headers + bodies) * ✅ TLS inspection without proxies * ✅ Batched, buffered writes for performance * ✅ Flexible routing to multiple destinations * ✅ Data sovereignty (sensitive data stays in your infrastructure) For questions or advanced configurations, see: * [Traffic Processing with Plugins](https://github.com/qpoint-io/documentation/blob/main/guides/getting-started/qtap/configuration/traffic-processing-with-plugins.md) * [Storage Configuration](https://github.com/qpoint-io/documentation/blob/main/guides/getting-started/qtap/configuration/storage-configuration.md) # Debugging & Troubleshooting Tools and techniques for debugging production HTTP/S traffic with Qtap. ## Interactive Debugging [**DevTools - Interactive Traffic Inspection**](/getting-started/qtap/configuration/devtools) - Browser-based interface for real-time HTTP/S debugging See exactly what your applications are sending and receiving, with Chrome DevTools-style inspection for server-side traffic. [**Production Debugging with HTTPS Visibility**](/guides/qtap-guides/debugging/production-debugging-with-https-visibility) - Emergency troubleshooting guide ## Related Resources * [BPF Trace - Advanced Debugging](/appendix/bpf-trace) - Syscall-level debugging for complex issues * [What a Healthy Capture Looks Like](/appendix/healthy-capture-patterns) - Verify Qtap is working correctly # Production Debugging with HTTPS Visibility Get HTTPS traffic visibility in 30 seconds for urgent production debugging - no permanent installation needed. ## Who This Is For **Use this guide when you:** * Have a production issue RIGHT NOW that needs immediate investigation * Need to see inside HTTPS traffic without certificates or proxies * Want to debug authentication failures (401/403), API errors, or rate limiting * Need temporary visibility without permanent installation **This IS for:** * Emergency troubleshooting and incident response * Short-term traffic capture (hours to days) * One-off debugging sessions * Proof that Qtap works in your environment **This is NOT for:** * Long-term production monitoring (use [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) for that) * Selective capture with complex rules (use [Complete Guide](/guides/qtap-guides/getting-started/getting-started-complete-guide) for rulekit) * Multi-environment visibility (consider [Qplane](/guides/qplane-guides/poc-kick-off-guide)) **Time to results:** 30 seconds (demo mode) or 5 minutes (full install) > **🔒 Your Data Stays Local** > > This guide uses standalone Qtap with stdout output—**all captured traffic stays on your server**. No data is sent to Qpoint or any external service. Everything you capture remains in your terminal or log files under your control. > > Want centralized visibility across a fleet with alerting? That's [Qplane](/guides/qplane-guides/poc-kick-off-guide) (our cloud control plane), but this guide keeps everything local and private. *** ## Temporary Installation **The fastest way to see HTTPS traffic.** No installation, no configuration file—just one command that downloads qtap to `/tmp` and starts capturing immediately. ### One-Line Demo SSH into your production server and run: ```bash curl -s https://get.qpoint.io/demo | sudo LOG_LEVEL=warn sh ``` > **Prefer not to pipe curl to sudo?** We get it. See [manual installation instructions](https://docs.qpoint.io/getting-started/qtap/installation/linux-binary#manual-installation) to download and verify the binary yourself before running. Qtap will: 1. Download the latest qtap binary to `/tmp` (no permanent installation) 2. Start capturing all HTTP/HTTPS traffic with full visibility 3. Display readable console output in real-time (access\_logs plugin, full mode) ### What You'll See Captured traffic appears immediately: {% hint style="success" %} **Visual Alternative**: For interactive inspection with a Chrome DevTools-style interface, see the [DevTools UI](/getting-started/qtap/configuration/devtools) guide. Perfect for exploring traffic visually or troubleshooting via SSH tunnel from your laptop. {% endhint %} ``` ============================================================================================================= ■ /usr/bin/curl → GET https://httpbin.org/get 200 OK ============================================================================================================= ------------------ META ------------------ PID: 12345 Exe: /usr/bin/curl Direction: egress-external Bytes Sent: 41 Bytes Received: 395 ------------------ REQUEST ------------------ GET httpbin.org http2 :method: GET :path: /get :scheme: https User-Agent: curl/8.10.1 Accept: */* :authority: httpbin.org ------------------ REQUEST BODY ------------------ (empty) ------------------ RESPONSE ------------------ 200 OK Content-Type: application/json Content-Length: 255 Server: gunicorn/19.9.0 Access-Control-Allow-Origin: * :status: 200 Date: Wed, 29 Oct 2025 19:51:23 GMT ------------------ RESPONSE BODY ------------------ { "args": {}, "headers": { "Accept": "*/*", "Host": "httpbin.org", "User-Agent": "curl/8.10.1" }, "origin": "73.71.138.108", "url": "https://httpbin.org/get" } ``` ### Stop the Demo Press **Ctrl+C** to stop qtap. The binary remains in `/tmp` and will be cleaned up on system reboot. **Pro tip:** The demo script uses a default configuration (access\_logs, full mode, all traffic). For production-safe filtering and custom rules, continue to the sections below. *** ## Full Installation This option gives you full control with custom configuration. Perfect for production-safe debugging with filters and rules. ### Prerequisites * SSH access to production server (Linux with kernel 5.10+) * Root/sudo privileges * Running application to debug ### Step 1: Create Minimal Configuration SSH into your production server and create `/tmp/qtap-debug.yaml`: ```yaml # Minimal production debugging configuration # Captures all HTTP traffic with readable console output version: 2 services: event_stores: - type: stdout # Connection metadata to console object_stores: - type: stdout # Request/response bodies to console stacks: debug: plugins: - type: access_logs config: mode: full # (summary|details|full) - Capture everything format: console # (console|json) - Human-readable output tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # Skip localhost traffic audit_include_dns: false # Skip DNS queries http: stack: debug ``` ### Step 2: Install and Start Qtap Install qtap with a single command: ```bash # Install qtap curl -s https://get.qpoint.io/install | sudo sh # Verify installation qtap --version ``` > **Prefer not to pipe curl to sudo?** See [manual installation instructions](https://docs.qpoint.io/getting-started/qtap/installation/linux-binary#manual-installation) to download, extract, and verify the binary yourself. Start qtap (runs in foreground, shows captured traffic in real-time): ```bash sudo qtap --config=/tmp/qtap-debug.yaml --log-level=info --log-encoding=console | tee /tmp/qtap-output.log ``` Qtap displays captured HTTPS traffic in real-time. Output is also saved to `/tmp/qtap-output.log` for later analysis. ### Step 3: Generate Test Traffic **Open a second SSH session** to your server (keep qtap running in the first session), then generate test traffic: ```bash # Test authentication failure (returns 401) curl https://httpbin.org/status/401 # Test successful request (returns 200) curl https://httpbin.org/get # Test error response (returns 500) curl https://httpbin.org/status/500 ``` Or simply wait for your production application to generate traffic—you'll see it appear in real-time in your first SSH session. ### Step 4: What You'll See **Readable console output** (example 401 error from httpbin): ``` ========================================================================= ■ /usr/bin/curl → GET https://httpbin.org/status/401 401 Unauthorized ========================================================================= ------------------ META ------------------ PID: 12345 Exe: /usr/bin/curl Direction: egress-external Bytes Sent: 54 Bytes Received: 0 ------------------ REQUEST ------------------ GET httpbin.org http2 User-Agent: curl/8.10.1 Accept: */* :authority: httpbin.org :method: GET :path: /status/401 :scheme: https ------------------ RESPONSE ------------------ 401 Unauthorized :status: 401 Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: * Content-Length: 0 Date: Wed, 29 Oct 2025 19:30:00 GMT Server: gunicorn/19.9.0 ------------------ RESPONSE BODY ------------------ (empty) ``` **Key indicators of successful capture:** * ✅ Process identified (`/usr/bin/curl`) * ✅ Request headers visible (including Authorization token!) * ✅ Response body readable (no base64 encoding) * ✅ Complete HTTPS traffic despite encryption ### Step 5: Cleanup When debugging is complete, press **Ctrl+C** in the qtap session to stop it, then: ```bash # Remove configuration and logs rm /tmp/qtap-debug.yaml /tmp/qtap-output.log ``` *** ## Production Debugging Configuration The quick start captures **everything** at full detail. For production use, this single configuration works for all common debugging scenarios—just adjust what you look for in the output. > **Need advanced capture options?** This guide uses the `access_logs` plugin for simplicity. For selective capture with rules, custom filtering, or other plugins (like `http_capture`), see [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins). ### Base Configuration Create `/tmp/qtap-debug.yaml`: ```yaml # Production debugging configuration # Captures all HTTP/HTTPS traffic with full details version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug: plugins: - type: access_logs config: mode: full # (summary|details|full) - Full capture for debugging format: console # (console|json) - Readable output tap: direction: all # (egress|egress-external|egress-internal|ingress|all) ignore_loopback: true # Skip localhost traffic audit_include_dns: false # Skip DNS queries http: stack: debug ``` **Start capturing**: ```bash sudo qtap --config=/tmp/qtap-debug.yaml --log-level=info --log-encoding=console | tee /tmp/qtap-output.log ``` Now use a second SSH session to generate traffic (or just wait for your production app to make or receive requests). *** ## Common Debugging Scenarios Same config, different things to look for in the captured output. ### Scenario 1: Authentication Failures **Problem**: Third-party API returning 401/403 errors **Generate test traffic**: ```bash curl https://api.stripe.com/v1/customers -H "Authorization: Bearer sk_test_invalid" curl https://httpbin.org/status/403 ``` **What to look for in qtap output**: * **Status codes**: `401 Unauthorized` or `403 Forbidden` * **Authorization header**: Check if token is present and formatted correctly * **Response body**: Error message often explains the issue (expired token, wrong scope, etc.) * **Process**: Which service is making the failing request (`exe: /usr/bin/python3`, container name) *** ### Scenario 2: Slow API Responses **Problem**: External APIs taking too long to respond **What to look for in qtap output**: * **Duration**: Response time shown in milliseconds at top of each capture * **Bytes Sent/Received**: Large payloads can cause slowness * **Process**: Which service is making slow requests * **Host**: Which external API is slow **Pro tip**: Use `mode: summary` instead of `full` if you don't need bodies—just timing data. *** ### Scenario 3: Rate Limiting **Problem**: Hitting API rate limits (429 responses) **Generate test traffic**: ```bash curl https://httpbin.org/status/429 ``` **What to look for in qtap output**: * **Status code**: `429 Too Many Requests` * **Rate-limit headers**: `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` * **Request frequency**: Count how many requests to the same host in a short time window * **Process**: Which service is making too many requests *** ### Scenario 4: External API Errors **Problem**: Third-party API integration suddenly failing **What to look for in qtap output**: * **Status codes**: `500`, `502`, `503`, `504` (server errors) * **Error response bodies**: Often contain specific error messages or codes * **Request bodies**: Verify payload format is correct * **Headers**: Check required headers are present (`Content-Type`, API version headers) * **Process**: Which service is affected *** ## Reducing Noise with Filters In busy production environments, you'll capture a lot of traffic. Use filters to focus on what matters. ### Filter by Process Ignore noisy health checks, monitoring agents, or specific services: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug: plugins: - type: access_logs config: mode: full format: console tap: direction: all ignore_loopback: true audit_include_dns: false http: stack: debug filters: groups: - qpoint # Don't capture qtap's own traffic custom: - exe: /usr/bin/health-check strategy: exact # Exact path match - exe: /usr/bin/ strategy: prefix # All executables in /usr/bin/ - exe: .*node.* strategy: regex # Any path matching pattern ``` **Common filters**: * Health checks: `/usr/bin/health-check`, `/usr/bin/curl` (if used for monitoring) * Monitoring agents: `.*datadog.*`, `.*newrelic.*` * Internal tools: `/usr/local/bin/metrics-collector` ### Filter by Domain Capture only specific domains by setting the default stack to `none` and overriding for specific endpoints: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug: plugins: - type: access_logs config: mode: full format: console tap: direction: all ignore_loopback: true audit_include_dns: false # Default: capture nothing http: stack: none # Override: capture only these specific domains endpoints: - domain: 'api.stripe.com' http: stack: debug - domain: 'api.github.com' http: stack: debug ``` **How it works**: The default `http.stack` applies to all traffic. Specific domains under `endpoints` override the default. With `stack: none` as the default, only the listed domains are captured. **When to use domain filtering**: * Debugging a specific third-party integration * High-volume environments where you need to focus * Investigating issues with a particular external service ### Adjust Capture Level Start with less detail, increase as needed by changing the `mode` value in your config: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: debug: plugins: - type: access_logs config: mode: summary # Change this: summary | details | full format: console tap: direction: all ignore_loopback: true audit_include_dns: false http: stack: debug ``` **Available modes**: * `summary`: Method, URL, status, timing - no headers or bodies (minimal overhead) * `details`: Summary + all headers - no bodies (auth/rate-limit debugging) * `full`: Everything including headers and bodies (complete debugging) **Tip**: Start with `summary`, look at the list of requests, then restart qtap with `full` mode to dig into specific issues. ### Filter by Response Code Focus on errors, authentication failures, or rate limiting by capturing only specific HTTP status codes: ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_error expr: http.res.status >= 400 stacks: errors_only: plugins: - type: access_logs config: mode: summary # Start with summary for all traffic format: console rules: - name: "Capture errors at full detail" expr: is_error() mode: full # Upgrade to full mode for errors tap: direction: all ignore_loopback: true audit_include_dns: false http: stack: errors_only ``` **How it works**: All traffic is captured at `summary` level (minimal overhead), but requests with status codes ≥ 400 are automatically upgraded to `full` mode with complete headers and bodies. **Common response code filters**: * Authentication issues: `http.res.status == 401 || http.res.status == 403` * Rate limiting: `http.res.status == 429` * Client errors: `http.res.status >= 400 && http.res.status < 500` * Server errors: `http.res.status >= 500` **When to use response code filtering**: * High-volume production debugging where you only care about failures * Authentication troubleshooting (capture 401/403 fully, ignore successful 200s) * Rate limit investigation (capture 429 responses with all headers) * Error diagnosis without capturing thousands of successful requests *** ## Understanding Captured HTTPS Data Qtap's eBPF hooks capture traffic at the kernel level **before TLS encryption**, giving you complete visibility into HTTPS traffic without proxies or certificate management. ### What You Can See **Request details**: * HTTP method (GET, POST, PUT, DELETE, etc.) * Full URL (scheme, host, path, query parameters) * All request headers (Authorization, Content-Type, custom headers) * Request body (when mode is `full`) **Response details**: * Status code (200, 401, 500, etc.) * All response headers (Content-Type, rate-limit headers, cookies) * Response body (when mode is `full`) * Response timing (duration in milliseconds) **Process attribution** (unique to eBPF-based capture): * Executable path (`/usr/bin/curl`, `/usr/bin/python3`, etc.) * Process ID (PID) * Container name (Docker/Kubernetes) * Container/pod labels **TLS information**: * TLS detected (`is_tls: true`) * TLS library used (`tlsProbeTypesDetected: ["openssl"]`) * Protocol version (`http1`, `http2`) ### Capture Levels Explained | Level | What's Captured | Use Case | | --------- | ---------------------------------------------- | --------------------------------------- | | `summary` | Method, URL, status, timing, process info | High-level monitoring, minimal overhead | | `details` | Summary + all headers (no bodies) | Debug headers, auth tokens, rate limits | | `full` | Everything (headers + request/response bodies) | Complete debugging, error diagnosis | ### Example Captured Request (Full Mode) This is what you'll see with `mode: full` and `format: console`: ``` ================================================================================= ■ /usr/local/bin/python3 → POST https://api.stripe.com/v1/charges 402 Payment Required ================================================================================= ------------------ META ------------------ PID: 54321 Exe: /usr/local/bin/python3 Container: payment-service Direction: egress-external Bytes Sent: 156 Bytes Received: 287 Duration: 187ms ------------------ REQUEST ------------------ POST api.stripe.com http2 :method: POST :path: /v1/charges :scheme: https :authority: api.stripe.com Authorization: Bearer sk_test_... Content-Type: application/x-www-form-urlencoded Stripe-Version: 2023-10-16 ------------------ REQUEST BODY ------------------ amount=2000¤cy=usd&source=tok_visa ------------------ RESPONSE ------------------ 402 Payment Required :status: 402 Content-Type: application/json Request-Id: req_abc123 Stripe-Version: 2023-10-16 ------------------ RESPONSE BODY ------------------ { "error": { "type": "card_error", "code": "card_declined", "message": "Your card was declined." } } ``` *** ## Production Safety ### Performance * Start with `mode: summary` for minimal overhead * Use process or domain filters to reduce volume * Typical overhead: 1-3% CPU ### Sensitive Data Captured data may contain API keys and PII. For quick debugging: * Data stays on your server in `/tmp` * Remove logs when done: `rm /tmp/qtap-*.log` ### Cleanup ```bash # Stop qtap (Ctrl+C in qtap session) rm /tmp/qtap-*.yaml /tmp/qtap-*.log ``` **Need help?** For additional resources: * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - Complete plugin and rules syntax * [Linux Kernel Lockdown for eBPF Applications](/troubleshooting/linux-kernel-lockdown-for-ebpf-applications) - Permission issues * Contact # Advanced Use Cases # HTTPS Header Capture Without Proxies This guide demonstrates how to use QTap to transparently capture HTTP headers from applications without *proxies, code changes, or certificate management*. QTap uses eBPF to monitor traffic at the kernel level, capturing data before encryption. ### Common Use Cases * **Service Usage Analytics**: Track which users or services access internal applications * **API Monitoring**: Capture headers for authentication, rate limiting, or debugging * **Traffic Recording**: Record production requests for testing or replay * **Security Auditing**: Monitor for unauthorized access or suspicious headers * **Service Migration**: Understand dependencies before deprecating endpoints ### How It Works QTap attaches to the kernel using eBPF and intercepts traffic at the TLS/SSL layer, providing visibility into encrypted traffic without managing certificates or deploying proxies. All capture happens out-of-band with minimal performance impact. ### Installation #### Quick Install ```bash # Install/Update QTap curl -s https://get.qpoint.io/install | sudo sh # Verify installation sudo qtap --version ``` ### Configuration Create a configuration file at `/etc/qtap/qtap-config.yaml`: ```yaml version: 2 # Storage Configuration services: # Event stores for connection metadata (anonymized) event_stores: - type: stdout # Object stores for request/response content object_stores: - type: s3 endpoint: s3.amazonaws.com # Or your S3-compatible endpoint bucket: traffic-capture region: us-east-1 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY # Processing Stacks stacks: header_capture: plugins: # HTTP Capture plugin - captures and stores to S3 - type: http_capture config: level: headers # Capture headers (use 'full' for bodies too) format: json # Traffic Capture Settings tap: direction: all # Options: ingress, egress, all ignore_loopback: true audit_include_dns: false http: stack: header_capture ``` ### Running QTap #### Direct Execution ```bash # Set S3 credentials export AWS_ACCESS_KEY_ID="your-access-key" export AWS_SECRET_ACCESS_KEY="your-secret-key" # Run QTap sudo qtap --config=/etc/qtap/qtap-config.yaml ``` #### Systemd Service (Recommended) 1. Create environment file for credentials: ```bash sudo tee /etc/qtap/environment << EOF AWS_ACCESS_KEY_ID=your-access-key AWS_SECRET_ACCESS_KEY=your-secret-key EOF sudo chmod 600 /etc/qtap/environment ``` 2. Create systemd service: ```bash sudo tee /etc/systemd/system/qtap.service << 'EOF' [Unit] Description=QTAP Traffic Capture Service After=network.target [Service] Type=simple User=root EnvironmentFile=/etc/qtap/environment ExecStart=/usr/local/bin/qtap --config=/etc/qtap/qtap-config.yaml Restart=always RestartSec=10 [Install] WantedBy=multi-user.target EOF ``` 3. Start and enable the service: ```bash sudo systemctl daemon-reload sudo systemctl enable qtap sudo systemctl start qtap # Check status sudo systemctl status qtap # View logs sudo journalctl -u qtap -f ``` ### Configuration Examples #### Example 1: Capture Only Internal Traffic ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: traffic-capture region: us-east-1 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY stacks: internal_only: plugins: - type: http_capture config: level: headers format: json rules: # Only capture internal domains - name: "Internal traffic" expr: http.req.host matches /\.(internal|local|private)$/ || http.req.host matches /^(10|172|192)\./ level: headers tap: direction: ingress # Monitor incoming traffic to services ignore_loopback: true audit_include_dns: false http: stack: internal_only ``` #### Example 2: Debug Specific Services ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: traffic-capture region: us-east-1 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY stacks: selective_debug: plugins: - type: http_capture config: level: summary # Default: minimal capture format: json rules: # Full capture for specific API - name: "Payment API debugging" expr: http.req.host == "payment.api.local" || http.req.path contains "/payment/" level: full # Capture headers for auth endpoints - name: "Auth monitoring" expr: http.req.path contains "/auth/" || http.req.path contains "/login" level: headers # Debug containers with label - name: "Container debugging" expr: src.container.labels.debug == "true" level: full tap: direction: all ignore_loopback: true audit_include_dns: false http: stack: selective_debug ``` #### Example 3: Production Traffic Recording ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: s3 endpoint: s3.amazonaws.com bucket: traffic-capture region: us-east-1 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY stacks: production_recording: plugins: - type: http_capture config: level: full # Capture everything for replay format: json rules: # Skip health checks - name: "Ignore health checks" expr: http.req.path in ["/health", "/ping", "/metrics"] level: none # Sample high-volume endpoints - name: "Sample read endpoints" expr: http.req.method == "GET" && http.req.path contains "/api/v1/list" level: summary # Reduce data for high-volume reads tap: direction: ingress ignore_loopback: true audit_include_dns: false http: stack: production_recording # Filter out noisy processes filters: groups: - kubernetes - qpoint custom: - exe: /usr/bin/prometheus strategy: exact ``` #### Alternative: Using MinIO (S3-Compatible Storage) For self-hosted S3-compatible storage like MinIO, adjust the object store configuration: ```yaml object_stores: - type: s3 endpoint: minio.internal:9000 # Your MinIO endpoint bucket: traffic-capture region: us-east-1 access_url: http://minio.internal:9000/{{BUCKET}}/{{DIGEST}} insecure: true # Set to false if using HTTPS access_key: type: env value: MINIO_ACCESS_KEY secret_key: type: env value: MINIO_SECRET_KEY ``` ### Understanding Capture Levels #### http\_capture Plugin Levels * **none**: No capture (disables the rule) * **summary**: Basic metadata (method, path, status code) * **details**: Includes all headers (recommended for user tracking) * **full**: Complete request/response including bodies #### Rule Expression Syntax QTAP uses Rulekit for filtering. Common expressions: ```yaml # Match by host expr: http.req.host == "api.example.com" # Match by path pattern expr: http.req.path contains "/api/" # Match by status code expr: http.res.status >= 400 # Match by header expr: http.req.headers.authorization != "" # Combine conditions expr: http.req.method == "POST" && http.res.status == 200 # Match container labels expr: src.container.labels.app == "frontend" # Match IP ranges expr: http.req.host matches /^10\./ || http.req.host matches /^192\.168\./ ``` ### Captured Data Format Example of captured HTTP transaction with headers: ```json { "timestamp": "2024-10-15T10:23:45Z", "direction": "ingress", "source": { "ip": "10.0.1.50", "port": 54321, "process": { "binary": "/usr/bin/node" } }, "destination": { "ip": "10.0.2.100", "port": 8080 }, "http": { "method": "GET", "path": "/api/v1/users", "host": "api.internal.com", "headers": { "user-agent": "Mozilla/5.0", "authorization": "Bearer eyJ...", "x-request-id": "abc-123", "x-user-id": "user-456", "content-type": "application/json" }, "status": 200, "response_headers": { "content-type": "application/json", "x-response-time": "124ms" } } } ``` ### Analyzing Captured Data #### Query S3 with AWS CLI ```bash # List captured files aws s3 ls s3://traffic-capture/ --recursive # Download and analyze aws s3 cp s3://traffic-capture/2024/10/15/capture.json.gz - | \ gunzip | \ jq '.http.headers' ``` #### Extract User Analytics ```bash # Find unique users accessing a service aws s3 cp s3://traffic-capture/ . --recursive --exclude "*" --include "*.json.gz" for file in *.json.gz; do gunzip -c "$file" | jq -r '.http.headers["x-user-id"]' done | sort | uniq -c ``` #### Create Usage Report ```python import json import gzip import boto3 from collections import defaultdict s3 = boto3.client('s3') bucket = 'traffic-capture' # Aggregate usage by endpoint usage = defaultdict(set) paginator = s3.get_paginator('list_objects_v2') for page in paginator.paginate(Bucket=bucket): for obj in page.get('Contents', []): response = s3.get_object(Bucket=bucket, Key=obj['Key']) with gzip.GzipFile(fileobj=response['Body']) as gz: data = json.load(gz) endpoint = f"{data['http']['method']} {data['http']['path']}" user = data['http']['headers'].get('x-user-id', 'anonymous') usage[endpoint].add(user) # Print usage report for endpoint, users in sorted(usage.items()): print(f"{endpoint}: {len(users)} unique users") ``` ### Performance Considerations * **Capture Level**: Use `details` for headers only, `full` only when bodies are needed * **Filtering**: Use rules to limit capture to relevant traffic * **Sampling**: For high-volume services, consider sampling strategies * **Storage**: Rotate S3 data based on retention requirements * **Process Filtering**: Exclude noisy system processes ### Troubleshooting #### Common Issues **No data captured:** * Verify QTAP is running: `sudo systemctl status qtap` * Check logs: `sudo journalctl -u qtap -n 100` * Ensure traffic matches direction setting (ingress vs egress) * Verify HTTP traffic is on expected ports **S3 upload failures:** * Test credentials: `aws s3 ls s3://traffic-capture/` * Check bucket permissions and region * Verify network connectivity to S3 **Missing headers:** * Ensure capture level is `details` or `full` * Verify the `http_capture` plugin is configured * Check that traffic is HTTP/HTTPS (not other protocols) **High memory usage:** * Reduce capture level from `full` to `details` * Add filtering rules to limit captured traffic * Increase sampling intervals for high-volume endpoints #### Debug Mode Run QTAP in debug mode for troubleshooting: ```bash sudo qtap --config=/etc/qtap/qtap-config.yaml --log-level=debug ``` ### Security Best Practices 1. **Credential Management**: Use `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables for S3 credentials. Rotate keys regularly 2. **Data Retention**: Implement S3 lifecycle policies for automatic data expiration 3. **Access Control**: Restrict S3 bucket access to authorized users only 4. **Sensitive Data**: Consider filtering out sensitive headers before storage 5. **Encryption**: Enable S3 server-side encryption for stored data ### Summary QTap provides transparent HTTP header capture without requiring proxies or code changes. By leveraging eBPF, it captures traffic at the kernel level with minimal performance impact, making it ideal for: * Understanding service dependencies * Tracking API usage patterns * Debugging production issues * Recording traffic for testing * Security auditing The combination of flexible filtering rules and native S3 integration makes QTAP a powerful tool for gaining visibility into your HTTP traffic. # CI Security Validation Your CI pipeline is a security boundary. Code running during builds and tests can make network requests—to package registries, external APIs, analytics endpoints, or worse. Without visibility into this traffic, you can't detect supply chain attacks, accidental credential exfiltration, or compliance violations before they reach production. This guide shows how to use Qtap in GitHub Actions to observe ALL network traffic during CI, enforce network allow-lists, and fail builds when unexpected hosts are contacted. ## Why This Matters ### The Problem **Your CI environment makes network calls you don't know about:** * **Supply chain attacks** - A compromised npm/pip package phones home during `npm install` or `pip install` * **Accidental production calls** - Integration tests hit production APIs instead of staging * **Secrets exfiltration** - Malicious code in dependencies POSTs environment variables to attacker-controlled servers * **Compliance violations** - HIPAA/SOC2/PCI require proving no data leaves approved networks * **Dependency drift** - New transitive dependencies fetch from unexpected registries Traditional approaches don't help: * **Network policies** - Not available in most CI runners (GitHub Actions, CircleCI, etc.) * **egress proxies** - Require app code changes and don't catch kernel-level traffic * **Log scraping** - Only shows what apps choose to log, misses silent exfiltration ### The Solution **Qtap captures ALL network traffic at the Linux kernel level** using eBPF: * **Zero code changes** - Observes traffic passively, no proxy configuration needed * **Complete visibility** - Sees inside TLS/HTTPS before encryption happens * **Process attribution** - Know exactly which command made each request * **Enforcement** - Fail builds automatically when disallowed hosts are contacted ### Use Cases This guide demonstrates the pattern with a simple curl example, but the same approach works for: 1. **Package manager validation** - Ensure `npm install`, `pip install`, `go get` only fetch from approved registries 2. **Integration test guardrails** - Verify your app only calls staging/test APIs, never production 3. **Docker build security** - Confirm `apt-get`, `curl` in Dockerfiles only reach approved mirrors 4. **Compliance auditing** - Generate tamper-proof logs of all CI network activity for security reviews 5. **Secrets leak prevention** - Detect when dependencies try to exfiltrate AWS keys, tokens, etc. ## Prerequisites * Repository uses GitHub Actions with Docker available on runners. * Basic familiarity with `docker compose`. * Python 3 (for the helper script). ## Quick Start: Basic Pattern This section demonstrates the core pattern using a simple curl command. Once you understand the mechanics, see the production examples below for real-world use cases. ### Directory Layout The snippets below assume this directory layout: ``` . ├── docker-compose.ci.yaml ├── qtap-ci.yaml ├── scripts/ci/verify_allowed_hosts.py └── .github/workflows/qtap-security.yml ``` ### Step 1 – Qtap Configuration (`qtap-ci.yaml`) ```yaml version: 2 services: # Stream metadata to stdout so the pipeline can collect it event_stores: - type: stdout # Stream HTTP request/response summaries to stdout as well object_stores: - type: stdout stacks: ci_monitoring: plugins: - type: http_capture config: level: summary format: json rules: # Flag suspicious traffic loudly in the logs - name: "ALERT: Disallowed host" expr: | http.req.host != "localhost" && http.req.host != "localhost:8000" && http.req.host != "approved-api" && http.req.host != "approved-api:8000" level: full tap: direction: all ignore_loopback: false http: stack: ci_monitoring ``` We're using `format: json` to make downstream parsing easy. ### Step 2 – Compose File (`docker-compose.ci.yaml`) ```yaml version: '3.9' services: approved-api: image: kennethreitz/httpbin container_name: approved-api ports: - "8000:80" qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-ci privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap-ci.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml ``` The GitHub runner host can reach `approved-api` on `localhost:8000`, and Qtap observes both the host traffic and container cross-talk. ### Step 3 – Verification Script (`scripts/ci/verify_allowed_hosts.py`) ```python #!/usr/bin/env python3 """ Fail the pipeline if Qtap saw traffic to hosts outside the allow list. """ from __future__ import annotations import re import subprocess import sys from pathlib import Path QTAP_CONTAINER = "qtap-ci" ALLOWED = {"localhost:8000", "localhost", "approved-api", "approved-api:8000"} LOG_PATH = Path("artifacts/qtap.log") # Match the "authority" field from Qtap's JSON output (e.g., "authority": "example.com") AUTHORITY_REGEX = re.compile(r'"authority"\s*:\s*"([^"]+)"') def collect_logs() -> str: result = subprocess.run( ["docker", "logs", QTAP_CONTAINER], capture_output=True, text=True, check=True, ) LOG_PATH.parent.mkdir(parents=True, exist_ok=True) LOG_PATH.write_text(result.stdout, encoding="utf-8") return result.stdout def extract_hosts(log_text: str) -> list[str]: return [match.group(1) for match in AUTHORITY_REGEX.finditer(log_text)] def main() -> int: log_text = collect_logs() hosts = extract_hosts(log_text) disallowed = sorted(set(h for h in hosts if h not in ALLOWED)) if disallowed: print("Disallowed hosts detected by Qtap:", file=sys.stderr) for host in disallowed: print(f" - {host}", file=sys.stderr) print(f"\nQtap logs saved to {LOG_PATH}", file=sys.stderr) return 1 print("Qtap host allow-list check passed.") return 0 if __name__ == "__main__": sys.exit(main()) ``` The script downloads the entire Qtap log, saves it as an artifact, and fails if any host falls outside the allowed set. ### Step 4 – GitHub Actions Workflow (`.github/workflows/qtap-security.yml`) ```yaml name: Validate Outbound Traffic on: pull_request: push: branches: [ main ] jobs: qtap-network-guard: runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@v4 - name: Start Qtap and approved API run: docker compose -f docker-compose.ci.yaml up -d qtap approved-api - name: Warm up Qtap run: sleep 6 - name: Generate expected traffic run: | docker run --rm --network host curlimages/curl:8.10.1 \ --silent --show-error http://localhost:8000/get > /dev/null - name: (Optional) Generate forbidden traffic example if: env.QTAP_NEGATIVE_TEST == 'true' run: | docker run --rm curlimages/curl:8.10.1 \ --silent --show-error https://example.com > /dev/null - name: Verify allowed hosts run: python scripts/ci/verify_allowed_hosts.py - name: Upload Qtap logs if: always() uses: actions/upload-artifact@v4 with: name: qtap-logs path: artifacts/qtap.log - name: Tear down if: always() run: docker compose -f docker-compose.ci.yaml down -v ``` Set `QTAP_NEGATIVE_TEST=true` in workflow dispatch to watch the job fail on purpose. In normal runs you’d omit that step, keeping the allow-list tight. ## Understanding the Qtap Output Qtap emits one JSON object per line for each HTTP transaction. Here's what the output looks like: ### Allowed Host Example Traffic to `localhost:8000` (on the allow-list): ```json { "metadata": { "process_id": "57081", "process_exe": "/usr/bin/curl", "bytes_sent": 81, "bytes_received": 419, "connection_id": "d3rvps07p3qhrpga199g", "endpoint_id": "localhost" }, "request": { "method": "GET", "url": "http://localhost:8000/get", "scheme": "http", "path": "/get", "authority": "localhost:8000", "protocol": "http1", "request_id": "d3rvps07p3qhrpga19ag", "user_agent": "curl/8.10.1" }, "response": { "status": 200, "content_type": "application/json" }, "transaction_time": "2025-10-21T21:36:48.691937519Z", "duration_ms": 2, "direction": "egress-external" } ``` Notice the `"authority": "localhost:8000"` field—the verification script extracts this to confirm it's on the allow-list. This request shows **summary level** capture (no `headers` or `body` fields) because it doesn't match the "Disallowed host" rule. ### Disallowed Host Example Traffic to `example.com` (NOT on the allow-list) - captured at **full level** because it triggers the "Disallowed host" rule: ```json { "metadata": { "process_id": "57156", "process_exe": "/usr/bin/curl", "bytes_sent": 37, "bytes_received": 679, "connection_id": "d3rvpso7p3qhrpga1a80", "endpoint_id": "example.com" }, "request": { "method": "GET", "url": "https://example.com/", "scheme": "https", "path": "/", "authority": "example.com", "protocol": "http2", "request_id": "d3rvpso7p3qhrpga1a8g", "user_agent": "curl/8.10.1", "headers": { ":authority": "example.com", ":method": "GET", ":path": "/", ":scheme": "https", "Accept": "*/*", "User-Agent": "curl/8.10.1" } }, "response": { "status": 200, "content_type": "text/html", "headers": { ":status": "200", "Alt-Svc": "h3=\":443\"; ma=93600", "Cache-Control": "max-age=86000", "Content-Length": "513", "Content-Type": "text/html", "Date": "Tue, 21 Oct 2025 21:36:51 GMT", "Etag": "\"bc2473a18e003bdb249eba5ce893033f:1760028122.592274\"", "Last-Modified": "Thu, 09 Oct 2025 16:42:02 GMT" }, "body": "PCFkb2N0eXBlIGh0bWw+PGh0bWwgbGFuZz0iZW4iPjxoZWFkPjx0aXRsZT5FeGFtcGxlIERvbWFpbjwvdGl0bGU+PG1ldGEgbmFtZT0idmlld3BvcnQiIGNvbnRlbnQ9IndpZHRoPWRldmljZS13aWR0aCwgaW5pdGlhbC1zY2FsZT0xIj48c3R5bGU+Ym9keXtiYWNrZ3JvdW5kOiNlZWU7d2lkdGg6NjB2dzttYXJnaW46MTV2aCBhdXRvO2ZvbnQtZmFtaWx5OnN5c3RlbS11aSxzYW5zLXNlcmlmfWgxe2ZvbnQtc2l6ZToxLjVlbX1kaXZ7b3BhY2l0eTowLjh9YTpsaW5rLGE6dmlzaXRlZHtjb2xvcjojMzQ4fTwvc3R5bGU+PGJvZHk+PGRpdj48aDE+RXhhbXBsZSBEb21haW48L2gxPjxwPlRoaXMgZG9tYWluIGlzIGZvciB1c2UgaW4gZG9jdW1lbnRhdGlvbiBleGFtcGxlcyB3aXRob3V0IG5lZWRpbmcgcGVybWlzc2lvbi4gQXZvaWQgdXNlIGluIG9wZXJhdGlvbnMuPHA+PGEgaHJlZj0iaHR0cHM6Ly9pYW5hLm9yZy9kb21haW5zL2V4YW1wbGUiPkxlYXJuIG1vcmU8L2E+PC9kaXY+PC9ib2R5PjwvaHRtbD4K" }, "transaction_time": "2025-10-21T21:36:51.525118272Z", "duration_ms": 47, "direction": "egress-external" } ``` Here `"authority": "example.com"` triggers the verification script to fail because `example.com` is not in the `ALLOWED` set. ### Key Fields to Note * `metadata.process_exe` – Identifies which process made the request (`/usr/bin/curl` in these examples) * `request.authority` – The host:port being accessed (what the verification script checks) * `direction` – `egress-external` means outbound to external hosts * `request.headers` – Only present at `level: full` (disallowed hosts in this example) * `response.headers` – Only present at `level: full` * `response.body` – Only present at `level: full` (base64-encoded) **Capture Levels:** * **Summary level** (allowed hosts): Basic metadata only - no headers or bodies * **Full level** (disallowed hosts): Complete request/response including headers and bodies If the job fails, download `qtap-logs` from the Actions run to inspect exactly which host triggered the alert. ## Production Examples Now that you understand the basic pattern, here's how to apply it to real-world scenarios. ### Example 1: Validate npm install **Scenario:** Ensure `npm install` only fetches packages from your approved registries (npmjs.com, your internal registry). **Complete Qtap configuration (`qtap-npm.yaml`):** ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: npm_monitoring: plugins: - type: http_capture config: level: summary format: json rules: - name: "ALERT: Unexpected npm registry" expr: | http.req.host != "registry.npmjs.org" && http.req.host != "registry.yarnpkg.com" && http.req.host != "npm.company.internal" level: full tap: direction: all ignore_loopback: false http: stack: npm_monitoring ``` **Workflow changes:** ```yaml - name: Start Qtap run: docker compose -f docker-compose.ci.yaml up -d qtap - name: Warm up Qtap run: sleep 6 - name: Install dependencies run: npm ci - name: Verify npm only used approved registries run: python scripts/ci/verify_allowed_hosts.py env: QTAP_ALLOWED_HOSTS: "registry.npmjs.org,registry.yarnpkg.com,npm.company.internal" ``` **What this catches:** * Malicious packages that phone home during install scripts * Typosquatting packages from unexpected registries * Dependency confusion attacks using public registries when internal ones expected ### Example 2: Validate Integration Tests **Scenario:** Your app has integration tests that should only call staging APIs, never production. **Complete Qtap configuration (`qtap-integration-tests.yaml`):** ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: integration_test_monitoring: plugins: - type: http_capture config: level: summary format: json rules: - name: "ALERT: Production API called during tests" expr: | http.req.host matches /^.*\.company\.com$/ && http.req.host != "api.staging.company.com" && http.req.host != "auth.staging.company.com" && http.req.host != "api.test.company.com" && http.req.host != "auth.test.company.com" level: full tap: direction: all ignore_loopback: false http: stack: integration_test_monitoring ``` **Workflow changes:** ```yaml - name: Start Qtap and test dependencies run: docker compose -f docker-compose.ci.yaml up -d - name: Warm up Qtap run: sleep 6 - name: Run integration tests run: npm test - name: Verify tests only called staging run: python scripts/ci/verify_allowed_hosts.py env: QTAP_ALLOWED_HOSTS: "api.staging.company.com,auth.staging.company.com,localhost,localhost:3000" ``` **What this catches:** * Hardcoded production URLs accidentally committed * Environment variable misconfigurations * Tests that modify production data ### Example 3: Validate Docker Build **Scenario:** Your Dockerfile runs `apt-get`, `curl`, `wget` - ensure they only fetch from approved mirrors. **Complete Qtap configuration (`qtap-docker-build.yaml`):** ```yaml version: 2 services: event_stores: - type: stdout object_stores: - type: stdout stacks: docker_build_monitoring: plugins: - type: http_capture config: level: summary format: json rules: - name: "ALERT: Docker build fetched from unexpected host" expr: | http.req.host != "deb.debian.org" && http.req.host != "security.debian.org" && http.req.host != "archive.ubuntu.com" && http.req.host != "security.ubuntu.com" && http.req.host != "github.com" && http.req.host != "mirror.company.internal" level: full tap: direction: all ignore_loopback: false http: stack: docker_build_monitoring ``` **Workflow changes:** ```yaml - name: Start Qtap run: docker compose -f docker-compose.ci.yaml up -d qtap - name: Warm up Qtap run: sleep 6 - name: Build Docker image run: docker build -t myapp:ci . - name: Verify build only used approved mirrors run: python scripts/ci/verify_allowed_hosts.py env: QTAP_ALLOWED_HOSTS: "deb.debian.org,security.debian.org,archive.ubuntu.com,github.com,mirror.company.internal" ``` **What this catches:** * Malicious RUN commands in Dockerfile * Compromised base images with modified package sources * wget/curl commands fetching from unexpected locations ## Adapting to Your Environment ### Update the Allow-List The verification script accepts environment variables for flexibility: ```bash export QTAP_ALLOWED_HOSTS="host1.com,host2.com,localhost:8080" python scripts/ci/verify_allowed_hosts.py ``` ### Add More Rulekit Rules Beyond host validation, you can flag other suspicious patterns: ```yaml rules: # Flag requests with Authorization headers (potential secret leak) - name: "ALERT: Authorization header detected" expr: http.req.headers.authorization != "" level: full # Flag POST requests (writes/uploads) - name: "ALERT: POST request to external host" expr: | http.req.method == "POST" && http.req.host != "localhost" && http.req.host != "approved-api" level: full # Flag non-TLS traffic to external hosts - name: "ALERT: Unencrypted external traffic" expr: | http.req.scheme == "http" && http.req.host != "localhost" && http.req.host != "approved-api" level: full ``` {% hint style="warning" %} **Rulekit Limitation:** The `not` operator does not currently work in rule expressions or macros. Use explicit negation with `!=` and `&&` operators instead. For example, instead of: ```yaml expr: not (http.req.host == "localhost" || http.req.host == "approved-api") ``` Use: ```yaml expr: http.req.host != "localhost" && http.req.host != "approved-api" ``` {% endhint %} ### Archive for Compliance For HIPAA/SOC2/PCI compliance, archive Qtap logs permanently: ```yaml - name: Upload Qtap logs to S3 (compliance archive) if: always() run: | aws s3 cp artifacts/qtap.log \ s3://compliance-artifacts/ci-logs/${{ github.run_id }}/qtap.log \ --metadata "repo=${{ github.repository }},commit=${{ github.sha }}" ``` ## Summary With this pattern, network policy drift becomes visible during every pull request—long before changes reach production. You can: * **Prevent supply chain attacks** by validating package manager traffic * **Enforce environment boundaries** by blocking accidental production calls * **Detect secrets exfiltration** by flagging unexpected outbound requests * **Generate compliance evidence** with complete network audit trails Continuous Integration now enforces both functional AND security expectations. # Service Mesh Lite: Observability Without Sidecars Service meshes like Istio and Linkerd provide powerful observability into microservice communication, but they come with significant operational overhead: complex deployments, sidecar proxies on every pod, added latency, and resource consumption. **What if you could get service mesh observability without the complexity?** This guide demonstrates how Qtap provides core service mesh observability features—service discovery, request tracing, error monitoring, and latency tracking—without sidecars, without control planes, and with zero latency impact. *** ## The Service Mesh Complexity Problem Service meshes solve real problems but introduce significant complexity: **Deployment Complexity:** * Control plane components (istiod, pilot, mixer) * Sidecar injection on every pod * Webhook configuration and RBAC setup * Certificate management for mTLS **Resource Overhead:** * Each sidecar consumes CPU/memory (multiply by number of pods) * Control plane resource requirements * Significant cluster resource increase **Latency Impact:** * Every request goes through proxy (adds milliseconds) * Data plane overhead on critical path * Performance degradation under load **Operational Burden:** * Complex troubleshooting (is it my app or the mesh?) * Version upgrades affect entire cluster * Learning curve for operators **Most teams want observability, not traffic control.** If you don't need advanced features like circuit breaking, traffic splitting, or mTLS enforcement, you're paying a high price for what you actually use. *** ## What Qtap Provides Instead Qtap offers a "Service Mesh Lite" approach focused purely on observability: **✅ What You Get:** * **Service Discovery**: Automatic mapping of service dependencies * **Request Tracing**: Follow requests across services * **Error Monitoring**: Track failures and error rates per service * **Latency Metrics**: Measure p50, p95, p99 response times * **Traffic Volume**: Connections per second, bytes transferred * **Zero Instrumentation**: No code changes or configuration needed **✅ How It's Different:** * **Out-of-band observation**: Qtap watches traffic passively, doesn't proxy it * **Zero latency**: No proxies in the request path * **No sidecars**: One DaemonSet per node (not per pod) * **No control plane**: Just install qtap and start observing **❌ What Qtap DOESN'T Do:** * Traffic routing or load balancing * Circuit breaking or retries * Traffic splitting (canary deployments) * mTLS between services * Rate limiting or quotas **Qtap is observation-only.** If you need traffic control, use a service mesh. If you just need visibility, Qtap is simpler. *** ## Demo Architecture We'll demonstrate service mesh observability with a multi-service application: ``` ┌─────────────┐ ┌─────────→│ Frontend │ │ │ (Client) │ │ └──────┬──────┘ │ │ │ ↓ [External] ┌─────────────┐ [Request] │ API │ (Port 5000) │ Gateway │ └──────┬──────┘ │ ┌──────┴──────┐ ↓ ↓ ┌─────────────┐ ┌─────────────┐ │ User │ │ Order │ │ Service │ │ Service │ │ (Port 5001) │ │ (Port 5002) │ └─────────────┘ └──────┬──────┘ │ ↓ ┌─────────────┐ │ httpbin.org│ (External API) └─────────────┘ ``` **Traffic Flows:** 1. **Client → API Gateway → User Service**: Simple service-to-service call 2. **Client → API Gateway → Order Service → httpbin.org**: Service calling external API 3. **Client → API Gateway → User Service + Order Service**: Fan-out to multiple services **Qtap captures ALL of this:** * Service A calling Service B (both egress from A and ingress to B) * External API calls from services * Error responses and status codes * Request/response timing *** ## Prerequisites * Linux host with Docker * `jq` for JSON parsing * Basic understanding of microservices *** ## Part 1: Deploy the Multi-Service Application We'll create three microservices that communicate with each other. ### API Gateway Service ```python #!/usr/bin/env python3 """ API Gateway - Routes requests to backend services """ from flask import Flask, jsonify import requests import os app = Flask(__name__) USER_SERVICE_URL = os.environ.get('USER_SERVICE_URL', 'http://localhost:5001') ORDER_SERVICE_URL = os.environ.get('ORDER_SERVICE_URL', 'http://localhost:5002') @app.route('/health') def health(): return jsonify({'status': 'healthy', 'service': 'api-gateway'}) @app.route('/api/users/') def get_user(user_id): """Proxy request to user service""" response = requests.get(f'{USER_SERVICE_URL}/users/{user_id}', timeout=5) return jsonify(response.json()), response.status_code @app.route('/api/orders/') def get_orders(user_id): """Proxy request to order service""" response = requests.get(f'{ORDER_SERVICE_URL}/orders/{user_id}', timeout=5) return jsonify(response.json()), response.status_code @app.route('/api/user-with-orders/') def get_user_with_orders(user_id): """Fan-out request to both services""" user_response = requests.get(f'{USER_SERVICE_URL}/users/{user_id}', timeout=5) orders_response = requests.get(f'{ORDER_SERVICE_URL}/orders/{user_id}', timeout=5) return jsonify({ 'user': user_response.json(), 'orders': orders_response.json() }), 200 if __name__ == '__main__': app.run(host='0.0.0.0', port=5000) ``` ### User Service ```python #!/usr/bin/env python3 """ User Service - Returns user data """ from flask import Flask, jsonify app = Flask(__name__) USERS = { '1': {'id': '1', 'name': 'Alice', 'email': 'alice@example.com'}, '2': {'id': '2', 'name': 'Bob', 'email': 'bob@example.com'}, '3': {'id': '3', 'name': 'Charlie', 'email': 'charlie@example.com'}, } @app.route('/health') def health(): return jsonify({'status': 'healthy', 'service': 'user-service'}) @app.route('/users/') def get_user(user_id): if user_id == '999': # Simulate error return jsonify({'error': 'User not found', 'service': 'user-service'}), 404 user = USERS.get(user_id) if user: return jsonify(user), 200 else: return jsonify({'error': 'User not found', 'service': 'user-service'}), 404 if __name__ == '__main__': app.run(host='0.0.0.0', port=5001) ``` ### Order Service ```python #!/usr/bin/env python3 """ Order Service - Returns orders and calls external payment API """ from flask import Flask, jsonify import requests app = Flask(__name__) ORDERS = { '1': [ {'id': 'order-101', 'user_id': '1', 'total': 99.99, 'status': 'completed'}, {'id': 'order-102', 'user_id': '1', 'total': 49.99, 'status': 'pending'}, ], '2': [ {'id': 'order-201', 'user_id': '2', 'total': 199.99, 'status': 'completed'}, ], '3': [], } @app.route('/health') def health(): return jsonify({'status': 'healthy', 'service': 'order-service'}) @app.route('/orders/') def get_orders(user_id): """Get orders and verify with external payment API""" try: # Call external API (httpbin for demo) payment_response = requests.get( 'https://httpbin.org/json', timeout=5, headers={'X-User-ID': user_id} ) payment_status = 'payment-api-ok' if payment_response.status_code == 200 else 'payment-api-error' except Exception as e: payment_status = f'payment-api-error: {str(e)}' orders = ORDERS.get(user_id, []) return jsonify({ 'user_id': user_id, 'orders': orders, 'payment_api_status': payment_status, 'service': 'order-service' }), 200 if __name__ == '__main__': app.run(host='0.0.0.0', port=5002) ``` *** ## Part 2: Qtap Configuration for Service Mesh Observability Create `/tmp/qtap-mesh-lite.yaml`: ```yaml version: 2 # Service Mesh Lite Configuration # Captures all service-to-service traffic for observability services: event_stores: - type: stdout object_stores: - type: stdout rulekit: macros: - name: is_error expr: http.res.status >= 400 && http.res.status < 600 - name: is_api_gateway expr: src.container.name == "api-gateway" - name: is_user_service expr: src.container.name == "user-service" - name: is_order_service expr: src.container.name == "order-service" stacks: mesh_observability: plugins: - type: http_capture config: level: summary # (none|summary|headers|full) - Lightweight by default format: json # (json|text) - JSON for easy parsing rules: # Capture full details for errors to debug failures - name: "Service errors" expr: is_error() level: full # Capture details for all API gateway traffic - name: "API Gateway traffic" expr: is_api_gateway() level: headers tap: direction: all # (egress|egress-external|egress-internal|ingress|all) # CRITICAL: Use 'all' to see BOTH sides of service calls ignore_loopback: false # (true|false) - MUST be false to see localhost traffic! audit_include_dns: false # (true|false) - Skip DNS noise http: stack: mesh_observability # Filter out qtap's own traffic filters: groups: - qpoint ``` **Key configuration points:** 1. **`direction: all`**: CRITICAL for service mesh observability. Captures both: * **Egress**: Requests leaving Service A * **Ingress**: Requests arriving at Service B * This gives you BOTH sides of every service-to-service call 2. **`ignore_loopback: false`**: Required to capture localhost traffic in Docker 3. **`level: summary` by default**: Lightweight capture (method, URL, status, timing) 4. **`level: full` for errors**: Capture complete request/response for debugging 5. **Rulekit macros**: Identify traffic by container name for filtering *** ## Part 3: Deploy with Docker Compose Create `/tmp/docker-compose-mesh-lite.yaml`: ```yaml version: '3.9' services: # API Gateway - Entry point api-gateway: image: python:3.11-slim container_name: api-gateway working_dir: /app network_mode: host # Required for qtap to see localhost traffic command: > bash -c "pip install -q flask requests && python api-gateway.py" volumes: - /tmp/api-gateway.py:/app/api-gateway.py environment: - USER_SERVICE_URL=http://localhost:5001 - ORDER_SERVICE_URL=http://localhost:5002 # User Service user-service: image: python:3.11-slim container_name: user-service working_dir: /app network_mode: host command: > bash -c "pip install -q flask && python user-service.py" volumes: - /tmp/user-service.py:/app/user-service.py # Order Service order-service: image: python:3.11-slim container_name: order-service working_dir: /app network_mode: host command: > bash -c "pip install -q flask requests && python order-service.py" volumes: - /tmp/order-service.py:/app/order-service.py # Qtap - Traffic capture qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap-mesh-lite privileged: true user: "0:0" cap_add: - CAP_BPF - CAP_SYS_ADMIN pid: host network_mode: host volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - /tmp/qtap-mesh-lite.yaml:/app/config/qtap.yaml environment: - TINI_SUBREAPER=1 ulimits: memlock: -1 command: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml ``` **Start the stack:** ```bash docker compose -f /tmp/docker-compose-mesh-lite.yaml up -d ``` **Wait for services to initialize:** ```bash sleep 10 ``` *** ## Part 4: Generate Traffic and Observe ### Test 1: Simple Service-to-Service Call ```bash curl -s http://localhost:5000/api/users/1 | jq . ``` **Output:** ```json { "email": "alice@example.com", "id": "1", "name": "Alice" } ``` **What Qtap captures:** * Client → API Gateway (egress from curl) * API Gateway → User Service (egress from gateway, ingress to user-service) * Complete request chain with timing *** ### Test 2: Service Calling External API ```bash curl -s http://localhost:5000/api/orders/1 | jq . ``` **Output:** ```json { "orders": [ { "id": "order-101", "status": "completed", "total": 99.99, "user_id": "1" }, { "id": "order-102", "status": "pending", "total": 49.99, "user_id": "1" } ], "payment_api_status": "payment-api-ok", "service": "order-service", "user_id": "1" } ``` **What Qtap captures:** * Client → API Gateway * API Gateway → Order Service * Order Service → httpbin.org (external API call) * **TLS inspection**: Qtap sees inside HTTPS to httpbin.org before encryption *** ### Test 3: Fan-Out Request ```bash curl -s http://localhost:5000/api/user-with-orders/2 | jq . ``` **Output:** ```json { "orders": { "orders": [ { "id": "order-201", "status": "completed", "total": 199.99, "user_id": "2" } ], "payment_api_status": "payment-api-ok", "service": "order-service", "user_id": "2" }, "user": { "email": "bob@example.com", "id": "2", "name": "Bob" } } ``` **What Qtap captures:** * API Gateway making parallel calls to User Service AND Order Service * Both service responses with timing * Service dependency mapping (gateway depends on both services) *** ### Test 4: Error Scenario ```bash curl -s http://localhost:5000/api/users/999 | jq . ``` **Output:** ```json { "error": "User not found", "service": "user-service" } ``` **What Qtap captures:** * 404 response captured with `level: full` (includes headers and body) * Error tracking per service * Useful for identifying which service is failing *** ## Part 5: Analyze Service Mesh Observability ### View Qtap Logs ```bash docker logs qtap-mesh-lite 2>&1 | grep '"method"' | jq . ``` ### Example 1: Service-to-Service Call (Egress Side) ```json { "metadata": { "process_exe": "/usr/local/bin/python3.11", "connection_id": "abc123" }, "request": { "method": "GET", "url": "http://localhost:5001/users/1", "authority": "localhost:5001", "protocol": "http1", "user_agent": "python-requests/2.32.5" }, "response": { "status": 200, "content_type": "application/json" }, "direction": "egress-external", "duration_ms": 2 } ``` **Interpretation:** * Python process (API Gateway) made request * Destination: User Service (port 5001) * Direction: **egress** (leaving API Gateway) * Response time: 2ms *** ### Example 2: Same Call (Ingress Side) ```json { "metadata": { "process_exe": "/usr/local/bin/python3.11", "connection_id": "abc124" }, "request": { "method": "GET", "url": "http://localhost:5001/users/1", "authority": "localhost:5001", "protocol": "http1" }, "response": { "status": 200 }, "direction": "ingress", "duration_ms": 1 } ``` **Interpretation:** * Same request, different perspective * Direction: **ingress** (arriving at User Service) * Qtap captured BOTH sides of the call * This is the core of service mesh observability *** ### Example 3: External API Call ```json { "metadata": { "process_exe": "/usr/local/bin/python3.11", "endpoint_id": "httpbin.org", "is_tls": true }, "request": { "method": "GET", "url": "https://httpbin.org/json", "authority": "httpbin.org", "protocol": "http1" }, "response": { "status": 200, "content_type": "application/json" }, "direction": "egress-external", "duration_ms": 1949 } ``` **Interpretation:** * Order Service called external API * Qtap saw inside HTTPS (before TLS encryption) * Much slower (1949ms) than internal calls (1-2ms) * Useful for identifying slow external dependencies *** ## Part 6: Service Mesh Features Demonstrated ### ✅ Service Discovery Qtap automatically discovered: * API Gateway depends on User Service (port 5001) * API Gateway depends on Order Service (port 5002) * Order Service depends on httpbin.org (external) **No configuration required.** Just observe traffic and map dependencies. *** ### ✅ Request Tracing Follow a single request through the system: ``` 1. Client → API Gateway [2ms] GET /api/user-with-orders/2 2. API Gateway → User Service [1ms] GET /users/2 Response: {"id": "2", "name": "Bob"} 3. API Gateway → Order Service [5270ms] ← Slow! GET /orders/2 4. Order Service → httpbin.org [1949ms] GET https://httpbin.org/json Total: ~7.2 seconds ``` **Insight:** Order Service is slow because it waits for external API. Consider caching or async processing. *** ### ✅ Error Monitoring Errors captured with full details: ```bash docker logs qtap-mesh-lite 2>&1 | grep '"status":404' | jq . ``` **Output shows:** * Which service returned 404 (User Service) * Request that caused it (GET /users/999) * Full response body with error message * Captured at `level: full` due to rulekit rule *** ### ✅ Latency Tracking Calculate latency percentiles from captured data: ```bash docker logs qtap-mesh-lite 2>&1 | \ grep '^{' | \ jq -r 'select(has("duration_ms")) | .duration_ms' | \ sort -n ``` The `grep '^{'` filter keeps only the JSON capture lines so `jq` doesn't choke on INFO prefixes. **Analyze:** * p50 (median): \~2ms for internal calls * p95: \~10ms * p99: \~2000ms (external API calls) **Insight:** Most internal calls are fast (<10ms), but external APIs add significant latency. *** ## Part 7: Comparison - Service Mesh vs Qtap | Feature | Service Mesh (Istio) | Qtap (Service Mesh Lite) | | -------------------------- | ------------------------------- | --------------------------- | | **Observability** | | | | Service discovery | ✅ Yes | ✅ Yes | | Request tracing | ✅ Yes | ✅ Yes (both ingress/egress) | | Error monitoring | ✅ Yes | ✅ Yes (with full capture) | | Latency metrics | ✅ Yes | ✅ Yes (per request) | | Traffic volume | ✅ Yes | ✅ Yes | | TLS inspection | ✅ Yes (via mTLS) | ✅ Yes (pre-encryption) | | **Traffic Control** | | | | Load balancing | ✅ Yes | ❌ No | | Circuit breaking | ✅ Yes | ❌ No | | Retries/timeouts | ✅ Yes | ❌ No | | Traffic splitting (canary) | ✅ Yes | ❌ No | | **Security** | | | | mTLS between services | ✅ Yes | ❌ No | | Authorization policies | ✅ Yes | ❌ No | | **Operational** | | | | Deployment complexity | ⚠️ High | ✅ Low (DaemonSet) | | Resource overhead | ⚠️ High (sidecars on every pod) | ✅ Minimal (one per node) | | Latency impact | ⚠️ Yes (adds \~2-5ms per hop) | ✅ Zero (out-of-band) | | Code changes required | ✅ None | ✅ None | | Control plane | ⚠️ Required (istiod, pilot) | ✅ None | | Configuration complexity | ⚠️ High (CRDs, policies) | ✅ Low (one YAML) | *** ## When to Use Which ### Use Qtap (Service Mesh Lite) When: ✅ **You need observability only** * Understand service dependencies * Track errors and latency * Debug microservice issues ✅ **You want zero latency overhead** * Performance-critical applications * High-throughput services * Latency-sensitive workloads ✅ **You want operational simplicity** * Small team without mesh expertise * Don't want to manage control plane * Avoid sidecar complexity ✅ **You have resource constraints** * Limited cluster resources * Cost-sensitive deployments * Want to minimize overhead *** ### Use a Service Mesh When: ⚠️ **You need traffic control** * Canary deployments and traffic splitting * Circuit breaking for resilience * Retry and timeout policies * Advanced load balancing ⚠️ **You require mTLS** * Zero-trust network requirements * Compliance mandates encryption * Need service-to-service authentication ⚠️ **You need authorization** * Fine-grained access control between services * Policy-based routing ⚠️ **You have dedicated platform team** * Team can manage mesh complexity * Already invested in service mesh skills * Have operational capacity *** ## Part 8: Production Considerations ### Scaling to Many Services This demo uses 3 services. In production with 50+ services: **Qtap scales linearly:** * One qtap per node (DaemonSet in Kubernetes) * Resource usage independent of service count * No per-service configuration needed **Service Mesh complexity grows:** * One sidecar per pod (100 pods = 100 sidecars) * Control plane resource requirements increase * Configuration complexity multiplies *** ### Integration with Observability Tools Qtap outputs JSON. Pipe to your observability stack: **Send to S3 for long-term storage:** ```yaml services: object_stores: - type: s3 bucket: my-mesh-observability region: us-east-1 ``` **Send events to ClickHouse/Elasticsearch:** ```yaml services: event_stores: - type: axiom api_token: $AXIOM_TOKEN dataset: mesh-traffic ``` **Build dashboards:** * Parse JSON logs to extract metrics * Visualize service dependencies * Alert on error rate increases *** ### Kubernetes Deployment For production Kubernetes, deploy qtap as a DaemonSet: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: qtap namespace: qpoint spec: selector: matchLabels: app: qtap template: metadata: labels: app: qtap spec: hostNetwork: true hostPID: true containers: - name: qtap image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 securityContext: privileged: true volumeMounts: - name: sys mountPath: /sys - name: docker-sock mountPath: /var/run/docker.sock - name: config mountPath: /app/config env: - name: TINI_SUBREAPER value: "1" args: - --log-level=info - --log-encoding=console - --config=/app/config/qtap.yaml volumes: - name: sys hostPath: path: /sys - name: docker-sock hostPath: path: /var/run/docker.sock - name: config configMap: name: qtap-config ``` *** ### Use Pod Labels for Service Attribution In Kubernetes, use pod labels to identify services: ```yaml rulekit: macros: - name: is_payment_service expr: src.pod.labels.app == "payment-service" - name: is_auth_service expr: src.pod.labels.app == "auth-service" stacks: mesh_observability: plugins: - type: http_capture config: level: summary rules: - name: "Payment service traffic" expr: is_payment_service() level: headers - name: "Auth service traffic" expr: is_auth_service() level: headers ``` *** ## Part 9: Cleanup ```bash docker compose -f /tmp/docker-compose-mesh-lite.yaml down ``` *** ## Key Takeaways 1. **Service mesh observability without service mesh complexity**: Qtap provides service discovery, request tracing, error monitoring, and latency tracking without sidecars or control planes. 2. **Zero latency impact**: Out-of-band observation means no proxies in the request path. Your services run at full speed. 3. **BOTH sides of every call**: `direction: all` captures egress (leaving Service A) and ingress (arriving at Service B), giving complete visibility. 4. **TLS inspection without certificates**: Qtap sees inside HTTPS at the kernel level before encryption happens. 5. **Operational simplicity**: One DaemonSet per node vs sidecars on every pod. Simple YAML config vs complex CRDs and policies. 6. **Know when you need more**: If you need traffic control (canary, circuit breaking) or mTLS, use a full service mesh. If you just need visibility, Qtap is simpler. *** ## Related Documentation * [HTTPS Header Capture Without Proxies](/guides/qtap-guides/advanced-use-cases/transparent-https-header-capture-without-proxies) - Deep dive on TLS inspection * [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings) - Understanding direction options * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - Rulekit and capture levels * [CI Security Validation](/guides/qtap-guides/advanced-use-cases/ci-security-validation-with-qtap) - Another advanced use case # Qplane Guides Cloud-connected deployment guides for Qtap agents managed through the Qplane control plane at [app.qpoint.io](https://app.qpoint.io). ## What You'll Learn These guides cover: * **Account setup and onboarding** - Getting started with Qplane * **Agent deployment** - Connecting Qtap agents to the control plane * **Dashboard navigation** - Understanding your traffic data * **Configuration management** - Using stacks, plugins, and routing rules * **Team collaboration** - Sharing visibility across your organization ## Available Guides ### [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) ⭐ **The fastest way to see Qpoint's value.** Get from zero to actionable insights in 10 minutes. **What you'll learn:** * Deploy your first agent (10 min) * Navigate dashboards and understand traffic (10 min) * Use powerful features: discovery, security scanners, sensitive data detection (10 min) * Achieve quick wins: attack surface mapping, shadow IT discovery, data exposure risks **Time to complete:** 30 minutes **Skill level:** Beginner **Recommended starting point:** If this is your first time with Qpoint *** ### [Hello World - Your First 30 Minutes](/guides/qplane-guides/qplane-hello-world) Your detailed walkthrough guide after account creation. Learn to: * Understand the default Qplane setup * Add additional agents to your environment * Generate test traffic and explore dashboards * Create custom stacks for different traffic types * Set up domain-specific routing rules **Time to complete:** 30 minutes **Skill level:** Beginner *** ### [MCP Getting Started](/guides/qplane-guides/mcp-getting-started) 🤖 Connect AI assistants to your traffic data via Model Context Protocol (MCP). Query connections, requests, errors, and PII exposure using natural language. **What you'll learn:** * Connect Claude Desktop, Cursor, or other MCP clients * Run your first traffic queries * Common investigation patterns **Time to complete:** 15 minutes **Skill level:** Beginner *** ### [MCP Tools Reference](/guides/qplane-guides/mcp-tools-reference) Complete reference for all MCP query tools — connections, requests, traffic aggregates, issues, and PII detection. Includes filter fields and example queries. **Skill level:** Intermediate *** ### [GitOps Configuration Management](/guides/qplane-guides/gitops-configuration-management) Manage Qplane configuration as code using Git workflows. **Skill level:** Intermediate *** ## Why Choose Qplane? **Centralized Management** * Configure all agents from a single control plane * Changes propagate automatically to connected agents * No YAML editing required **Advanced Analytics** * Real-time traffic dashboards * Vendor and endpoint inventory * Error tracking and alerting * Multi-environment visibility **Team Collaboration** * Share dashboards with team members * Role-based access control (RBAC) * Organization-wide policies *** ## Next Steps **New to Qplane?** Start with the [POC Kick Off Guide](/guides/qplane-guides/poc-kick-off-guide) for the fastest path to value (10 minutes). **Want a detailed walkthrough?** Follow the [Hello World guide](/guides/qplane-guides/qplane-hello-world) for step-by-step instructions (30 minutes). **Need more control?** Check out the [Qtap Guides](/guides/qtap-guides) for standalone deployment with YAML configuration. # POC Kick Off Guide This guide gets you up and running with Qplane's most valuable features. We'll focus on practical wins you can achieve quickly in a dev/testing environment. *** ### What You'll Get **Immediate visibility into:** * Every external API your backend connects to * Which services are talking to which vendors * What kind of data is being sent * Performance metrics and error patterns * Real-time traffic dashboards **Without:** * Code changes * Latency impact * Complex configuration * Standing up additional infrastructure *** ### Part 1: Your First Agent (10 minutes) #### What Qplane Does Qplane is your cloud control panel for managing lightweight agents (Qtap) that capture network traffic at the kernel level. Think of it as having X-ray vision into all your encrypted HTTPS traffic. #### Deploy Your First Agent When you create an account at [app.qpoint.io](https://app.qpoint.io/), the installation wizard walks you through deploying your first agent. You'll get a registration token and installation commands for your environment. The agent starts capturing traffic immediately. No restarts, no configuration files. *** ### Part 2: Understanding Your Dashboard (10 minutes) #### The Overview Page Navigate to **Overview** to see your traffic at a glance: * **Vendors**: External services your apps connect to * **Clients**: Internal services making requests * **Performance**: Connections per second, bytes transferred This is your "command center" view. #### The Inventory Pages **Vendors** (Inventory → Vendors) * Every external API or service you're connecting to * Metrics: request counts, data transferred, error rates * Answers: "What third-party services are we using?" **Endpoints** (Inventory → Endpoints) * Specific API endpoints being called * Example: `api.stripe.com/v1/charges` * Answers: "Which exact APIs are we hitting?" **Clients** (Inventory → Clients) * Your internal services making requests * Process attribution shows which container/app is responsible * Answers: "Who's making this call?" #### The Traffic Pages **Logs** (Traffic → Logs → HTTP Requests) * Real-time feed of every HTTP request * Click any request to see headers, timing, response codes * Perfect for debugging **Usage** (Traffic → Usage) * Performance metrics over time * P99 latency, error rates, bandwidth * Spot patterns and issues {% hint style="info" %} **Want to understand the complete architecture?** See [How It Fits Together](/getting-started/qplane/configuration/how-it-fits-together) to learn about data flow, YAML vs UI, and how all components connect. {% endhint %} *** ### Part 3: Key Features That Make Qplane Powerful (10 minutes) #### 1. Automatic Discovery You don't configure what to monitor. Qplane discovers everything automatically: * New services your team deploys * Shadow IT (services added without approval) * Forgotten integrations still running **Try it:** Deploy a new service that calls an external API. Within minutes, it appears in your Vendors list. #### 2. Process Attribution Every connection shows you exactly which process/container made it. This is huge for: * Multi-tenant environments * Debugging ("which service is hammering this API?") * Security ("who connected to that unknown domain?") **Check it:** Go to Traffic → Logs → Connections. See the "executable" column and click for more details pertaining to traffic source #### 3. Security Monitoring: Scanners & Sensitive Data The Security tab provides two powerful capabilities for understanding and protecting your data flows. **Vulnerability Scanners (Rules-Based Detection)** Navigate to **Security → Overview** to access the scanner setup wizard. Scanners are pre-built detection rules that monitor your connections for security threats and risky patterns across six categories: 1. **Shadow IT & Egress Control**: Detects unmanaged activity, direct-IP/DNS bypass, suspected C2-style behavior, backdoors, covert exfiltration 2. **Data Protection & Privacy**: Catches risky handling of PII/PHI/PCI and company secrets, exposure risks, mishandling indicators 3. **AI & LLM Governance**: Controls data exposure to AI systems, monitors LLM usage, new vendors/endpoints, enforces policies 4. **Third-Party & Compliance Governance**: Discovers vendors, tracks cross-border transfers, produces GDPR/SOC2/PCI/data-residency evidence 5. **Authentication & Credential Security**: Protects tokens and identities, detects misuse, leakage, anomalous issuance, authentication failures 6. **Transport Security & TLS Hygiene**: Enforces encryption standards, identifies deprecated TLS, plaintext channels, weak baselines **Enabling scanners:** 1. Go to **Security** tab 2. Click "Enable Scanners" on the Overview page 3. Select which vulnerability categories to activate (recommend all six initially) 4. Click "Activate" to enable the detection rules 5. The system immediately starts monitoring for these patterns {% hint style="info" %} **Important:** Scanners are different from alerting rules. Both work together to give you complete visibility. {% endhint %} **Sensitive Data Plugin** This feature gives you visibility into what sensitive data is actually flowing through your connections. It answers questions like: * "Are we sending PII to that analytics vendor?" * "Which apps are exposing customer email addresses?" * "Is anyone accidentally sending API keys or credentials?" **How it works:** 1. Navigate to **Security → Sensitive Data** 2. Click "Activate Plugin" to launch the setup wizard 3. Select which data types to detect (20+ types available) 4. The wizard automatically adds the "Scan Payloads & Data" plugin to your stack 5. View results in **Security → Sensitive Data** dashboards **What it detects:** * **Personal Information**: Names, emails, phone numbers, geographic locations, dates of birth * **Financial Data**: Credit cards, crypto addresses, IBAN codes, bank account numbers * **Government IDs**: SSNs, driver's licenses, passport numbers, ITIN numbers * **Technical Data**: IP addresses, URLs, API keys, tokens, credentials * **Medical Data**: Medical license numbers, affiliation groups **Smart scanning approach:** * Samples first 100 requests to each URL for baseline (configurable via "Sample Baseline") * Then samples 10% of ongoing traffic (configurable via "Sample Rate") * Configurable cache size (default: 4096 URLs) and TTL (default: 24 hours) * For production environments, all scanning happens in your environment using Qscan's self hosted docker container **Why this matters for security:** * **Discovery**: Find sensitive data you didn't know was being transmitted * **Risk Assessment**: See exactly what data each vendor receives * **Compliance**: Demonstrate data governance with concrete evidence for GDPR, CCPA, PCI-DSS, HIPAA * **Alerting**: Set up alerts when sensitive data goes to unexpected destinations **Example discoveries teams make:** * Logging service accidentally capturing credit card numbers * Analytics vendor receiving SSNs in URL parameters * API keys being sent to third-party error tracking * Customer PII flowing to non-compliant vendors **Note for dev/testing:** The default setup uses Qpoint's managed scanning service (Qscan at api-s.qpoint.io) for easy preview. For production with real customer data, you'll want to run your own Qscan container in your environment to keep sensitive data internal. #### 4. Zero Configuration Required Your default setup already includes: * Automatic error detection (500s, 401s, 404s, timeouts) * Performance monitoring (latency, throughput) * Traffic reporting to dashboards Everything just works out of the box. #### 5. Team Collaboration **Roles & Permissions** (Settings → Team) * Invite teammates with different access levels * Owner, Administrator, Traffic Manager, Viewer roles * Security Viewer, Log Viewer for specific access needs **Views** (Settings → Views) * Create filtered dashboards for different teams * Example: Payment team sees only payment-related traffic * Filter by container, domain, namespace, or any dimension *** ### Common Workflows #### "Show me everything calling external APIs" 1. Go to **Inventory → Vendors** 2. See the complete list with metrics 3. Click any vendor to see which clients are connecting #### "Debug slow API calls" 1. Go to **Traffic → Usage → Performance** 2. Look for high P99 latency 3. Click through to **Logs** to see specific slow requests 4. Inspect timing breakdown and response details #### "Set up error alerting" 1. Go to **Alerting → Rules** 2. The default Detect Errors plugin already captures errors automatically 3. Create alerting rules to get notified about errors: * Click **+ New Rule** * Choose from templates (High Error Rate, Low Availability, etc.) * Or create custom rules using 50+ metrics 4. Add integrations at **Alerting → Webhooks** (Slack, webhooks, PagerDuty) 5. Alerts fire when your defined thresholds are exceeded **Note:** The Detect Errors plugin (in your default stack) captures error details. Alerting rules (in the Alerting tab) notify you when errors occur. They work together but serve different purposes. #### "Discover security risks" **Unknown vendors (Shadow IT):** 1. Go to **Inventory → Vendors** 2. Look for domains you don't recognize 3. Click through to see which apps are connecting 4. These are your shadow IT discoveries **Sensitive data exposure:** 1. Go to **Security** tab 2. Click "Enable Scanners" if not already enabled 3. Select vulnerability categories (recommend all six) 4. Click "Activate Plugin" on the Sensitive Data page 5. Choose which data types to detect 6. Wait a few minutes for scanning to begin 7. Check **Security → Sensitive Data** to see results 8. Identify PII, credentials, or financial data in transit #### "Find who's using a specific service" 1. Go to **Inventory → Vendors** 2. Find the vendor (e.g., `api.openai.com`) 3. Click it to see all clients connecting 4. See which containers/processes are responsible *** ### What's Actually Happening #### Data Flow (Simplified) 1. **Qtap agent** captures network traffic at kernel level 2. **Connection metadata** (who, what, when, performance) goes to Qplane dashboards 3. **Request/response data** can be stored in your own S3 (for dev/testing, Qpoint provides temporary storage) #### What Gets Captured **Always captured:** * Source (which process/container) * Destination (domain, IP, port) * Performance (latency, bytes, errors) * Timing (when connections happen) **Optionally captured:** * HTTP headers and bodies (configurable) #### Privacy & Security * Metadata flows to Qplane for dashboards * Sensitive payload data stays in your infrastructure when you configure your own storage * Agents run read-only (no traffic modification) * Near-zero latency impact on your apps (out-of-band operation) * Qscan for sensitive data detection runs in your environment *** ### Quick Wins You Can Achieve Today #### Win 1: Discover Your Attack Surface **Time: 5 minutes** Go to Inventory → Vendors. You now have a complete, real-time inventory of every external service your infrastructure connects to. Most teams discover 30-50% more vendors than they knew about. #### Win 2: Identify Shadow IT **Time: 2 minutes** Look for vendors you don't recognize. Filter by "Unknown" if available. These are services your teams added without going through procurement or security review. #### Win 3: Debug Production Issues Faster **Time: Next time something breaks** When an integration fails, go to Traffic → Logs, filter to that vendor, and see the exact requests, responses, and error codes. No more guessing. #### Win 4: Identify Data Exposure Risks **Time: 5 minutes** Go to the Security tab and use the wizards to enable scanners and activate the sensitive data plugin. Within minutes, see what sensitive data your apps are transmitting and what security risks exist. Discover PII, credentials, or financial data you didn't know was being sent to external services. #### Win 5: Set Up Proactive Alerting **Time: 5 minutes** The default error detection rules are already active. Add a Slack webhook at Alerting → Webhooks, and you'll get notified immediately when services start failing or slowing down. *** ### Next Steps #### Add More Agents Deploy agents to other environments: 1. Go to **Settings → Installations** 2. Create new installation (e.g., "Staging", "Dev") 3. Get the registration token 4. Deploy agent with that token 5. All environments visible in one dashboard #### Create Custom Stacks Stacks control how traffic is processed. For now, the default stack does everything you need. When you're ready to customize: 1. Go to **Plugins → Stacks** 2. Create stack for specific use cases (e.g., detailed capture for debugging) 3. Route specific domains to specific stacks **Learn more:** [Stacks & Plugins Guide](/getting-started/qplane/configuration/stacks-and-plugins) for complete plugin reference and advanced configuration #### Set Up Routing Rules Route different domains to different stacks for processing: 1. Go to **Plugins → Routing** 2. Add domain patterns 3. Assign to stacks 4. Example: Capture full payloads from `*.yourdomain.com` for debugging **Learn more:** [Routing Rules](/getting-started/qplane/configuration/stacks-and-plugins#routing-rules) for advanced domain matching patterns *** ### Key Benefits Recap **Centralized Management** * One dashboard for all environments * Configuration changes propagate automatically * No YAML editing required **Automatic Discovery** * No configuration needed * Discovers new services automatically * Complete visibility with zero setup **Team Collaboration** * Role-based access control (RBAC) * Filtered views for different teams * SSO support available **Zero Friction** * Deploy in minutes * No code changes * Near-zero latency impact (out-of-band operation) * No proxy or sidecar overhead **Immediate Value** * See all external dependencies instantly * Debug issues faster with request details * Monitor performance without instrumentation * Alert on errors automatically * Identify sensitive data exposure risks *** ### Common Questions **Q: Do I need to configure what to capture?** A: No. The agent captures everything automatically. You can add filters later if needed. **Q: Will this slow down my applications?** A: No. The agent runs out-of-band and doesn't touch your actual traffic flow. Near-zero latency impact because it observes at the kernel level after encryption/decryption happens. **Q: What happens to sensitive data?** A: In dev/testing with Qpoint Cloud storage, it's temporary and managed by Qpoint. For production, you configure your own S3-compatible object store (Settings → Deploy → Services → Object Stores) so sensitive payload data never leaves your infrastructure. Event metadata (anonymized connection info) goes to Qplane for dashboards. **Q: Can I use this in production?** A: Yes. Many teams run Qplane in production. For production, you'll want to configure your own object storage for sensitive data. **Q: How do I add more team members?** A: Settings → Team → + Invite Member. Choose their role and send invite. **Q: What if I have multiple environments?** A: Create separate installations for each environment. All visible in one dashboard with environment tags. ### You're Ready You now understand: * How Qplane gives you network visibility * How to navigate the dashboard * How to accomplish common tasks * The key features that make it powerful Start exploring your Inventory and Traffic pages. You'll be surprised what you discover about your own infrastructure. # Hello World - Your First 30 Minutes This guide walks you through your first steps with Qplane after deploying your initial agent. You've completed the installation wizard, you see the default setup—now what? Follow this guide to understand what you're seeing, generate test traffic, and start customizing your configuration. **What you'll accomplish:** * Understand the default Qplane setup * Add additional Qtap agents to your environment * Generate test traffic and see it in your dashboards * Customize your first stack and routing rule **Time to complete:** 30 minutes **Prerequisites:** * A Qplane account at [app.qpoint.io](https://app.qpoint.io) with your first agent deployed * A Linux server or Kubernetes cluster to install additional Qtap agents (optional) * Docker, Kubernetes, or Linux with systemd {% hint style="info" %} When you create a Qplane account, the installation wizard guides you through deploying your first Qtap agent. You must complete this initial deployment before you can access the Qplane interface. This guide picks up after that first agent is running. {% endhint %} *** ## Step 1: Understanding Your Default Setup Now that you've deployed your first agent via the installation wizard, let's explore the pre-configured environment that Qplane has set up for you. ### What's Already Configured **Default Stack: "Basic - Reporting and Error Detection"** Navigate to **Plugins → Stacks** to see your default stack. It includes: 1. **Report Usage Plugin** * Sends anonymized metrics to Qplane * Powers your Traffic, Inventory, and Dashboard pages * Required for alerting to work 2. **Detect Errors Plugin** (with 7 pre-configured rules) * App Error (500) * Infrastructure Outage (502, 503, 520-523, 525-526, 530) * Client Error (400) * Authentication Error (401, 403, 407) * Rate Limited (429) * Not Found (404) * Slow Response (configurable threshold) **Default Services** Navigate to **Settings → Deploy → Services** to see where your data goes: * **Event Store**: Qpoint Cloud (Pulse) - anonymized metadata for dashboards * **Object Store**: Qpoint Cloud - captured HTTP request/response details {% hint style="info" %} **For production use**, you should configure your own S3-compatible object store to keep sensitive data within your network. The Qpoint Cloud object store is for preview and testing only. {% endhint %} **Default Routing** Navigate to **Plugins → Routing** to see traffic routing: * **Default Stack**: All traffic uses "Basic - Reporting and Error Detection" * **Endpoints**: None configured yet (we'll add one later) ### What This Setup Does With the default configuration, Qtap will: * ✅ Capture all outbound (egress) HTTP/HTTPS traffic * ✅ Send anonymized connection metadata to Qplane dashboards * ✅ Capture full request/response details when errors occur (4xx, 5xx status codes) * ✅ Upload error details to Qpoint Cloud object store * ✅ Show traffic in real-time on your dashboards *** ## Step 2: Add Additional Qtap Agents (Optional) You already deployed your first agent during account creation. If you want to add more agents to monitor additional servers or clusters, follow these steps. {% hint style="info" %} **Skip this step** if you only want to explore Qplane with your existing agent. You can always add more agents later. {% endhint %} ### Create a Registration Token 1. In Qplane, navigate to **Settings → Environments** 2. Click **+ Installation** 3. Enter a name (e.g., "Production Cluster" or "Staging Environment") 4. Copy the generated token—you'll need it for installation {% hint style="warning" %} Treat this token as a secret! It provides access to your organization. Store it securely. {% endhint %} ### Choose Your Installation Method {% tabs %} {% tab title="Docker" %} **Run the preflight check** (optional but recommended): ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` **Install Qtap:** ```bash docker run -d \ --name qtap \ --user 0:0 \ --privileged \ --cap-add CAP_BPF \ --cap-add CAP_SYS_ADMIN \ --pid=host \ --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --registration-token=YOUR_TOKEN_HERE \ --log-level=info \ --log-encoding=console ``` Replace `YOUR_TOKEN_HERE` with your registration token. **Verify it's running:** ```bash docker logs qtap 2>&1 | head -20 ``` You should see initialization logs showing the agent connecting to Qplane. {% endtab %} {% tab title="Kubernetes (Helm)" %} **Add the Qpoint Helm repository:** ```bash helm repo add qpoint https://qpoint-io.github.io/charts helm repo update ``` **Install Qtap:** ```bash helm install qtap qpoint/qtap \ --set config.registrationToken=YOUR_TOKEN_HERE \ --set config.logLevel=info ``` Replace `YOUR_TOKEN_HERE` with your registration token. **Verify it's running:** ```bash kubectl get pods -l app.kubernetes.io/name=qtap kubectl logs -l app.kubernetes.io/name=qtap --tail=20 ``` {% endtab %} {% tab title="Linux Binary" %} **Download and install:** ```bash curl -sSL https://get.qpoint.io/install | sudo bash ``` **Configure with your token:** ```bash sudo qtap \ --registration-token=YOUR_TOKEN_HERE \ --log-level=info \ --log-encoding=console ``` Replace `YOUR_TOKEN_HERE` with your registration token. **Run as a service** (optional): ```bash sudo systemctl enable qtap sudo systemctl start qtap sudo systemctl status qtap ``` {% endtab %} {% endtabs %} ### Verify Agent Connection 1. In Qplane, navigate to **Settings → Environments** 2. You should see your installation listed *** ## Step 3: Generate Test Traffic Now that Qtap is running, let's generate some test traffic to see it appear in Qplane. ### Generate HTTP Requests **From the same machine where Qtap is running**, generate some test traffic: ```bash # Success request (should appear in dashboards) curl https://httpbin.org/get # Error request (should trigger error detection) curl https://httpbin.org/status/500 # Another error curl https://httpbin.org/status/404 ``` **If using Kubernetes**, exec into any pod: ```bash kubectl run test-curl --rm -it --image=curlimages/curl --restart=Never -- sh # Inside the pod: curl https://httpbin.org/get curl https://httpbin.org/status/500 curl https://httpbin.org/status/404 exit ``` ### Wait for Data to Appear Give it 30-60 seconds for data to flow through the system. *** ## Step 4: Explore Your Dashboards Let's see the traffic you just generated in Qplane's dashboards. ### View Traffic 1. Navigate to **Traffic** (main navigation) 2. You should see entries for `httpbin.org` 3. Click on an entry to see details: * Request method, path, status code * Response time * Source process (curl) * Timestamp ### View Inventory 1. Navigate to **Inventory** 2. You'll see `httpbin.org` listed as a vendor 3. Click on it to see: * Total requests * Error rate * Endpoints called * Clients making requests ### View Error Details 1. Navigate to **Traffic → HTTP Requests** 2. Look for the 500 and 404 status codes you generated 3. Click on an error entry to see: * Full request headers * Full response headers and body * Which process made the request * Timestamp and duration **This is the power of Qplane**: You can see inside HTTPS traffic without proxies or certificate management, and errors are automatically captured with full details. *** ## Step 5: Customize Your First Stack Now let's create a custom stack for a specific use case. We'll create a "Debug Stack" for troubleshooting. ### Create a Debug Stack 1. Navigate to **Plugins → Stacks** 2. Click **+ New Stack** 3. Configure the stack: * **Name**: `Debug Stack` * **Description**: `Full logging and capture for troubleshooting specific integrations` 4. Click into the new stack to add plugins 5. Click **+ Add Plugin** and add: * **Report Usage** (no configuration needed) * **Access Logs**: * Mode: `full` * **Detect Errors**: * Keep default rules enabled 6. Enable **Auto generate release on every change** 7. Click **Save** ### Enable Debug Rule for Full Capture To capture all traffic (not just errors) to your object store: 1. In your Debug Stack, click the **Reliability** tab 2. Find the **Debug** rule 3. Enable it by toggling it on 4. Click **Save** This rule captures all HTTP transactions with full details and uploads them to your object store. ### What This Stack Does The Debug Stack provides maximum visibility: * **Report Usage**: Metrics still flow to dashboards * **Access Logs**: Prints all HTTP transactions to the agent's stdout (visible in logs) * **Detect Errors**: Default rules capture errors and upload to object store * **Debug Rule (when enabled)**: Captures ALL traffic (including successes) and uploads to object store {% hint style="info" %} **Access Logs vs. Detect Errors/Debug**: Access Logs only prints to stdout for viewing in agent logs. To capture and upload traffic to your object store, use the Debug rule under Reliability → Rules. {% endhint %} This is useful when troubleshooting a specific integration—you get complete request/response details both in agent logs (Access Logs) and in your object store (Debug rule). *** ## Step 6: Create Your First Routing Rule Let's assign a specific domain to use your new Debug Stack. Routing in Qplane determines which stack processes traffic from each domain—it's not about proxying or redirecting traffic, but rather which processing pipeline (stack) handles it. ### Add a Routing Rule 1. Navigate to **Plugins → Routing** 2. Scroll to the **Endpoints** section 3. Click **+ Add Endpoint** 4. Configure the endpoint: * **Domain**: `httpbin.org` * **Stack**: `Debug Stack` 5. Click **Save** {% hint style="info" %} **What is Routing?** Routing assigns domains to stacks. When Qtap captures traffic to `httpbin.org`, it will process that traffic using your Debug Stack (with Access Logs and Debug rule enabled) instead of the default stack. All other traffic continues using the default "Basic - Reporting and Error Detection" stack. {% endhint %} ### Test the Routing Now traffic to `httpbin.org` will be processed by your Debug Stack instead of the default stack. **Generate test traffic again:** ```bash curl https://httpbin.org/get ``` **Check the agent logs** to see full request/response details: {% tabs %} {% tab title="Docker" %} ```bash docker logs qtap 2>&1 | grep -A 30 "httpbin.org" ``` {% endtab %} {% tab title="Kubernetes" %} ```bash kubectl logs -l app.kubernetes.io/name=qtap --tail=50 | grep -A 30 "httpbin.org" ``` {% endtab %} {% tab title="Linux Binary" %} ```bash sudo journalctl -u qtap -n 50 | grep -A 30 "httpbin.org" ``` {% endtab %} {% endtabs %} **What you should see in agent logs (from Access Logs plugin):** * Full URL: `https://httpbin.org/get` * Request headers (Host, User-Agent, etc.) * Response headers (Content-Type, etc.) * Response body (the JSON response from httpbin) This is the Access Logs plugin printing to stdout because you routed `httpbin.org` to your Debug Stack. **Additionally (from Debug rule):** The full transaction is also captured and uploaded to your object store, where you can view it later in Qplane under **Traffic → HTTP Requests**. *** ## Step 7: Understanding the Data Flow Now that you've seen it in action, let's understand what's happening: ### When Traffic Occurs 1. **Qtap captures traffic** at the kernel level using eBPF 2. **Traffic flows through the assigned stack**: * For `httpbin.org`: Uses your Debug Stack * For everything else: Uses the default "Basic - Reporting and Error Detection" stack 3. **Plugins process the traffic**: * **Report Usage**: Sends anonymized metadata to Qplane (Pulse) * **Access Logs** (Debug Stack only): Writes full details to stdout * **Detect Errors**: Checks status codes, captures errors 4. **Data reaches destinations**: * **Events** (anonymized metadata): → Qplane dashboards (Traffic, Inventory) * **Objects** (full HTTP details): → Qpoint Cloud object store (for errors) * **Logs** (Access Logs output): → Agent stdout/logs ### What's Anonymized vs. Sensitive **Anonymized event metadata** (goes to Qplane dashboards): * Connection counts * HTTP method, status code * Domain, path (URL structure) * Response time, bytes transferred * Source container/process name * **No headers, no bodies** **Sensitive object data** (goes to object store): * Full request headers * Full request body * Full response headers * Full response body * **Only captured for errors** (with default stack) *** ## What's Next? 🎉 **Congratulations!** You've completed the Qplane Hello World. You now know how to: * ✅ Understand the default Qplane setup * ✅ Install and connect a Qtap agent * ✅ Generate traffic and see it in dashboards * ✅ Create custom stacks for different traffic types * ✅ Route specific domains to specific stacks * ✅ Understand the data flow and what's captured ### Next Steps **Level Up Your Configuration:** * [Stacks & Plugins](/getting-started/qplane/configuration/stacks-and-plugins) - Deep dive into all available plugins * [Alerting](/getting-started/qplane/configuration/alerting) - Set up alerts for errors and latency * [Settings](/getting-started/qplane/configuration/settings) - Configure filters, services, and advanced options **Production Readiness:** * **Configure S3 storage**: Keep sensitive data in your network * Navigate to **Settings → Deploy → Services → Object Stores** * Click **+ Add Object Store** and select **S3 Endpoint** * Configure your MinIO, AWS S3, or GCS bucket * **Filter noisy processes**: Reduce noise from health checks * Navigate to **Settings → Qtap → Filters** * Enable filters for Kubernetes, Container Runtimes, etc. * **Refine error detection**: Customize which errors to capture * Navigate to **Plugins → Stacks → Basic - Reporting and Error Detection** * Click **Reliability** tab to adjust error rules ### Need Help? * Report issues: [GitHub Issues](https://github.com/qpoint-io/qtap/issues) * Check compatibility: Run the [preflight script](https://github.com/qpoint-io/preflight) *** *This guide uses the default Qplane cloud services for simplicity. For production deployments with sensitive data, configure your own S3-compatible object storage to ensure data sovereignty.* # MCP Getting Started The Model Context Protocol (MCP) lets AI assistants query your API traffic directly. Ask questions in natural language and get instant insights about connections, requests, errors, and data exposure. **What you'll accomplish:** * Get your authentication token * Connect an MCP client to Qplane * Run your first traffic query **Time to complete:** 10 minutes **Prerequisites:** * An active Qplane account with at least one Qtap agent deployed * An MCP client (Claude Code, Codex CLI, Cursor, or similar) *** ## What MCP Enables With MCP, you can ask your AI assistant questions like: * "Show me all connections to Stripe in the last hour" * "What errors occurred on the payments endpoint yesterday?" * "Which endpoints are receiving the most traffic?" * "Is there any PII detected in outbound traffic?" The AI assistant translates your questions into structured queries against your Qplane data, giving you instant visibility without navigating dashboards or writing queries manually. {% hint style="info" %} MCP provides read-only access to your traffic data. It cannot modify configurations or affect traffic flow. {% endhint %} *** ## Step 1: Get Your Token 1. Log in to [app.qpoint.io](https://app.qpoint.io) 2. Navigate to **Settings → Deployments** 3. Click **+ New Registration Token** 4. Name it something descriptive (e.g., "MCP Access") 5. Copy the generated **Auth Token** {% hint style="warning" %} Store your token securely. It provides access to your organization's traffic data. {% endhint %} *** ## Step 2: Connect Your MCP Client ### Claude Code ```bash claude mcp add qpoint --transport http https://pulse.qpoint.io/mcp/v1 \ --header "Authorization: Bearer YOUR_TOKEN" ``` Verify the connection: ```bash claude mcp list ``` ### Codex CLI Add to `~/.codex/config.toml`: ```toml [mcp_servers.qpoint] url = "https://pulse.qpoint.io/mcp/v1" bearer_token_env_var = "QPOINT_TOKEN" ``` Set your token: ```bash export QPOINT_TOKEN=your-token-here ``` ### Cursor Add to `~/.cursor/mcp.json`: ```json { "mcpServers": { "qpoint": { "transport": "http", "url": "https://pulse.qpoint.io/mcp/v1", "headers": { "Authorization": "Bearer YOUR_TOKEN" } } } } ``` ### Other Clients Configure with: * **URL:** `https://pulse.qpoint.io/mcp/v1` * **Transport:** Streamable HTTP * **Header:** `Authorization: Bearer YOUR_TOKEN` *** ## Step 3: Your First Query Once connected, try these starter queries: ### Check Connection Status > "What endpoints have I connected to in the last hour?" This verifies the MCP connection is working and shows recent traffic. ### View Traffic Summary > "Give me a traffic summary for today" Returns connection counts, request volumes, and error rates. ### Find Errors > "Show me any 500 errors from today" Lists requests that returned server errors. *** ## Common Use Cases ### Security & Compliance > "What external APIs are my applications calling that aren't from known vendors?" > "Show me requests that failed authentication (401/403 errors) in the past week" > "Is there any PII detected in traffic to third-party APIs?" ### Debugging & Troubleshooting > "What 500 errors occurred on the /api/payments endpoint today?" > "Which endpoints have the slowest average response times?" ### Operations & Monitoring > "Compare traffic volume this week vs last week" > "List all unique endpoints my applications connected to this month" *** ## Tips for Effective Queries 1. **Be specific about time ranges.** "In the last hour" or "yesterday" helps narrow results. 2. **Use endpoint names.** If you know the domain (e.g., `api.stripe.com`), include it. 3. **Start broad, then narrow.** Begin with summaries, then drill into specific issues. 4. **Ask follow-up questions.** The AI maintains context, so you can refine your queries. *** ## Enhance with Claude Code Skills For smarter AI assistance, add a Qpoint skill to your Claude Code configuration. Skills give the AI context about when and how to use Qpoint tools effectively. Create `~/.claude/skills/qpoint/SKILL.md`: ```markdown # Qpoint Traffic Analysis Use the Qpoint MCP tools for API traffic investigation, security audits, and debugging. ## When to Use - Investigating API traffic patterns and connections - Security audits (auth failures, PII exposure, shadow IT) - Debugging (error rates, slow endpoints, failed requests) - Compliance checks (data exposure, vendor inventory) ## Workflow 1. Start with `traffic_summary` to get an overview of activity 2. Use `list_connections` or `list_requests` with filters to drill down 3. Check `list_issues` for policy violations and errors 4. Use `pii_summary` to audit sensitive data exposure ## Query Tips - Always specify time ranges for faster queries (e.g., "last hour", "yesterday") - Use `*Ish` filters (endpointIsh, pathIsh) for fuzzy/substring matching - Start broad, then narrow with additional filters - Use `count_*` tools first to understand data volume before listing ## Common Investigations - **Shadow IT:** "What endpoints are we connecting to that aren't known vendors?" - **Auth failures:** "Show 401/403 errors in the past week" - **Performance:** "Which endpoints have the slowest response times?" - **Data exposure:** "Is there PII in outbound traffic?" ``` Once saved, Claude Code will automatically use this guidance when working with your Qpoint traffic data. *** ## Troubleshooting ### "Authentication required" or 401 errors * Verify your token is correct * Check the token hasn't expired * Ensure the Authorization header format is correct: `Bearer YOUR_TOKEN` ### "No data returned" * Verify your Qtap agents are running and sending data * Check that the time range includes periods with traffic *** ## Next Steps * [**MCP Tools Reference**](/guides/qplane-guides/mcp-tools-reference) — Complete reference for all available query tools * [**Qplane Dashboards**](/guides/qplane-guides/qplane-hello-world) — Visual exploration of your traffic data # MCP Tools Reference Complete reference for all MCP tools available in the Qpoint integration. Use these tools through your MCP client (Claude Desktop, Cursor, etc.) by asking natural language questions—the AI assistant translates your queries into the appropriate tool calls. *** ## Common Parameters All tools share these common input formats: ### Timestamps Use RFC 3339 format for time parameters: ``` 2026-02-12T10:30:00Z ``` * `startTimestamp` defaults to 24 hours ago * `endTimestamp` defaults to now ### Granularity For time-series data, specify aggregation granularity: * `minute` — per-minute resolution * `5minutes` — 5-minute buckets * `15minutes` — 15-minute buckets * `hour` — hourly aggregation * `day` — daily aggregation * `week` — weekly aggregation * `month` — monthly aggregation * `year` — yearly aggregation If omitted, granularity is auto-detected based on the time range. ### Tags Key-value metadata in `key:value` format: ``` env:prod bin:curl protocol:http2 ``` Use `key:` (with colon, no value) to match any resource with that tag key. ### Labels Simple string identifiers for filtering: ``` user-shell is-root direct-ip unencrypted deprecated-tls unknown-tls ``` *** ## Connections Network connections observed by Qtap. Each connection represents a TCP session between your application and an external endpoint. ### Available Tools | Tool | Description | | ----------------------- | --------------------------------- | | `count_connections` | Total connection count | | `list_connections` | List individual connections | | `get_connection` | Get a specific connection by ID | | `histogram_connections` | Connection counts over time | | `connection_frequency` | Most frequent connection patterns | | `connection_endpoints` | Endpoints by connection count | | `connection_attr_vals` | Distinct values for a field | | `connection_tags` | Available tag keys | | `connection_tag_vals` | Values for a specific tag | ### Fields | Field | Description | | ------------------------- | -------------------------------------------------- | | `connectionId` | Unique connection identifier | | `endpointId` | Destination domain (e.g., `api.example.com`) | | `direction` | `egress-external`, `egress-internal`, or `ingress` | | `timestamp` | Connection start time | | `finalized` | Whether the connection is complete | | `tags[]` | Key-value metadata | | `labels[]` | String labels | | `sourceAddress` | Origin IP address | | `sourcePort` | Origin port | | `sourceExe` | Path to the executable | | `sourceHostname` | Hostname of the source machine | | `sourceSystemUserId` | UID of the process owner | | `destinationAddress` | Peer endpoint IP | | `destinationPort` | Peer endpoint port | | `bytesSent` | Bytes sent | | `bytesReceived` | Bytes received | | `tlsVersion` | TLS version used | | `tlsProbeIntrospected` | Whether TLS was successfully hooked | | `tlsProbeTypesDetected[]` | TLS libraries detected (`openssl`, `gotls`, etc.) | | `qpointAgent` | Agent identifier | | `instanceId` | Instance identifier | | `qpointHostname` | Agent hostname | ### Filter Fields **Exact match:** `endpointId`, `connectionId`, `direction`, `qpointAgent`, `sourceAddress`, `sourcePort`, `sourceExe`, `sourceHostname`, `destinationAddress`, `destinationPort`, `tlsVersion` **Fuzzy search:** `endpointIsh` (substring match on endpoint) ### Example Queries > "List all connections to stripe.com in the last hour" > "How many connections used TLS 1.2 yesterday?" > "Show connections from executable /usr/bin/curl" > "What endpoints did hostname prod-server-01 connect to?" *** ## Requests Individual HTTP requests observed within connections. Provides detailed request/response information including methods, paths, status codes, and timing. ### Available Tools | Tool | Description | | ---------------------------- | ------------------------------------ | | `count_requests` | Total request count | | `list_requests` | List individual requests | | `get_request` | Get a specific request by ID | | `histogram_requests` | Request counts over time | | `request_frequency` | Most frequent request patterns | | `request_endpoints` | Endpoints by request count | | `request_attr_vals` | Distinct values for a field | | `request_tags` | Available tag keys | | `request_tag_vals` | Values for a specific tag | | `request_duration_aggregate` | Avg/min/max duration statistics | | `request_duration_line` | Duration metrics over time | | `request_tag_count` | Count distinct tag value occurrences | | `request_tag_line` | Tag value trends over time | ### Fields | Field | Description | | ----------------- | --------------------------------- | | `requestId` | Unique request identifier | | `connectionId` | Parent connection ID | | `endpointId` | Destination domain | | `vendorId` | Vendor identifier | | `direction` | Traffic direction | | `timestamp` | Request time | | `url` | Full request URL | | `path` | URL path | | `method` | HTTP method (`GET`, `POST`, etc.) | | `status` | HTTP status code | | `duration` | Request duration in microseconds | | `contentType` | Response content type | | `category` | Request category | | `agent` | User agent string | | `tags[]` | Key-value metadata | | `authTokenMask` | Masked auth token | | `authTokenHash` | Auth token hash | | `authTokenSource` | Where auth token was found | | `authTokenType` | Type of auth token | | `bytesSent` | Request body size | | `bytesReceived` | Response body size | ### Filter Fields **Exact match:** `endpointId`, `connectionId`, `vendorId`, `direction`, `method`, `path`, `contentType`, `agent`, `status`, `authTokenMask`, `authTokenHash`, `authTokenSource`, `authTokenType`, `dataType` **Fuzzy search:** `endpointIsh`, `vendorIsh`, `pathIsh` ### Example Queries > "Show me all POST requests to api.example.com/v1/payments" > "What requests returned 500 errors today?" > "List requests with content-type application/json" > "Which API paths have the highest request volume?" *** ## Traffic Aggregated traffic metrics across connections, requests, and issues. Provides high-level counts, performance, availability, and bandwidth metrics with breakdowns by location and geography. ### Available Tools **Counts & Summaries:** | Tool | Description | | ------------------------------ | --------------------------- | | `traffic_count` | Total traffic count | | `traffic_duration` | Traffic duration statistics | | `traffic_availability` | Availability metrics | | `traffic_summary` | Overall traffic summary | | `traffic_summary_performance` | Performance summary | | `traffic_summary_availability` | Availability summary | | `traffic_summary_bandwidth` | Bandwidth summary | **Time Series:** | Tool | Description | | --------------------------- | ------------------------------ | | `traffic_line` | Traffic metrics over time | | `traffic_line_performance` | Performance metrics over time | | `traffic_line_availability` | Availability metrics over time | | `traffic_line_bandwidth` | Bandwidth metrics over time | **Location & Geographic:** | Tool | Description | | -------------------------------- | ----------------------------- | | `traffic_locations` | Traffic breakdown by location | | `traffic_locations_performance` | Performance by location | | `traffic_locations_availability` | Availability by location | | `traffic_locations_bandwidth` | Bandwidth by location | | `traffic_geo_countries` | Traffic by country | | `traffic_geo_regions` | Traffic by region | | `traffic_geo_cities` | Traffic by city | ### Time-Series Group Values **Performance metrics:** `connections_per_second`, `connections_per_minute`, `requests_per_second`, `requests_per_minute`, `avg_duration`, `max_duration`, `p_99_duration`, `p_95_duration`, `p_50_duration` **Availability metrics:** `connections`, `requests`, `issues`, `issues_per_second`, `issues_per_minute`, `avg_availability`, `p_99_availability`, `p_95_availability` **Bandwidth metrics:** `total_bandwidth`, `total_bytes_received`, `total_bytes_sent`, `bytes_received_per_second`, `bytes_sent_per_second`, `bytes_received_per_minute`, `bytes_sent_per_minute`, `p_99_bytes_received`, `p_99_bytes_sent`, `max_bytes_received`, `max_bytes_sent` ### Filter Fields Combines connection, request, and issue filters: **Exact match:** `endpointId`, `vendorId`, `connectionId`, `direction`, `qpointAgent`, `sourceAddress`, `sourcePort`, `sourceExe`, `sourceHostname`, `destinationAddress`, `destinationPort`, `tlsVersion`, `method`, `path`, `contentType`, `agent`, `status`, `authTokenMask`, `authTokenHash`, `authTokenSource`, `authTokenType`, `dataType`, `error`, `mitigated` **Fuzzy search:** `endpointIsh`, `vendorIsh`, `pathIsh`, `errorIsh` ### Example Queries > "Give me a traffic summary for the past week" > "Show requests per minute for api.stripe.com over the last 24 hours" > "What's the p95 latency for the /api/checkout endpoint?" > "Which countries is my API traffic going to?" > "Compare bandwidth usage between staging and production" *** ## Issues API issues detected by policy rules—schema violations, auth failures, rate limits, errors, and more. ### Available Tools | Tool | Description | | ------------------ | ---------------------------- | | `count_issues` | Total issue count | | `list_issues` | List individual issues | | `histogram_issues` | Issue counts over time | | `issue_frequency` | Most frequent issue patterns | | `issue_endpoints` | Endpoints by issue count | | `issue_errors` | Error types and counts | | `issue_attr_vals` | Distinct values for a field | ### Fields | Field | Description | | --------------------- | ----------------------------------- | | `timestamp` | When the issue was detected | | `connectionId` | Parent connection ID | | `endpointId` | Destination domain | | `vendorId` | Vendor identifier | | `requestId` | Associated request ID | | `direction` | Traffic direction | | `error` | Error message | | `url` | Full request URL | | `path` | URL path | | `method` | HTTP method | | `status` | HTTP status code | | `mitigated` | Whether the issue was mitigated | | `tags[]` | Key-value metadata | | `triggerConditions[]` | Conditions that triggered the issue | | `triggerReasons[]` | Reasons for the trigger | ### Filter Fields **Exact match:** `endpointId`, `vendorId`, `connectionId`, `direction`, `method`, `status`, `path`, `error`, `mitigated` **Fuzzy search:** `endpointIsh`, `vendorIsh`, `pathIsh`, `errorIsh` ### Example Queries > "How many 500 errors occurred today?" > "Show me all rate limit issues (429 status) from the past week" > "Which endpoints have the most authentication failures?" > "List unmitigated issues for api.example.com" > "What error types are we seeing on the payments endpoint?" *** ## PII Personally Identifiable Information detected in API traffic. Identifies sensitive data exposure including emails, phone numbers, credit cards, and more. ### Available Tools | Tool | Description | | --------------- | ----------------------------- | | `count_pii` | Total PII detection count | | `pii_summary` | Summary of PII types detected | | `histogram_pii` | PII detections over time | | `pii_frequency` | Most frequent PII patterns | | `pii_attr_vals` | Distinct values for a field | ### Fields | Field | Description | | -------------- | ---------------------------------------------------------- | | `entityType` | Type of PII (`email`, `phone_number`, `credit_card`, etc.) | | `score` | Confidence score of the detection | | `endpointId` | Destination domain | | `vendorId` | Vendor identifier | | `connectionId` | Parent connection ID | | `direction` | Traffic direction | ### Filter Fields **Exact match:** `endpointId`, `vendorId`, `connectionId`, `direction`, `entityType`, `score` **Fuzzy search:** `endpointIsh`, `vendorIsh` ### Example Queries > "Is there any PII detected in outbound traffic?" > "Show me all credit card detections from the past week" > "Which endpoints have the most PII exposure?" > "List high-confidence (score > 0.9) email detections" > "How many phone numbers were detected in traffic to third-party APIs?" *** ## Tips for Effective Querying ### Use `limit` for Large Results List operations return full objects and can be large. Use `limit` to control result size: > "List the last 10 requests to stripe.com" ### Start with Counts Before listing detailed records, get a count to understand the volume: > "How many connections to api.example.com occurred today?" Then refine: > "List the 20 most recent connections to api.example.com" ### Use `attr_vals` for Discovery Find what values exist for a field before filtering: > "What unique endpoints have connections?" > "What HTTP methods are being used?" ### Combine Filters Narrow results by combining multiple criteria: > "Show POST requests to /api/payments that returned 500 errors in the last hour" ### Leverage Fuzzy Search Use `*Ish` filters when you don't know the exact value: > "Find traffic to endpoints containing 'stripe'" Uses `endpointIsh` for substring matching. ### Time Range Best Practices * Be specific: "in the last hour" vs "recently" * Use relative ranges for recent data: "today", "yesterday", "past week" * Use absolute timestamps for precise investigation: "between 2pm and 3pm on February 15th" # GitOps Configuration Management Manage Qtap configuration through Git while using Qplane for visualization, search, and dashboards. *** ## Why This Approach **GitOps benefits:** * Version-controlled configuration with full audit trail * Pull request reviews for config changes * Reproducible deployments across environments * Rollback capability through git history * Infrastructure-as-code workflows (ArgoCD, Flux, etc.) **Qplane benefits:** * Real-time dashboards and traffic visualization * Object storage search and retrieval * Alerting and issue detection * No infrastructure to manage for the control plane **The tradeoff:** Configuration lives in git, not Qplane's UI. Changes made in Qplane won't affect your deployed agents. *** ## How It Works ``` ┌─────────────────────────────────────────────────────────────────┐ │ Your Workflow │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. Configure in Qplane UI │ │ ↓ │ │ 2. Export snapshot (YAML) │ │ ↓ │ │ 3. Commit to git repository │ │ ↓ │ │ 4. Deploy via GitOps (Helm, ArgoCD, etc.) │ │ ↓ │ │ 5. Qtap runs with static config │ │ ↓ │ │ 6. Events flow to Qplane for visualization │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` Qtap agents run with your git-managed configuration but still send telemetry to Qplane. You get GitOps workflow control with Qplane's visualization capabilities. *** ## Step 1: Configure in Qplane Start by setting up your configuration in Qplane's UI: 1. Log in to [app.qpoint.io](https://app.qpoint.io/) 2. Configure your stacks, plugins, and capture rules 3. Set up error detection rules 4. Configure PII/sensitive data monitors 5. Define endpoint-specific behaviors 6. Set up tagging for your environment Use the UI to experiment and validate your configuration before committing to git. *** ## Step 2: Export Your Configuration Snapshot Once your configuration is ready: 1. Navigate to **Settings → Snapshot** 2. Copy the YAML configuration The snapshot includes everything: * Services (event stores, object stores) * Stacks and plugins configuration * Tag extraction rules * Tap settings (direction, filters, endpoints) **Example snapshot:** ```yaml version: 2 services: event_stores: - id: qpoint type: pulse url: https://api-pulse.qpoint.io token: type: env value: REGISTRATION_TOKEN object_stores: - id: my_s3_storage type: s3 endpoint: s3.amazonaws.com bucket: qtap-objects region: us-east-1 access_url: https://s3.amazonaws.com/{{BUCKET}}/{{DIGEST}} insecure: false access_key: type: env value: S3_ACCESS_KEY secret_key: type: env value: S3_SECRET_KEY stacks: default: plugins: - type: report_usage - type: qscan config: monitors: - type: CREDIT_CARD record_value: false - type: US_SSN record_value: false - type: detect_errors config: rules: - name: App Error trigger_status_codes: - '500' report_as_issue: true record_req_headers: true record_res_body: true tap: direction: egress-external ignore_loopback: true audit_include_dns: false http: stack: default filters: groups: - qpoint - container-runtimes ``` {% hint style="info" %} The `pulse` event store with your `REGISTRATION_TOKEN` is what connects Qtap to Qplane. Events and metrics flow to Qplane for visualization even though configuration is managed through git. {% endhint %} **How object storage works with Qplane:** When you configure your own S3 object store, the `access_url` tells Qplane how to link to your stored objects. When you investigate a connection in the Qplane UI that was captured by a rule (like `detect_errors`), Qplane renders a direct link to your S3 using this URL template. For example, with `access_url: http://100.75.117.58:3904/qpoint/{{DIGEST}}`, clicking on a captured request in Qplane opens your object directly from your storage. Your sensitive HTTP payloads stay in your infrastructure while Qplane provides the search and navigation interface. *** ## Step 3: Secure Your Secrets The snapshot references secrets via environment variables - it doesn't contain the actual values: ```yaml token: type: env value: REGISTRATION_TOKEN # References env var, not the actual token ``` **This means the snapshot is safe to commit to git.** But you need to provision the actual secret values separately using your secret management solution. ### Secrets You'll Need | Secret | Purpose | Where to Find | | -------------------- | ----------------------------------------- | -------------------------------------------------------------- | | `REGISTRATION_TOKEN` | Connects Qtap to Qplane for telemetry | **Settings → Environments → Manage → Copy Registration Token** | | `S3_ACCESS_KEY` | Object store authentication (if using S3) | Your S3 provider | | `S3_SECRET_KEY` | Object store authentication (if using S3) | Your S3 provider | ### With External Secrets Operator [External Secrets Operator](https://external-secrets.io/) syncs secrets from your vault to Kubernetes: ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: qtap-secrets namespace: qpoint spec: refreshInterval: 1h secretStoreRef: name: vault-backend # Your SecretStore kind: SecretStore target: name: qtap-secrets data: - secretKey: REGISTRATION_TOKEN remoteRef: key: qpoint/qtap property: registration-token - secretKey: S3_ACCESS_KEY remoteRef: key: qpoint/qtap property: s3-access-key - secretKey: S3_SECRET_KEY remoteRef: key: qpoint/qtap property: s3-secret-key ``` ### With Sealed Secrets [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) encrypts secrets for safe git storage: ```bash # Encrypt your secrets (safe to commit the output) kubectl create secret generic qtap-secrets \ --from-literal=REGISTRATION_TOKEN='your-token' \ --from-literal=S3_ACCESS_KEY='your-key' \ --from-literal=S3_SECRET_KEY='your-secret' \ --dry-run=client -o yaml | kubeseal -o yaml > sealed-qtap-secrets.yaml ``` Commit `sealed-qtap-secrets.yaml` to git. The Sealed Secrets controller decrypts it in-cluster. ### With HashiCorp Vault Agent If using Vault Agent Injector, annotate your Qtap deployment to inject secrets: ```yaml annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "qtap" vault.hashicorp.com/agent-inject-secret-config: "qpoint/qtap" vault.hashicorp.com/agent-inject-template-config: | {{- with secret "qpoint/qtap" -}} export REGISTRATION_TOKEN="{{ .Data.data.registration_token }}" export S3_ACCESS_KEY="{{ .Data.data.s3_access_key }}" export S3_SECRET_KEY="{{ .Data.data.s3_secret_key }}" {{- end -}} ``` ### Referencing Secrets in Helm Once your secrets exist in the cluster, reference them in your Helm deployment: ```yaml # values.yaml extraEnv: - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: name: qtap-secrets key: REGISTRATION_TOKEN - name: S3_ACCESS_KEY valueFrom: secretKeyRef: name: qtap-secrets key: S3_ACCESS_KEY - name: S3_SECRET_KEY valueFrom: secretKeyRef: name: qtap-secrets key: S3_SECRET_KEY ``` {% hint style="success" %} **GitOps-friendly pattern:** Store the encrypted/external secret reference in git alongside your snapshot. The actual secret values live in your vault, never in git. {% endhint %} *** ## Step 4: Commit Snapshot to Git Save the exported snapshot directly to your infrastructure repository: ``` infrastructure/ ├── qtap/ │ └── qtap-config.yaml # Your exported snapshot (use as-is) ``` {% hint style="warning" %} **Don't modify the snapshot.** The snapshot from Qplane is your source of truth. Editing it manually defeats the purpose of using Qplane for configuration and introduces drift between what Qplane shows and what's deployed. {% endhint %} **Deploying with Helm:** The Qtap Helm chart accepts your config file directly via `--set-file`: ```bash helm install qtap qpoint/qtap \ -n qpoint \ --create-namespace \ --set-file config=./qtap-config.yaml ``` The chart creates the necessary ConfigMap from your file automatically. For complete Helm installation instructions, see: * [Qtap Helm Chart](/getting-started/qtap/installation/helm-chart) *** ## Step 5: Deploy via GitOps ### With ArgoCD ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: qtap namespace: argocd spec: project: default source: repoURL: https://github.com/your-org/infrastructure targetRevision: main path: qtap/prod helm: releaseName: qtap fileParameters: - name: config path: qtap-config.yaml destination: server: https://kubernetes.default.svc namespace: qpoint syncPolicy: automated: prune: true selfHeal: true ``` ### With Flux ```yaml apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: qtap namespace: qpoint spec: interval: 5m chart: spec: chart: qtap sourceRef: kind: HelmRepository name: qpoint namespace: flux-system valuesFrom: - kind: ConfigMap name: qtap-config valuesKey: qtap-config.yaml ``` ### Manual Helm ```bash helm upgrade --install qtap qpoint/qtap \ -n qpoint \ --set-file config=./qtap-config.yaml ``` *** ## Configuration Drift {% hint style="warning" %} **Important:** Once you deploy via GitOps, changes made in Qplane's UI will NOT affect your running agents. Your git repository is the source of truth. {% endhint %} **To update configuration:** 1. Make changes in Qplane UI 2. Test and validate in the Qplane interface 3. Export a new snapshot 4. Replace the config in your git repository 5. Submit a pull request for review 6. Merge and let GitOps sync the changes {% hint style="danger" %} **Avoid editing the YAML directly in git.** Manual edits create drift between Qplane's view and your deployed configuration. Always make changes in Qplane first, then export. {% endhint %} *** ## User Permissions To prevent accidental configuration drift from UI changes: 1. Navigate to **Settings → Team Members** 2. Set most users to **Viewer** role 3. Reserve **Admin** or **Editor** roles for config managers Viewers can: * Access all dashboards and visualizations * Search and view captured objects * See alerts and issues Viewers cannot: * Modify stacks or plugins * Change capture rules * Alter endpoint configurations This ensures the git repository remains the authoritative source while your team still benefits from Qplane's visualization. *** ## What You Get from Qplane Even with GitOps-managed configuration, Qplane provides: | Feature | Description | | ----------------------------- | ----------------------------------------------------------- | | **Traffic Dashboards** | Real-time visibility into all captured traffic | | **Vendor Inventory** | Automatic discovery of external APIs | | **Object Search & Retrieval** | Find captured payloads and click through to your S3 storage | | **Issue Detection** | Alerts based on your `detect_errors` rules | | **PII Monitoring** | Results from `qscan` plugin analysis | | **Usage Metrics** | Performance data from `report_usage` plugin | The `pulse` event store streams all this data to Qplane regardless of how the configuration was deployed. {% hint style="info" %} **Why Qplane for object search?** When Qtap captures a request/response to your S3, Qplane indexes the metadata and provides a searchable interface. Click any captured connection to open the full payload directly from your storage via the `access_url` you configured. Without Qplane, you'd need to build your own search and retrieval tooling. {% endhint %} *** ## Environment-Specific Configuration If you need different configurations per environment (dev, staging, prod), manage them as separate snapshots in Qplane: 1. Create separate Qplane environments or organizations 2. Configure each environment appropriately in the UI 3. Export a snapshot for each environment 4. Store each snapshot in your git repository under environment-specific paths ``` infrastructure/ ├── qtap/ │ ├── dev/ │ │ └── qtap-config.yaml # Snapshot from dev environment │ ├── staging/ │ │ └── qtap-config.yaml # Snapshot from staging environment │ └── prod/ │ └── qtap-config.yaml # Snapshot from prod environment ``` This keeps each environment's configuration as a complete, unmodified snapshot from Qplane rather than trying to patch or overlay configurations. *** ## Workflow Summary | Step | Where | Action | | ---- | --------------- | ----------------------------------- | | 1 | Qplane UI | Configure and experiment | | 2 | Qplane Settings | Export snapshot | | 3 | Vault | Store secrets (token, S3 keys) | | 4 | Git | Commit snapshot + secret references | | 5 | CI/CD | Review via pull request | | 6 | GitOps | Deploy to clusters | | 7 | Qplane | Monitor and visualize | *** ## What's Next * [Stacks & Plugins Reference](/getting-started/qplane/configuration/stacks-and-plugins) - Detailed plugin configuration * [Object Storage Configuration](/getting-started/qplane/configuration/object-storage) - S3 setup for captured payloads * [Alerting](/getting-started/qplane/configuration/alerting) - Configure notifications for issues # Devtools Guides QTap DevTools is a browser-based interface for real-time HTTP/S traffic inspection. It provides Chrome DevTools-like visibility into your server's network activity, showing you exactly what your applications are sending and receiving - including inside HTTPS. DevTools is designed for **single-host emergency troubleshooting**. When you need to debug a specific server right now, enable DevTools and watch traffic flow in real-time. No service restarts required. ## Available Guides | Guide | Description | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [Getting Started with DevTools](/guides/devtools-guides/getting-started-with-devtools) | Install Qtap, enable DevTools, and see your first traffic in 5 minutes | | [DevTools Interface Guide](/guides/devtools-guides/devtools-interface-guide) | Detailed walkthrough of the DevTools UI - tabs, filters, and features | | [DevTools API](/guides/devtools-guides/devtools-api) | Programmatic access to the real-time event stream for scripting and tooling | | [AI-Assisted Troubleshooting with Ollama](/guides/devtools-guides/ai-troubleshooting-with-devtools) | Local LLM setup with Qwen3 + Ollama for traffic analysis | | [DevTools MCP Server](/guides/devtools-guides/devtools-mcp-server) | Connect Codex CLI, ChatGPT, or Claude Code to DevTools via MCP | | [Traffic Analysis MCP Server](/guides/devtools-guides/traffic-analysis-mcp-server) | Type inference, test generation, and schema validation from captured traffic | | [Server-Side Event Filtering](/guides/devtools-guides/server-side-event-filtering) | Subscribe to specific event types with rulekit expressions | | [Enhanced MCP with Server-Side Filtering](/guides/devtools-guides/enhanced-mcp-troubleshooting) | Optimized MCP server using server-side filters for targeted AI troubleshooting | ## Key Features * **See inside HTTPS** without proxies or certificates * **Process attribution** - know which binary/container made each request * **Real-time streaming** - watch traffic as it happens * **No restarts required** - enable with a flag while apps are running * **Local operation** - all data stays on your server ## Quick Start ```bash # Install Qtap curl -s https://get.qpoint.io/install | sudo sh # Run with DevTools enabled sudo qtap --enable-dev-tools # Access in your browser open http://localhost:10001/devtools ``` # Getting Started with DevTools Get DevTools running and see your first HTTP/S traffic in under 5 minutes. DevTools is a browser-based interface for real-time HTTP/S traffic inspection. Think Chrome DevTools Network tab, but for your Linux server. * See inside HTTPS without proxies or certificates * Know which process/container/pod made each request * Full request/response details including headers and bodies * No application restarts or code changes required
*** ## Prerequisites * Linux host with kernel 5.10+ * Root/sudo access * A browser to access the DevTools UI {% hint style="info" %} **Verify compatibility:** Run the preflight check to confirm your system supports eBPF: {% code overflow="wrap" %} ```bash curl -sSL https://github.com/qpoint-io/preflight/releases/latest/download/preflight.sh | sudo bash ``` {% endcode %} {% endhint %} *** ## Step 1: Install Qtap {% tabs %} {% tab title="Linux Binary" %} ```bash curl -s https://get.qpoint.io/install | sudo sh ``` This installs the `qtap` binary to your system path. For manual installation, see [Linux Binary Installation](/getting-started/qtap/installation/linux-binary). {% endtab %} {% tab title="Docker" %} ```bash docker pull us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` For detailed Docker setup, see [Docker Container Installation](/getting-started/qtap/installation/docker-container). {% endtab %} {% tab title="Kubernetes" %} For Kubernetes deployments, install via Helm: ```bash helm repo add qpoint https://charts.qpoint.io helm repo update helm install qtap qpoint/qtap --set devtools.enabled=true ``` For detailed Kubernetes setup, see [Helm Chart Installation](/getting-started/qtap/installation/helm-chart). {% endtab %} {% endtabs %} *** ## Step 2: Run with DevTools Enabled {% tabs %} {% tab title="Linux Binary" %} ```bash sudo qtap --enable-dev-tools ``` Qtap starts capturing traffic immediately. DevTools is now accessible. {% endtab %} {% tab title="Docker" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \ --log-level=info ``` The `-e ENABLE_DEV_TOOLS=true` environment variable enables the DevTools UI. {% endtab %} {% tab title="Kubernetes" %} If you installed with `--set devtools.enabled=true`, DevTools is already enabled. Port-forward to access DevTools locally: ```bash kubectl port-forward pod/qtap-xxxxx 10001:10001 ``` Replace `qtap-xxxxx` with your actual pod name from `kubectl get pods`. {% endtab %} {% endtabs %} *** ## Step 3: Access DevTools Open your browser and navigate to: ``` http://localhost:10001/devtools ``` You should see the DevTools interface with three tabs: **Processes**, **Connections**, and **Requests**. For remote servers, use SSH port forwarding: ```bash # From your local machine ssh -L 10001:localhost:10001 user@remote-server # Then open in your local browser open http://localhost:10001/devtools ``` *** ## Step 4: Verify It Works Generate some test traffic to confirm DevTools is capturing: ```bash # In a separate terminal on the same host curl https://httpbin.org/get curl -X POST https://httpbin.org/post -d "test=data" ``` Switch to the **Requests** tab in DevTools. You should see: * The `GET` and `POST` requests to `httpbin.org` * Status codes (200) * The process that made the request (`curl`) * Request timing Click on any request to see: * Full request headers * Response body * A copy-pasteable cURL command {% hint style="success" %} **Success!** If you see the requests appear in DevTools, everything is working. You're now capturing HTTP/S traffic with full visibility. {% endhint %} *** ## Quick Tour DevTools has three tabs:
TabWhat It Shows
ProcessesReal-time inventory of all processes on the host
ConnectionsNetwork connections with process attribution
RequestsIndividual HTTP transactions with full details
For a detailed walkthrough of the interface, see the [DevTools Interface Guide](/guides/devtools-guides/devtools-interface-guide). *** ## Next Steps * [**DevTools Interface Guide**](/guides/devtools-guides/devtools-interface-guide) - Learn all the features of the DevTools UI * [**Storage Configuration**](/getting-started/qtap/configuration/storage-configuration) - Save captured traffic to S3 for permanent storage * [**Traffic Capture Settings**](/getting-started/qtap/configuration/traffic-capture-settings) - Filter and customize what traffic is captured * [**Qplane**](/getting-started/qplane) - Centralized management for multi-host deployments *** ## Troubleshooting ### DevTools Not Loading 1. **Check Qtap is running:** * Binary: `ps aux | grep qtap` * Docker: `docker ps | grep qtap` 2. **Verify DevTools is enabled:** * Binary: Must include `--enable-dev-tools` flag * Docker: Must include `-e ENABLE_DEV_TOOLS=true` 3. **Check the port:** * Ensure nothing else is using port 10001 * Verify with `curl http://localhost:10001/devtools` ### No Traffic Appearing 1. **Qtap must be running BEFORE traffic is generated** - Restart qtap, then make requests 2. **Check traffic direction** - Default captures egress (outbound) traffic 3. **Verify the event stream is working** - Check the raw API to confirm events are flowing: ```bash curl -sN http://localhost:10001/devtools/api/events | head -20 ``` You should see SSE events like `event: process.started` and `event: connection.opened`. If events are streaming but the UI is empty, try refreshing your browser. # Devtools Interface Guide QTap DevTools provides a browser-based interface for inspecting HTTP/S traffic in real-time. The interface is organized into three tabs, each providing a different level of visibility into your system's network activity. Access DevTools at `http://localhost:10001/devtools` after enabling with `--enable-dev-tools` or `ENABLE_DEV_TOOLS=true`.
*** ### Global Controls The toolbar at the top of every tab includes three controls:
ControlIconFunction
Pause⏸Stops the real-time stream. New events are buffered but not displayed until you resume. Useful when you've captured the event you're looking for and want to inspect it without new entries pushing it down the list.
Clear Data🚫Wipes the current buffer and starts fresh. Helpful when you want a clean slate before reproducing a specific issue.
Filter🔽Opens the filter builder. Lets you narrow down to specific traffic patterns.
*** ### Filter System The filter UI uses a structured query builder with three components: **Key** → **Operator** → **Value** Select a key from the dropdown, choose an operator, and type a value to filter the current view. Multiple filters can be combined to narrow results further. #### Available Filter Keys by Tab **Requests Tab** | Key | Description | Example | | ----------- | ---------------------------- | ---------------------- | | `method` | HTTP method | GET, POST, PUT, DELETE | | `status` | HTTP status code | 200, 401, 404, 500 | | `endpoint` | Target domain | api.github.com | | `path` | URL path | /v1/users | | `direction` | Traffic direction | egress, ingress | | `process` | Binary that made the request | /usr/local/bin/ruby | **Connections Tab** | Key | Description | Example | | ---------------- | --------------------------------- | ------------------- | | `direction` | Traffic direction | egress, ingress | | `source` | Source IP address | 192.168.3.218 | | `destination` | Destination IP address | 140.82.112.5 | | `status` | Connection state | OPEN, CLOSED | | `socketProtocol` | Transport protocol | tcp | | `l7Protocol` | Application protocol | http1 | | `process` | Binary that opened the connection | /usr/local/bin/ruby | | `user` | System user | root, www-data | *** ### Processes Tab A real-time inventory of every process on the host.
#### Columns | Column | Description | | ------------- | ------------------------------------------------ | | **Timestamp** | When the process was observed | | **Status** | RUNNING (green) or EXITED (gray) | | **PID** | Process ID | | **Binary** | Executable name (python3.12, docker, ruby, etc.) | | **Path** | Full filesystem path to the binary | | **User** | Which system user is running the process | | **Container** | Container name, if applicable | | **Pod** | Kubernetes pod name, if applicable | | **Duration** | How long the process has been running | #### Detail Panel Click any process row to open the detail panel showing: * **Process Info**: Binary name, full path, hostname, PID * **User Info**: Username and UID #### Use Cases * See what's actually running on a host, especially in containerized environments * Identify unexpected processes * Correlate processes with network activity in other tabs *** ### Connections Tab Shows every network connection with process attribution.
#### Columns | Column | Description | | --------------- | -------------------------------------------- | | **Timestamp** | When the connection was observed | | **Direction** | Egress (outbound) or ingress (inbound) | | **Source** | Local IP:port | | **Destination** | Remote IP:port | | **Status** | OPEN (green) or CLOSED (gray) | | **Socket** | Transport protocol (tcp) | | **L7** | Detected application protocol (http1, http2) | | **Process** | Binary that opened the connection | | **Duration** | Connection lifetime | #### Detail Panel Click any connection to see: * **Connection Info**: Unique connection ID, resolved endpoint hostname (e.g., `api.mailgun.net`), creation timestamp * **Source**: IP address, hostname, executable path * **Destination**: Remote IP address * **System**: Hostname, agent type, agent instance ID * **TLS Detection**: Which TLS library was detected (openssl, gotls, nodetls, javassl) * **Tags**: Auto-generated metadata including binary, environment, host, protocol #### Use Cases * See which external services your application connects to * Identify unexpected outbound connections * Debug connection timeouts or failures * Understand connection duration patterns *** ### Requests Tab Individual HTTP transactions with full request and response details. This is the closest equivalent to Chrome DevTools' Network tab.
#### Columns | Column | Description | | ------------- | ------------------------------------------------------------------ | | **Timestamp** | When the request was made | | **Direction** | Egress or ingress | | **Endpoint** | Target domain (api.github.com, api.stripe.com) | | **Path** | URL path (/v1/customers, /rate\_limit) | | **Status** | HTTP status code, color-coded (green=2xx, yellow=3xx, red=4xx/5xx) | | **Method** | HTTP method (GET, POST, PUT, DELETE) | | **Type** | Response content type (application/json, text/html) | | **Process** | Binary that made the request | | **Size** | Response body size | | **Time** | Request duration in milliseconds | #### Detail Panel Click any request to see the full transaction details: **Header Bar** * Method and status code with timing * Full URL * Size, process path, and container name * Link to the underlying connection
**Tabs** | Tab | Contents | | ------------ | ------------------------------------------------------------------------------------------------ | | **Headers** | Complete request headers (including Authorization, cookies, custom headers) and response headers | | **Response** | Full response body | | **cURL** | Copy-pasteable cURL command to reproduce the request | #### Use Cases * Debug authentication failures by seeing the exact Authorization header sent * Inspect API error responses that your application logs don't capture * Compare request structure against API documentation * Export requests as cURL commands for testing *** ### Tips #### Debugging a Specific Issue 1. Click **Clear Data** to start with an empty buffer 2. Set up filters to isolate the traffic you care about (e.g., `endpoint = api.stripe.com`) 3. Reproduce the issue in your application 4. Click **Pause** once you see the relevant request 5. Click into the request to inspect headers and response #### Finding Unexpected Traffic 1. Filter by `process` to isolate a specific service 2. Look at the Connections tab to see all destinations 3. Check for connections to unexpected endpoints #### Investigating Slow Requests 1. Sort by the **Time** column to find slow requests 2. Check the detail panel for timing breakdown 3. Look at connection duration in the Connections tab to distinguish network latency from processing time *** ### Data Retention DevTools uses a ring buffer in your browser's memory. Data is ephemeral and cleared on page refresh. For permanent storage, configure S3-compatible object storage in your Qtap configuration. # DevTools API The DevTools API provides programmatic access to the same real-time traffic data shown in the DevTools browser UI. Use it to build custom tooling, integrate with scripts, or pipe traffic data to other systems. {% hint style="info" %} **When to use the API vs the UI**: The browser UI is great for interactive debugging. The API is better when you want to filter/process data programmatically, integrate with other tools, or capture output to a file. {% endhint %} *** ## Endpoint When DevTools is enabled, Qtap exposes an SSE (Server-Sent Events) stream at: ``` http://localhost:10001/devtools/api/events ``` The stream delivers events in real-time as traffic flows through the system. *** ## Quick Start ```bash # Stream all events curl -sN http://localhost:10001/devtools/api/events # Stream and filter for HTTP transactions only curl -sN http://localhost:10001/devtools/api/events | grep "request.http_transaction" # Pretty-print event data curl -sN http://localhost:10001/devtools/api/events | \ grep "^data:" | sed 's/^data: //' | jq . ``` *** ## Event Types The API streams six event types: | Event | Description | | -------------------------- | --------------------------------------------------------- | | `system.connected` | Initial connection to the event stream | | `process.started` | New process detected on host | | `process.stopped` | Process terminated | | `connection.opened` | New network connection established | | `connection.updated` | Connection metadata updated (protocol detected, TLS info) | | `connection.closed` | Connection terminated | | `request.http_transaction` | Complete HTTP request/response captured | *** ## Event Format Events follow the SSE format with `event:` and `data:` lines: ``` event: process.started data: {"data":{...},"ts":"2025-12-12T16:58:10.452Z"} event: connection.opened data: {"data":{...},"ts":"2025-12-12T16:58:11.123Z"} ``` *** ## Event Schemas ### process.started / process.stopped Process lifecycle events include container attribution when applicable. ```json { "data": { "binary": "python3.11", "path": "/usr/local/bin/python3.11", "pid": 60895, "hostname": "my-container", "user": { "id": 0, "name": "root" }, "container": { "id": "1781322a598f", "name": "my-app", "image": "python:3.11-slim", "labels": { "app": "frontend", "env": "production" } }, "createdAt": "2025-12-12T16:42:58.300Z" }, "ts": "2025-12-12T16:58:10.452Z" } ``` | Field | Description | | ------------------ | ----------------------------------------------- | | `binary` | Executable name | | `path` | Full path to binary | | `pid` | Process ID | | `hostname` | Container hostname or system hostname | | `user` | User ID and name running the process | | `container` | Container metadata (when running in Docker/K8s) | | `container.labels` | Labels from Docker or Kubernetes | ### connection.opened / connection.updated / connection.closed Connection events include source/destination info, protocol detection, and TLS details. ```json { "data": { "data": { "meta": { "connectionId": "d4u4dho7p3qk26l914bg", "endpointId": "api.stripe.com", "tlsProbeTypesDetected": ["openssl"], "tlsProbeIntrospected": true }, "direction": "egress-external", "socketProtocol": "tcp", "l7Protocol": "http1", "tlsVersion": "TLS 1.3", "source": { "address": {"ip": "172.18.0.7", "port": 34438}, "hostname": "my-container", "exe": "/usr/local/bin/python3.11", "user": "root", "container": { "id": "ac55e22af23b", "name": "payment-service", "image": "python:3.11-slim" } }, "destination": { "address": {"ip": "52.4.128.73", "port": 443} }, "tags": { "bin": ["python3.11"], "host": ["my-container"] } } }, "ts": "2025-12-12T16:58:40.980Z" } ``` | Field | Description | | ---------------------------- | ------------------------------------------------------------- | | `meta.connectionId` | Unique connection identifier | | `meta.endpointId` | Resolved hostname of destination | | `meta.tlsProbeTypesDetected` | TLS libraries detected (openssl, gotls, nodetls, javassl) | | `meta.tlsProbeIntrospected` | Whether TLS was successfully introspected | | `direction` | Traffic direction (egress-external, egress-internal, ingress) | | `l7Protocol` | Detected application protocol (http1, http2, other) | | `tlsVersion` | TLS version (TLS 1.2, TLS 1.3) | | `source` | Source process, container, and address info | | `destination` | Destination address | | `tags` | Metadata tags from container labels | ### request.http\_transaction HTTP transaction events contain the full request/response with headers and bodies. ```json { "data": { "data": { "connectionId": "d4u4jhg7p3qk26l91icg", "endpointId": "api.example.com", "requestId": "d4u4jhg7p3qk26l91id0", "type": "http_transaction", "data": "eyJtZXRhZGF0YSI6ey4uLn0sInJlcXVlc3QiOns...}", "contentType": "application/json", "summary": { "request_method": "POST", "request_host": "api.example.com", "request_scheme": "https", "request_protocol": "http1", "response_status": 200, "duration_ms": 81, "container_name": "my-app", "process_exe": "/usr/local/bin/python3.11", "direction": "egress-external" } } }, "ts": "2025-12-12T16:58:46.516Z" } ``` The `data` field contains base64-encoded JSON with the full transaction: ```json { "metadata": { "process_id": "60895", "process_exe": "/usr/local/bin/python3.11", "container_name": "my-app", "container_image": "python:3.11-slim", "bytes_sent": 517, "bytes_received": 324 }, "request": { "method": "POST", "url": "https://api.example.com/v1/users", "scheme": "https", "path": "/v1/users", "authority": "api.example.com", "protocol": "http1", "user_agent": "python-requests/2.31.0", "content_type": "application/json", "headers": { "Authorization": "Bearer token123", "Content-Type": "application/json" }, "body": "eyJ1c2VyIjogImpvaG4ifQ==" }, "response": { "status": 200, "content_type": "application/json", "headers": { "Content-Type": "application/json" }, "body": "eyJpZCI6IDEyM30=" }, "duration_ms": 81, "direction": "egress-external" } ``` {% hint style="warning" %} Request and response bodies are base64-encoded within the transaction data. Decode them to see the actual content. {% endhint %} *** ## Examples ### Filter for errors only ```bash curl -sN http://localhost:10001/devtools/api/events | \ grep "request.http_transaction" -A1 | \ grep "^data:" | \ sed 's/^data: //' | \ jq 'select(.data.data.summary.response_status >= 400)' ``` ### Watch connections to a specific host ```bash curl -sN http://localhost:10001/devtools/api/events | \ grep "connection" -A1 | \ grep "^data:" | \ sed 's/^data: //' | \ jq 'select(.data.data.meta.endpointId == "api.stripe.com")' ``` ### Decode HTTP transaction bodies ```bash curl -sN http://localhost:10001/devtools/api/events | \ grep "request.http_transaction" -A1 | \ grep "^data:" | \ sed 's/^data: //' | \ jq -r '.data.data.data' | \ base64 -d | \ jq . ``` ### Export to file for later analysis ```bash curl -sN http://localhost:10001/devtools/api/events > traffic-capture.txt # Then analyze later grep "request.http_transaction" traffic-capture.txt | wc -l ``` *** ## Comparison with Plugin Output | Aspect | DevTools API | Plugins (stdout/S3) | | ------------------------- | -------------------- | ------------------------ | | **Delivery** | Real-time SSE stream | Log output or S3 objects | | **Filtering** | Client-side | Server-side via Rulekit | | **Persistence** | Ephemeral | Configurable storage | | **Process visibility** | All processes | Only HTTP traffic | | **Connection visibility** | All connections | HTTP transactions only | | **Use case** | Live debugging | Production capture | For production monitoring and compliance storage, use [plugins with S3 storage](/getting-started/qtap/configuration/storage-configuration). The DevTools API is optimized for real-time debugging sessions. *** ## Security Considerations {% hint style="warning" %} The DevTools API exposes complete HTTP traffic including headers, bodies, and authentication tokens. Keep port 10001 secured. {% endhint %} * DevTools listens only on `localhost` by default * Use SSH tunneling for remote access: `ssh -L 10001:localhost:10001 user@server` * Never expose port 10001 to the public internet * Consider network segmentation in production environments # AI-Assisted Troubleshooting with Ollama Use a local LLM to analyze your network traffic in real-time. This guide shows how to set up Qwen3 with Ollama so you can ask questions about your traffic in plain English - and the AI fetches the data itself. {% hint style="info" %} **Looking for OpenAI/ChatGPT?** See the [MCP Server guide](/guides/devtools-guides/devtools-mcp-server) for connecting Codex CLI or ChatGPT to DevTools. {% endhint %} **What you'll build:** ``` You: "What external APIs is my app calling?" Qwen3: *fetches traffic from DevTools API* "Based on the traffic I captured, your application is connecting to: - api.stripe.com (payment-service container) - api.openai.com (risk-ai-sprawl container) - hooks.slack.com (multiple containers) ..." ``` No copy/paste. No manual data wrangling. The LLM calls the DevTools API directly. *** ## Prerequisites * Linux host with Qtap installed * \~3GB disk space for the Qwen3 model *** ## Quick Start (No Code) The simplest approach: capture traffic, paste into any LLM. ```bash # Capture 10 seconds of traffic curl -sN --max-time 10 http://localhost:10001/devtools/api/events | \ grep "^data:" | head -30 | sed 's/^data: //' ``` Copy the output and paste it into ChatGPT, Claude, or `ollama run qwen3:4b` with a question like: > "Here's my network traffic. What external services am I calling? Are there any errors?" This works great for one-off debugging. For a more integrated experience, continue below. *** ## Setup ### Step 1: Install Ollama and Python Library ```bash # Install Ollama curl -fsSL https://ollama.ai/install.sh | sh # Pull the model ollama pull qwen3:4b # Install Python library pip install ollama ``` ### Step 2: Start Qtap with DevTools {% tabs %} {% tab title="Binary" %} ```bash sudo qtap --enable-dev-tools ``` {% endtab %} {% tab title="Docker" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` {% endtab %} {% endtabs %} Verify DevTools is accessible: ```bash curl -s http://localhost:10001/devtools/api/events | head -3 ``` ### Step 3: Create the Traffic Analysis Script Save this as `traffic-chat.py`: ```python #!/usr/bin/env python3 """ Chat with an AI that can see your network traffic. Uses Qwen3 via Ollama with tool calling to query the DevTools API. """ import subprocess import json from ollama import chat def get_traffic(seconds: int = 5) -> str: """Capture recent network traffic from the DevTools API. Args: seconds: How many seconds of traffic to capture (default 5) Returns: JSON array of traffic events including HTTP requests, connections, and process information """ result = subprocess.run( ["curl", "-sN", "--max-time", str(seconds), "http://localhost:10001/devtools/api/events"], capture_output=True, text=True ) # Parse SSE format into JSON array events = [] for line in result.stdout.split('\n'): if line.startswith('data: '): try: events.append(json.loads(line[6:])) except json.JSONDecodeError: pass # Return summary to fit in context window return json.dumps(events[:50], indent=2) def chat_loop(): """Interactive chat loop with traffic analysis capability.""" print("Traffic Analysis Chat (Qwen3 + DevTools)") print("Ask questions about your network traffic.") print("Type 'quit' to exit.\n") messages = [] model = "qwen3:4b" # System prompt messages.append({ "role": "system", "content": """You are a network traffic analyst. You have access to a tool that captures real-time network traffic from a Linux server. The traffic data includes HTTP requests, connections, process information, and container metadata. When the user asks about traffic, use the get_traffic tool to fetch current data. Analyze the results and provide specific, actionable insights. Reference actual endpoints, status codes, container names, and other concrete details from the data. Keep responses concise and focused on what the user asked.""" }) while True: try: user_input = input("You: ").strip() except (KeyboardInterrupt, EOFError): print("\nGoodbye!") break if not user_input: continue if user_input.lower() in ('quit', 'exit', 'q'): print("Goodbye!") break messages.append({"role": "user", "content": user_input}) # First call - model may request tool use response = chat( model=model, messages=messages, tools=[get_traffic], ) # Handle tool calls if response.message.tool_calls: # Add assistant's tool request to history messages.append(response.message) for tool_call in response.message.tool_calls: print(f"[Fetching traffic data...]") # Execute the tool args = tool_call.function.arguments result = get_traffic(**args) # Add tool result to history messages.append({ "role": "tool", "content": result, }) # Get final response with tool results response = chat( model=model, messages=messages, tools=[get_traffic], ) # Print response assistant_message = response.message.content messages.append({"role": "assistant", "content": assistant_message}) print(f"\nQwen3: {assistant_message}\n") if __name__ == "__main__": chat_loop() ``` Make it executable: ```bash chmod +x traffic-chat.py ``` ### Step 4: Start Chatting ```bash python3 traffic-chat.py ``` Example session: ``` Traffic Analysis Chat (Qwen3 + DevTools) Ask questions about your network traffic. Type 'quit' to exit. You: What external services is my application calling? [Fetching traffic data...] Qwen3: Based on the captured traffic, your application is connecting to: 1. **api.stripe.com** - from container `stripe-payment-service` 2. **api.openai.com** - from container `risk-ai-sprawl` 3. **hooks.slack.com** - from container `risk-intern-mistakes` 4. **generativelanguage.googleapis.com** - from container `risk-ai-sprawl` The Stripe and OpenAI connections are using TLS 1.3 with HTTP/2. You: Are there any failed requests? [Fetching traffic data...] Qwen3: Yes, I found 2 failed requests: 1. **POST healthlake.us-east-1.amazonaws.com/patients** → 400 Bad Request - Container: prove-healthcare-api - Error: "Interaction not supported" 2. **POST prometheus.io** → 404 Not Found - Container: inventory-metrics-collector You: quit Goodbye! ``` *** ## Example Questions **Debugging:** * "What external APIs is my app calling?" * "Are there any failed requests?" * "What's making requests to api.stripe.com?" * "Which requests are taking the longest?" **Security:** * "What data is being sent to external services?" * "Are there any unexpected outbound connections?" * "Which containers are making the most external calls?" **Analysis:** * "Summarize the traffic patterns" * "What's the breakdown of HTTP status codes?" * "How many unique endpoints are being contacted?" *** ## Troubleshooting ### "No traffic captured" Make sure Qtap is running and DevTools is enabled: ```bash curl -s http://localhost:10001/devtools/api/events | head -5 ``` ### Model seems slow Try reducing the capture window or using a smaller context: ```python # In get_traffic(), reduce events returned return json.dumps(events[:20], indent=2) # Fewer events ``` ### Tool not being called If Qwen3 responds without fetching data, be more explicit: ``` "Fetch the current traffic and tell me what services we're calling" ``` ### Ollama connection errors Ensure Ollama is running: ```bash ollama serve ``` *** ## Next Steps * [DevTools API Reference](/guides/devtools-guides/devtools-api) - Full API documentation * [DevTools Interface Guide](/guides/devtools-guides/devtools-interface-guide) - Use the browser UI alongside the AI * [Traffic Processing with Plugins](/getting-started/qtap/configuration/traffic-processing-with-plugins) - Configure what traffic gets captured # DevTools MCP Server Create an MCP (Model Context Protocol) server that exposes your DevTools traffic data to AI assistants. This enables Codex CLI, ChatGPT, and Claude Code to query your network traffic directly. {% hint style="info" %} **Looking for local-only?** See the [Ollama guide](/guides/devtools-guides/ai-troubleshooting-with-devtools) for a fully local setup without external dependencies. {% endhint %} **What you'll build:** ``` You: "What external APIs is my app calling?" AI: [Calling get_hosts tool...] "Your application is connecting to 16 external APIs: - api.stripe.com (3 requests, all 200s) - /v1/customers, /v1/charges - api.openai.com (2 requests, 401 errors) - /v1/chat/completions - api.segment.io (2 requests, all 200s) - /v1/track ..." You: "Are we leaking any sensitive data?" AI: [Calling check_sensitive_data tool...] "Found 5 requests with sensitive data: - api.github.com: X-Api-Key header exposed - httpbin.org: Password in request body - www.googleapis.com: Database credentials in payload ..." ``` *** ## Available Tools | Tool | Use Case | Default Capture | | ---------------------- | ------------------------------------- | ------------------ | | `get_traffic` | Full HTTP details for debugging | 60s, 500 requests | | `get_traffic_summary` | Quick overview of traffic patterns | 60s, 1000 requests | | `get_hosts` | Map all external API dependencies | 60s, 1000 requests | | `get_errors` | Debug 4xx/5xx failures | 60s, 1000 requests | | `check_sensitive_data` | Security audit for leaked secrets | 60s, 500 requests | | `get_processes` | Which process/container made requests | 60s, 1000 requests | | `get_payloads` | Inspect request/response bodies | 60s, 500 requests | | `search_traffic` | Filter by host, method, status | 60s, 1000 requests | | `get_connections` | Network/TLS connection details | 60s, 2000 events | All tools support up to **5 minutes** of capture with `seconds=300`. *** ## Prerequisites * Linux host with Qtap installed and DevTools enabled * Python 3.10+ *** ## Create the MCP Server ### Step 1: Install Dependencies ```bash pip install mcp httpx ``` ### Step 2: Create the Server Save this as `devtools_mcp.py`: ```python #!/usr/bin/env python3 """ MCP Server that exposes Qtap DevTools traffic data to AI assistants. Works with Codex CLI, ChatGPT, and Claude Code. Tools: - get_traffic: Full HTTP transactions with headers and bodies - get_traffic_summary: Quick overview of traffic patterns - check_sensitive_data: Scan for API keys, tokens, passwords - get_errors: All 4xx/5xx responses with full details - get_hosts: External API dependency mapping - get_connections: Active network connections - get_processes: Process/container attribution - get_payloads: Request/response bodies for data inspection - search_traffic: Filter traffic by host, path, status, method """ import base64 import json import re import time import httpx from mcp.server.fastmcp import FastMCP mcp = FastMCP("Qtap DevTools") DEVTOOLS_URL = "http://localhost:10001/devtools/api/events" # Sensitive data patterns for security scanning SENSITIVE_PATTERNS = [ (r'[Aa]uthorization', 'Authorization header'), (r'[Bb]earer\s+[A-Za-z0-9\-_\.]+', 'Bearer token'), (r'[Aa]pi[-_]?[Kk]ey', 'API key'), (r'[Ss]ecret', 'Secret'), (r'[Pp]assword', 'Password'), (r'[Tt]oken', 'Token'), (r'[Cc]ookie', 'Cookie'), (r'[Ss]ession[-_]?[Ii]d', 'Session ID'), (r'[Cc]redential', 'Credential'), (r'[Pp]rivate[-_]?[Kk]ey', 'Private key'), (r'sk-[a-zA-Z0-9]{20,}', 'OpenAI API key'), (r'ghp_[a-zA-Z0-9]{36}', 'GitHub PAT'), (r'xox[baprs]-[a-zA-Z0-9-]+', 'Slack token'), ] def _capture_raw_events(seconds: int = 30, max_events: int = 500): """Capture raw events from DevTools SSE stream.""" seconds = min(seconds, 300) # Up to 5 minutes try: with httpx.Client(timeout=seconds + 10) as client: events = [] current_event_type = None first_event_time = None with client.stream("GET", DEVTOOLS_URL, timeout=seconds + 10) as response: start = time.time() for line in response.iter_lines(): if time.time() - start > seconds: break if line.startswith("event: "): current_event_type = line[7:] elif line.startswith("data: "): try: event_data = json.loads(line[6:]) event_ts = event_data.get("ts", "") if first_event_time is None: first_event_time = event_ts # Skip initial process dump if current_event_type == "process.started": if event_ts[:20] == first_event_time[:20]: continue events.append({ "type": current_event_type, "ts": event_ts, "data": event_data.get("data", {}) }) except json.JSONDecodeError: pass if len(events) >= max_events: break return events except httpx.ConnectError: return {"error": "Cannot connect to DevTools API at localhost:10001", "hint": "Start Qtap with --enable-dev-tools or ENABLE_DEV_TOOLS=true"} except Exception as e: return {"error": str(e)} def _decode_http_transaction(event): """Decode a raw http_transaction event into structured data.""" inner = event.get("data", {}) if isinstance(inner, dict) and "data" in inner: payload = inner.get("data") if isinstance(payload, dict) and "data" in payload: payload = payload.get("data") if not isinstance(payload, str): return None try: decoded = json.loads(base64.b64decode(payload)) # Decode response body res = decoded.get("response", {}) if res.get("body"): try: res["body"] = base64.b64decode(res["body"]).decode('utf-8', errors='replace') except: res["body"] = "[binary data]" # Decode request body if present req = decoded.get("request", {}) if req.get("body"): try: req["body"] = base64.b64decode(req["body"]).decode('utf-8', errors='replace') except: req["body"] = "[binary data]" return { "ts": event.get("ts"), "metadata": decoded.get("metadata", {}), "request": req, "response": res } except: pass return None def _capture_http(seconds: int = 30, max_http: int = 500): """Capture and decode HTTP transactions.""" events = _capture_raw_events(seconds=seconds, max_events=2000) if isinstance(events, dict) and "error" in events: return events http_events = [] for event in events: if event.get("type") == "request.http_transaction": decoded = _decode_http_transaction(event) if decoded: http_events.append(decoded) if len(http_events) >= max_http: break return http_events # ============================================================================= # HTTP Traffic Tools # ============================================================================= @mcp.tool() def get_traffic(seconds: int = 60, max_requests: int = 500) -> list | dict: """ Capture HTTP traffic with full request/response details. Best for: Debugging specific requests, seeing exactly what's being sent/received. Args: seconds: Capture duration (default 60, max 300 = 5 minutes) max_requests: Max HTTP requests to return (default 500, max 1000) Returns: JSON array of HTTP transactions with method, URL, headers, and bodies. """ max_requests = min(max_requests, 1000) events = _capture_http(seconds=seconds, max_http=max_requests) if isinstance(events, dict) and "error" in events: return events if not events: return {"message": "No HTTP traffic captured", "hint": "Try increasing capture time or generate some traffic"} return events @mcp.tool() def get_traffic_summary(seconds: int = 60) -> dict: """ Quick overview of HTTP traffic patterns. Best for: "What's happening right now?", "How much traffic?", "Any errors?" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Host counts, method breakdown, status code distribution, error list. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events hosts = {} status_codes = {} methods = {} errors = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") method = req.get("method", "?") status = res.get("status", 0) hosts[host] = hosts.get(host, 0) + 1 methods[method] = methods.get(method, 0) + 1 status_codes[str(status)] = status_codes.get(str(status), 0) + 1 if status >= 400: errors.append({ "host": host, "method": method, "path": req.get("path", "/")[:60], "status": status }) return { "total_requests": len(events), "unique_hosts": len(hosts), "hosts": dict(sorted(hosts.items(), key=lambda x: -x[1])[:25]), "methods": methods, "status_codes": dict(sorted(status_codes.items())), "error_count": len(errors), "errors": errors[:30] } @mcp.tool() def search_traffic( seconds: int = 60, host: str = None, path_contains: str = None, method: str = None, status_min: int = None, status_max: int = None, max_results: int = 200 ) -> dict: """ Search and filter HTTP traffic. Best for: "Show me requests to api.stripe.com", "Find all POST requests", "Show 500 errors" Args: seconds: Capture duration (default 60, max 300) host: Filter by host (partial match) path_contains: Filter by path substring method: Filter by HTTP method (GET, POST, etc.) status_min: Minimum status code status_max: Maximum status code max_results: Max results to return (default 200) Returns: Filtered HTTP transactions matching criteria. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events filtered = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) if host and host.lower() not in req.get("authority", "").lower(): continue if path_contains and path_contains.lower() not in req.get("path", "").lower(): continue if method and req.get("method", "").upper() != method.upper(): continue if status_min and res.get("status", 0) < status_min: continue if status_max and res.get("status", 0) > status_max: continue filtered.append(event) return { "total_scanned": len(events), "matches": len(filtered), "requests": filtered[:max_results] } # ============================================================================= # Security Tools # ============================================================================= @mcp.tool() def check_sensitive_data(seconds: int = 60) -> dict: """ Scan traffic for sensitive data: API keys, tokens, passwords, credentials. Best for: "Are we leaking secrets?", "What sensitive data is being transmitted?" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Report of sensitive patterns found in headers and bodies. """ events = _capture_http(seconds=seconds, max_http=500) if isinstance(events, dict) and "error" in events: return events findings = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") path = req.get("path", "/") event_findings = [] # Check request headers for header, value in req.get("headers", {}).items(): for pattern, label in SENSITIVE_PATTERNS: if re.search(pattern, header, re.IGNORECASE) or re.search(pattern, str(value)): event_findings.append({ "location": f"request header: {header}", "type": label, "preview": str(value)[:80] + ("..." if len(str(value)) > 80 else "") }) break # Check response headers for header, value in res.get("headers", {}).items(): for pattern, label in SENSITIVE_PATTERNS: if re.search(pattern, header, re.IGNORECASE): event_findings.append({ "location": f"response header: {header}", "type": label }) break # Check request body req_body = str(req.get("body", "")) if req_body: for pattern, label in SENSITIVE_PATTERNS: match = re.search(f'.{{0,30}}{pattern}.{{0,30}}', req_body, re.IGNORECASE) if match: event_findings.append({ "location": "request body", "type": label, "context": match.group(0) }) # Check response body res_body = str(res.get("body", "")) if res_body and len(res_body) < 50000: for pattern, label in SENSITIVE_PATTERNS: match = re.search(f'.{{0,30}}{pattern}.{{0,30}}', res_body, re.IGNORECASE) if match: event_findings.append({ "location": "response body", "type": label, "context": match.group(0) }) if event_findings: findings.append({ "host": host, "method": req.get("method"), "path": path[:100], "findings": event_findings }) return { "requests_scanned": len(events), "requests_with_sensitive_data": len(findings), "risk_level": "HIGH" if len(findings) > 5 else "MEDIUM" if len(findings) > 0 else "LOW", "findings": findings } # ============================================================================= # Error Analysis Tools # ============================================================================= @mcp.tool() def get_errors(seconds: int = 60) -> dict: """ Get all HTTP errors (4xx and 5xx) with full details for debugging. Best for: "What's failing?", "Why are requests erroring?", "Debug 500 errors" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Failed requests with status, headers, and response bodies. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events errors = [] for event in events: res = event.get("response", {}) status = res.get("status", 0) if status >= 400: req = event.get("request", {}) errors.append({ "ts": event.get("ts"), "host": req.get("authority"), "method": req.get("method"), "path": req.get("path"), "url": req.get("url"), "status": status, "request_headers": req.get("headers", {}), "request_body": req.get("body", "")[:1000] if req.get("body") else None, "response_headers": res.get("headers", {}), "response_body": res.get("body", "")[:3000] if res.get("body") else None }) by_status = {} for err in errors: status = str(err["status"]) by_status[status] = by_status.get(status, 0) + 1 return { "total_requests": len(events), "error_count": len(errors), "by_status": by_status, "errors": errors[:200] } # ============================================================================= # Infrastructure Tools # ============================================================================= @mcp.tool() def get_hosts(seconds: int = 60) -> dict: """ Map all external hosts/APIs being called. Best for: "What external services do we depend on?", "API inventory" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: List of hosts with request counts, methods, status codes, sample paths. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events hosts = {} for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") if host not in hosts: hosts[host] = { "count": 0, "methods": set(), "status_codes": set(), "paths": [], "errors": 0 } hosts[host]["count"] += 1 hosts[host]["methods"].add(req.get("method", "?")) hosts[host]["status_codes"].add(res.get("status", 0)) if res.get("status", 0) >= 400: hosts[host]["errors"] += 1 if len(hosts[host]["paths"]) < 5: hosts[host]["paths"].append(req.get("path", "/")[:60]) result = {} for host, data in sorted(hosts.items(), key=lambda x: -x[1]["count"]): result[host] = { "requests": data["count"], "errors": data["errors"], "methods": sorted(data["methods"]), "status_codes": sorted(data["status_codes"]), "sample_paths": data["paths"] } return { "capture_seconds": seconds, "total_requests": len(events), "unique_hosts": len(result), "hosts": result } @mcp.tool() def get_connections(seconds: int = 60) -> dict: """ Get active network connections with TLS and protocol info. Best for: "What connections are open?", "TLS versions?", "Connection issues?" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Connection details including remote addresses, TLS info, and states. """ events = _capture_raw_events(seconds=seconds, max_events=2000) if isinstance(events, dict) and "error" in events: return events connections = {} for event in events: event_type = event.get("type", "") data = event.get("data", {}) if event_type in ["connection.opened", "connection.updated", "connection.closed"]: conn_id = data.get("connectionId") or data.get("id") if not conn_id: continue if conn_id not in connections: connections[conn_id] = { "id": conn_id, "events": [], "remote": None, "local": None, "tls": None, "protocol": None } connections[conn_id]["events"].append(event_type.split(".")[-1]) if data.get("remote"): connections[conn_id]["remote"] = data["remote"] if data.get("local"): connections[conn_id]["local"] = data["local"] if data.get("tls"): connections[conn_id]["tls"] = data["tls"] if data.get("protocol"): connections[conn_id]["protocol"] = data["protocol"] conn_list = list(connections.values()) return { "capture_seconds": seconds, "total_connections": len(conn_list), "connections": conn_list[:200] } @mcp.tool() def get_processes(seconds: int = 60) -> dict: """ Get processes making network requests with container attribution. Best for: "Which process is calling X?", "What's this container doing?" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Process info including executable path, container name, and request counts. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events processes = {} for event in events: meta = event.get("metadata", {}) req = event.get("request", {}) proc_exe = meta.get("process_exe", "unknown") proc_id = meta.get("process_id", "?") key = proc_exe if key not in processes: processes[key] = { "executable": proc_exe, "process_ids": set(), "request_count": 0, "hosts_called": set(), "methods": set() } processes[key]["process_ids"].add(proc_id) processes[key]["request_count"] += 1 processes[key]["hosts_called"].add(req.get("authority", "unknown")) processes[key]["methods"].add(req.get("method", "?")) result = [] for proc in sorted(processes.values(), key=lambda x: -x["request_count"]): result.append({ "executable": proc["executable"], "process_ids": sorted(proc["process_ids"])[:5], "requests": proc["request_count"], "hosts": sorted(proc["hosts_called"])[:10], "methods": sorted(proc["methods"]) }) return { "capture_seconds": seconds, "total_requests": len(events), "unique_processes": len(result), "processes": result[:30] } # ============================================================================= # Payload Inspection Tools # ============================================================================= @mcp.tool() def get_payloads(seconds: int = 60, include_responses: bool = True, max_results: int = 200) -> dict: """ Get request and response bodies for data inspection. Best for: "What data is being sent?", "Inspect API payloads", "Debug request bodies" Args: seconds: Capture duration (default 60, max 300 = 5 minutes) include_responses: Include response bodies (default True) max_results: Max payloads to return (default 200) Returns: Request/response bodies with metadata. """ events = _capture_http(seconds=seconds, max_http=500) if isinstance(events, dict) and "error" in events: return events payloads = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) req_body = req.get("body") res_body = res.get("body") if include_responses else None if not req_body and not res_body: continue payload = { "host": req.get("authority"), "method": req.get("method"), "path": req.get("path", "/")[:100], "status": res.get("status"), "content_type": res.get("content_type") } if req_body: payload["request_body"] = req_body[:5000] payload["request_body_size"] = len(req_body) if res_body: payload["response_body"] = res_body[:10000] payload["response_body_size"] = len(res_body) payloads.append(payload) return { "capture_seconds": seconds, "requests_with_bodies": len(payloads), "payloads": payloads[:max_results] } if __name__ == "__main__": mcp.run() ``` Make it executable: ```bash chmod +x devtools_mcp.py ``` *** ## Connect to AI Assistants ### Option 1: Codex CLI [Codex CLI](https://github.com/openai/codex) is OpenAI's terminal-based coding assistant that supports MCP servers natively. **Add the MCP server to your config:** ```bash codex mcp add devtools -- python3 /path/to/devtools_mcp.py ``` Or edit `~/.codex/config.toml` directly: ```toml [mcp_servers.devtools] command = "python3" args = ["/path/to/devtools_mcp.py"] ``` {% hint style="info" %} **Using a virtual environment?** Replace `python3` with your venv Python path: ```bash codex mcp add devtools -- /path/to/venv/bin/python /path/to/devtools_mcp.py ``` {% endhint %} **Use it:** ```bash codex > What external APIs is my application calling? Use the devtools server. ``` Codex will call `get_hosts` and analyze the results. *** ### Option 2: ChatGPT Developer Mode ChatGPT can connect to MCP servers via Developer Mode connectors. Since ChatGPT requires HTTPS, you'll need to expose your local server. **1. Start the MCP server with HTTP transport:** ```python # Add to devtools_mcp.py, replace the if __name__ block: if __name__ == "__main__": mcp.run(transport="http", host="0.0.0.0", port=8080) ``` **2. Expose your local server via HTTPS tunnel:** {% tabs %} {% tab title="Cloudflared (Recommended)" %} ```bash # One-liner, no account required cloudflared tunnel --url http://localhost:8080 ``` This creates a temporary `*.trycloudflare.com` URL you can use immediately. {% endtab %} {% tab title="ngrok" %} ```bash ngrok http 8080 ``` Requires a free ngrok account. {% endtab %} {% endtabs %} **3. In ChatGPT:** * Go to Settings > Connectors > Advanced Settings * Enable Developer Mode * Add a new connector with your tunnel URL *** ### Option 3: Claude Code Claude Code supports MCP servers natively via the CLI. **Add the MCP server:** ```bash claude mcp add --transport stdio devtools -- python3 /path/to/devtools_mcp.py ``` {% hint style="info" %} **Using a virtual environment?** Replace `python3` with your venv Python path: ```bash claude mcp add --transport stdio devtools -- /path/to/venv/bin/python /path/to/devtools_mcp.py ``` {% endhint %} **Verify it's connected:** ```bash claude mcp list # Should show: devtools: ... - ✓ Connected ``` **Use it in Claude Code:** ``` What external APIs is my app calling? ``` Claude Code will automatically use the appropriate tools to analyze your traffic. *** ## Example Prompts Once connected, try these prompts: **Quick Overview:** * "What's happening with my network traffic?" * "Give me a summary of all HTTP requests" * "How many errors are there?" **API Discovery:** * "What external APIs is my app calling?" * "Which services are we depending on?" * "Show me all the hosts we're connecting to" **Security Audit:** * "Are we leaking any sensitive data?" * "Check for exposed API keys or passwords" * "What credentials are being transmitted?" **Error Debugging:** * "What's failing? Show me all errors" * "Why are requests to stripe.com failing?" * "Debug the 401 errors" **Process Attribution:** * "Which process is calling the OpenAI API?" * "What requests is the Python process making?" * "Show me traffic by container" **Payload Inspection:** * "What data is being sent to external services?" * "Show me the request bodies" * "What's in the response from api.github.com?" **Filtered Search:** * "Show me all POST requests" * "Find requests to anything with 'stripe' in the host" * "Show only 500 errors" *** ## Example Output ### get\_hosts() ```json { "capture_seconds": 60, "total_requests": 25, "unique_hosts": 16, "hosts": { "api.stripe.com": { "requests": 3, "errors": 0, "methods": ["GET", "POST"], "status_codes": [200], "sample_paths": ["/v1/customers", "/v1/charges", "/v1/balance"] }, "api.openai.com": { "requests": 2, "errors": 2, "methods": ["POST"], "status_codes": [401], "sample_paths": ["/v1/chat/completions"] } } } ``` ### check\_sensitive\_data() ```json { "requests_scanned": 25, "requests_with_sensitive_data": 5, "risk_level": "MEDIUM", "findings": [ { "host": "api.github.com", "method": "GET", "path": "/user", "findings": [ { "location": "request header: Authorization", "type": "Authorization header", "preview": "Bearer ghp_xxxxxxxxxxxx..." } ] } ] } ``` *** ## Troubleshooting ### "Cannot connect to DevTools API" Ensure Qtap is running with DevTools enabled: {% tabs %} {% tab title="Binary" %} ```bash sudo qtap --enable-dev-tools ``` {% endtab %} {% tab title="Docker" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` {% endtab %} {% endtabs %} Verify it's accessible: ```bash curl -s http://localhost:10001/devtools/api/events | head -3 ``` ### "MCP server not found" in Codex Check your config path: ```bash cat ~/.codex/config.toml ``` Ensure the Python path is absolute and the script exists. ### "No HTTP traffic captured" * Ensure traffic is being generated during the capture window * Try increasing the capture time: `get_traffic(seconds=120)` * Check that Qtap is capturing HTTP (not just connections) ### ChatGPT can't reach the server * Verify cloudflared/ngrok is running and the URL is correct * Check that the MCP server is listening on the right port * Ensure no firewall is blocking the connection *** ## Next Steps * [Traffic Analysis MCP Server](/guides/devtools-guides/traffic-analysis-mcp-server) - Type inference and test generation from traffic * [DevTools API Reference](/guides/devtools-guides/devtools-api) - Full API documentation * [Ollama Guide](/guides/devtools-guides/ai-troubleshooting-with-devtools) - Local-only setup * [DevTools Interface Guide](/guides/devtools-guides/devtools-interface-guide) - Browser UI # Traffic Analysis MCP Server Extend DevTools with AI-powered traffic analysis for code modernization. Infer types from real API responses, generate test fixtures from captured traffic, and validate changes haven't broken API contracts. {% hint style="info" %} **Need real-time debugging instead?** See the [DevTools MCP Server](/guides/devtools-guides/devtools-mcp-server) for live traffic inspection. {% endhint %} **What you'll build:** ``` You: "Capture 2 minutes of traffic while I run my test suite" AI: [Calling store_traffic tool...] "Captured 156 requests across 12 unique routes. Ready for analysis." You: "What endpoints are we hitting?" AI: [Calling get_routes tool...] "Found 12 routes: - /v1/charges (23 requests, POST, all 200s) - /v1/customers/{id} (18 requests, GET, 200/404) - /v1/balance (12 requests, GET, all 200s) ..." You: "Generate types for the charges endpoint" AI: [Calling generate_types tool...] "Generated ChargesResponse with 45 typed fields including nested BillingDetails, PaymentMethodDetails, and Card objects." You: "Create test fixtures from this traffic" AI: [Calling generate_test_data tool...] "Created pytest fixtures with 5 test cases covering 200, 400, and 402 responses." ``` *** ## Available Tools | Tool | Use Case | Best For | | -------------------- | ------------------------------------- | --------------------------------- | | `store_traffic` | Capture live traffic to SQLite cache | Building a corpus before analysis | | `clear_traffic` | Remove old traffic from cache | Cleaning up after sessions | | `get_traffic_stats` | Cache statistics | Checking what's available | | `get_routes` | Discover routes grouped by pattern | Finding endpoints to analyze | | `get_route_examples` | Representative request/response pairs | Understanding payload shapes | | `infer_schema` | JSON Schema from observed traffic | Understanding data contracts | | `generate_types` | Type definitions in multiple formats | Adding types to code | | `generate_test_data` | Test fixtures from real traffic | Creating test cases | | `compare_traffic` | Schema drift detection | Validating changes | All tools work with cached traffic. Capture first with `store_traffic`, then analyze. {% hint style="warning" %} **Data Sensitivity:** Traffic is stored in `.traffic_cache.db` alongside the script, including full headers and bodies. This may contain API keys, tokens, and PII. Use `clear_traffic()` regularly and avoid committing the database to version control. {% endhint %} **Limits:** Capture is capped at 5 minutes and 1,000 HTTP transactions per call. Schema inference uses only successful (2xx) responses. *** ## Use Cases ### Adding Types to Legacy Code Capture traffic from your running application, then generate type definitions: ``` "Capture traffic while I run the app, then generate Pydantic models for all Stripe endpoints" ``` ### Generating Test Fixtures Create realistic test data from actual API responses: ``` "Generate pytest fixtures for the /v1/customers endpoint with examples of success and error responses" ``` ### Validating API Contracts Before and after dependency upgrades, compare traffic to detect breaking changes: ``` "Capture traffic before the upgrade, then again after - show me any schema drift" ``` ### Dependency Modernization Migrating between library versions (e.g., Pydantic v1 to v2): ``` "Generate both Pydantic v1 and v2 models for this endpoint so I can compare the migration" ``` *** ## Prerequisites * Linux host with Qtap installed and DevTools enabled * Python 3.10+ * MCP and httpx packages *** ## Create the MCP Server ### Step 1: Install Dependencies ```bash pip install mcp httpx ``` ### Step 2: Create the Server Save this as `traffic_analysis_mcp.py`: ```python #!/usr/bin/env python3 """ MCP Server for traffic-based code modernization. Captures HTTP traffic and provides tools for: - Route discovery and grouping - Schema inference from observed payloads - Type generation (Pydantic, TypedDict, dataclass, etc.) - Test data generation - Before/after traffic comparison """ import base64 import hashlib import json import re import sqlite3 import time from collections import defaultdict from datetime import datetime, timedelta from pathlib import Path from typing import Any import httpx from mcp.server.fastmcp import FastMCP mcp = FastMCP("Traffic Analysis") DEVTOOLS_URL = "http://localhost:10001/devtools/api/events" DB_PATH = Path(__file__).parent / ".traffic_cache.db" # ============================================================================= # Path Normalization Patterns # ============================================================================= DYNAMIC_PATTERNS = [ (re.compile(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', re.I), '{uuid}'), (re.compile(r'\d{4}-\d{2}-\d{2}'), '{date}'), (re.compile(r'[0-9a-f]{24}', re.I), '{object_id}'), (re.compile(r'[A-Za-z0-9_-]{20,}'), '{token}'), (re.compile(r'\d+'), '{id}'), ] def _normalize_path(path: str) -> str: """Normalize a path by replacing dynamic segments with placeholders.""" if '?' in path: path, query = path.split('?', 1) else: query = None segments = path.split('/') normalized = [] for segment in segments: if not segment: normalized.append(segment) continue replaced = segment for pattern, placeholder in DYNAMIC_PATTERNS: if pattern.fullmatch(segment): replaced = placeholder break normalized.append(replaced) return '/'.join(normalized) # ============================================================================= # SQLite Database Management # ============================================================================= def _get_db() -> sqlite3.Connection: """Get a database connection, creating schema if needed.""" db = sqlite3.connect(str(DB_PATH)) db.row_factory = sqlite3.Row db.executescript(""" CREATE TABLE IF NOT EXISTS traffic ( id INTEGER PRIMARY KEY, captured_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, host TEXT NOT NULL, method TEXT NOT NULL, path TEXT NOT NULL, route_pattern TEXT NOT NULL, status_code INTEGER, request_headers TEXT, request_body TEXT, response_headers TEXT, response_body TEXT, process_exe TEXT, content_type TEXT, body_hash TEXT ); CREATE INDEX IF NOT EXISTS idx_route ON traffic(route_pattern, host); CREATE INDEX IF NOT EXISTS idx_captured ON traffic(captured_at); CREATE INDEX IF NOT EXISTS idx_host ON traffic(host); """) return db def _hash_body(body: str | None) -> str | None: """Create a hash of body content for deduplication.""" if not body: return None return hashlib.sha256(body.encode()).hexdigest()[:16] # ============================================================================= # Traffic Capture (from DevTools API) # ============================================================================= def _capture_raw_events(seconds: int = 30, max_events: int = 500): """Capture raw events from DevTools SSE stream.""" seconds = min(seconds, 300) try: with httpx.Client(timeout=seconds + 10) as client: events = [] current_event_type = None first_event_time = None with client.stream("GET", DEVTOOLS_URL, timeout=seconds + 10) as response: start = time.time() for line in response.iter_lines(): if time.time() - start > seconds: break if line.startswith("event: "): current_event_type = line[7:] elif line.startswith("data: "): try: event_data = json.loads(line[6:]) event_ts = event_data.get("ts", "") if first_event_time is None: first_event_time = event_ts if current_event_type == "process.started": if event_ts[:20] == first_event_time[:20]: continue events.append({ "type": current_event_type, "ts": event_ts, "data": event_data.get("data", {}) }) except json.JSONDecodeError: pass if len(events) >= max_events: break return events except httpx.ConnectError: return {"error": "Cannot connect to DevTools API at localhost:10001", "hint": "Start Qtap with --enable-dev-tools or ENABLE_DEV_TOOLS=true"} except Exception as e: return {"error": str(e)} def _decode_http_transaction(event): """Decode a raw http_transaction event into structured data.""" inner = event.get("data", {}) if isinstance(inner, dict) and "data" in inner: payload = inner.get("data") if isinstance(payload, dict) and "data" in payload: payload = payload.get("data") if not isinstance(payload, str): return None try: decoded = json.loads(base64.b64decode(payload)) res = decoded.get("response", {}) if res.get("body"): try: res["body"] = base64.b64decode(res["body"]).decode('utf-8', errors='replace') except: res["body"] = "[binary data]" req = decoded.get("request", {}) if req.get("body"): try: req["body"] = base64.b64decode(req["body"]).decode('utf-8', errors='replace') except: req["body"] = "[binary data]" return { "ts": event.get("ts"), "metadata": decoded.get("metadata", {}), "request": req, "response": res } except: pass return None def _capture_http(seconds: int = 30, max_http: int = 500): """Capture and decode HTTP transactions.""" max_raw_events = max(seconds * 500, 10000) events = _capture_raw_events(seconds=seconds, max_events=max_raw_events) if isinstance(events, dict) and "error" in events: return events http_events = [] for event in events: if event.get("type") == "request.http_transaction": decoded = _decode_http_transaction(event) if decoded: http_events.append(decoded) if len(http_events) >= max_http: break return http_events # ============================================================================= # Schema Inference # ============================================================================= def _infer_type(value: Any) -> dict: """Infer JSON Schema type from a Python value.""" if value is None: return {"type": "null"} elif isinstance(value, bool): return {"type": "boolean"} elif isinstance(value, int): return {"type": "integer"} elif isinstance(value, float): return {"type": "number"} elif isinstance(value, str): if re.match(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}', value): return {"type": "string", "format": "date-time"} elif re.match(r'^\d{4}-\d{2}-\d{2}$', value): return {"type": "string", "format": "date"} elif re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', value): return {"type": "string", "format": "email"} elif re.match(r'^https?://', value): return {"type": "string", "format": "uri"} elif re.match(r'^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$', value, re.I): return {"type": "string", "format": "uuid"} return {"type": "string"} elif isinstance(value, list): if not value: return {"type": "array", "items": {}} item_schemas = [_infer_type(item) for item in value[:10] if item is not None] if item_schemas: merged = _merge_schemas(item_schemas) return {"type": "array", "items": merged} return {"type": "array", "items": {}} elif isinstance(value, dict): properties = {} for k, v in value.items(): properties[k] = _infer_type(v) return { "type": "object", "properties": properties, "required": list(value.keys()) } return {} def _merge_schemas(schemas: list[dict]) -> dict: """Merge multiple JSON schemas into one that accepts all observed shapes.""" if not schemas: return {} if len(schemas) == 1: return schemas[0] types = set() all_properties = defaultdict(list) all_required = None formats = set() item_schemas = [] for schema in schemas: schema_type = schema.get("type") if schema_type: types.add(schema_type) if schema.get("format"): formats.add(schema["format"]) if schema_type == "object" and "properties" in schema: for prop, prop_schema in schema["properties"].items(): all_properties[prop].append(prop_schema) required = set(schema.get("required", [])) if all_required is None: all_required = required else: all_required = all_required & required if schema_type == "array" and "items" in schema: item_schemas.append(schema["items"]) result = {} if len(types) == 1: result["type"] = types.pop() elif len(types) > 1: if types == {"integer", "number"}: result["type"] = "number" elif "null" in types: other_types = types - {"null"} if len(other_types) == 1: result["type"] = [other_types.pop(), "null"] else: result["type"] = list(types) else: result["type"] = list(types) if len(formats) == 1: result["format"] = formats.pop() if all_properties: merged_props = {} for prop, prop_schemas in all_properties.items(): merged_props[prop] = _merge_schemas(prop_schemas) result["properties"] = merged_props if all_required: result["required"] = sorted(all_required) if item_schemas: result["items"] = _merge_schemas(item_schemas) return result def _infer_schema_from_bodies(bodies: list[str], target_name: str = "Root") -> dict: """Infer JSON Schema from multiple body strings.""" schemas = [] for body in bodies: if not body or body == "[binary data]": continue try: data = json.loads(body) schemas.append(_infer_type(data)) except json.JSONDecodeError: continue if not schemas: return {"error": "No valid JSON bodies to analyze"} merged = _merge_schemas(schemas) merged["$schema"] = "http://json-schema.org/draft-07/schema#" merged["title"] = target_name return merged # ============================================================================= # Code Generation # ============================================================================= def _schema_to_python_type(schema: dict, name: str = "Value") -> str: """Convert JSON Schema to Python type annotation.""" schema_type = schema.get("type") if isinstance(schema_type, list): types = [_schema_to_python_type({"type": t}, name) for t in schema_type] return " | ".join(types) if schema_type == "null": return "None" elif schema_type == "boolean": return "bool" elif schema_type == "integer": return "int" elif schema_type == "number": return "float" elif schema_type == "string": return "str" elif schema_type == "array": items = schema.get("items", {}) item_type = _schema_to_python_type(items, f"{name}Item") return f"list[{item_type}]" elif schema_type == "object": return name return "Any" def _to_class_name(name: str) -> str: """Convert a string to PascalCase class name.""" parts = re.split(r'[_-]', name) return ''.join(word.capitalize() for word in parts) def _generate_pydantic_v2(schema: dict, class_name: str = "Model") -> str: """Generate Pydantic v2 model from JSON Schema.""" lines = [ "from pydantic import BaseModel, Field", "from typing import Any", "" ] def generate_class(schema: dict, name: str, indent: int = 0) -> list[str]: result = [] prefix = " " * indent if schema.get("type") != "object" or "properties" not in schema: return result for prop_name, prop_schema in schema.get("properties", {}).items(): if prop_schema.get("type") == "object" and "properties" in prop_schema: nested_name = _to_class_name(prop_name) result.extend(generate_class(prop_schema, nested_name, indent)) result.append("") required = set(schema.get("required", [])) result.append(f"{prefix}class {name}(BaseModel):") properties = schema.get("properties", {}) if not properties: result.append(f"{prefix} pass") return result for prop_name, prop_schema in properties.items(): is_required = prop_name in required if prop_schema.get("type") == "object" and "properties" in prop_schema: py_type = _to_class_name(prop_name) else: py_type = _schema_to_python_type(prop_schema, _to_class_name(prop_name)) if not is_required: py_type = f"{py_type} | None" field_args = [] if not is_required: field_args.append("default=None") if field_args: result.append(f"{prefix} {prop_name}: {py_type} = Field({', '.join(field_args)})") else: result.append(f"{prefix} {prop_name}: {py_type}") return result lines.extend(generate_class(schema, class_name)) return "\n".join(lines) def _generate_pydantic_v1(schema: dict, class_name: str = "Model") -> str: """Generate Pydantic v1 compatible model from JSON Schema.""" lines = [ "from pydantic import BaseModel", "from typing import Any, List, Optional", "" ] def generate_class(schema: dict, name: str) -> list[str]: result = [] if schema.get("type") != "object" or "properties" not in schema: return result for prop_name, prop_schema in schema.get("properties", {}).items(): if prop_schema.get("type") == "object" and "properties" in prop_schema: nested_name = _to_class_name(prop_name) result.extend(generate_class(prop_schema, nested_name)) result.append("") required = set(schema.get("required", [])) result.append(f"class {name}(BaseModel):") properties = schema.get("properties", {}) if not properties: result.append(" pass") return result for prop_name, prop_schema in properties.items(): is_required = prop_name in required if prop_schema.get("type") == "object" and "properties" in prop_schema: py_type = _to_class_name(prop_name) else: py_type = _schema_to_python_type(prop_schema, _to_class_name(prop_name)) py_type = py_type.replace(" | None", "") py_type = py_type.replace("list[", "List[") if not is_required: result.append(f" {prop_name}: Optional[{py_type}] = None") else: result.append(f" {prop_name}: {py_type}") return result lines.extend(generate_class(schema, class_name)) return "\n".join(lines) def _generate_typeddict(schema: dict, class_name: str = "Model") -> str: """Generate TypedDict from JSON Schema.""" lines = [ "from typing import TypedDict, NotRequired, Any", "" ] def generate_class(schema: dict, name: str) -> list[str]: result = [] if schema.get("type") != "object" or "properties" not in schema: return result for prop_name, prop_schema in schema.get("properties", {}).items(): if prop_schema.get("type") == "object" and "properties" in prop_schema: nested_name = _to_class_name(prop_name) result.extend(generate_class(prop_schema, nested_name)) result.append("") required = set(schema.get("required", [])) result.append(f"class {name}(TypedDict):") properties = schema.get("properties", {}) if not properties: result.append(" pass") return result for prop_name, prop_schema in properties.items(): is_required = prop_name in required if prop_schema.get("type") == "object" and "properties" in prop_schema: py_type = _to_class_name(prop_name) else: py_type = _schema_to_python_type(prop_schema, _to_class_name(prop_name)) if not is_required: result.append(f" {prop_name}: NotRequired[{py_type}]") else: result.append(f" {prop_name}: {py_type}") return result lines.extend(generate_class(schema, class_name)) return "\n".join(lines) def _generate_dataclass(schema: dict, class_name: str = "Model") -> str: """Generate dataclass from JSON Schema.""" lines = [ "from dataclasses import dataclass, field", "from typing import Any", "" ] def generate_class(schema: dict, name: str) -> list[str]: result = [] if schema.get("type") != "object" or "properties" not in schema: return result for prop_name, prop_schema in schema.get("properties", {}).items(): if prop_schema.get("type") == "object" and "properties" in prop_schema: nested_name = _to_class_name(prop_name) result.extend(generate_class(prop_schema, nested_name)) result.append("") required = set(schema.get("required", [])) result.append("@dataclass") result.append(f"class {name}:") properties = schema.get("properties", {}) if not properties: result.append(" pass") return result required_props = [(k, v) for k, v in properties.items() if k in required] optional_props = [(k, v) for k, v in properties.items() if k not in required] for prop_name, prop_schema in required_props: if prop_schema.get("type") == "object" and "properties" in prop_schema: py_type = _to_class_name(prop_name) else: py_type = _schema_to_python_type(prop_schema, _to_class_name(prop_name)) result.append(f" {prop_name}: {py_type}") for prop_name, prop_schema in optional_props: if prop_schema.get("type") == "object" and "properties" in prop_schema: py_type = _to_class_name(prop_name) else: py_type = _schema_to_python_type(prop_schema, _to_class_name(prop_name)) result.append(f" {prop_name}: {py_type} | None = None") return result lines.extend(generate_class(schema, class_name)) return "\n".join(lines) # ============================================================================= # MCP Tools - Traffic Persistence # ============================================================================= @mcp.tool() def store_traffic(seconds: int = 60) -> dict: """ Capture live traffic and store it in the SQLite cache for analysis. Best for: Building up a corpus of traffic examples before running analysis. Args: seconds: Capture duration (default 60, max 300 = 5 minutes) Returns: Number of requests captured and stored. """ events = _capture_http(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events db = _get_db() stored = 0 for event in events: req = event.get("request", {}) res = event.get("response", {}) meta = event.get("metadata", {}) host = req.get("authority", "unknown") method = req.get("method", "GET") path = req.get("path", "/") route_pattern = _normalize_path(path) status_code = res.get("status", 0) req_body = req.get("body", "") res_body = res.get("body", "") db.execute(""" INSERT INTO traffic ( host, method, path, route_pattern, status_code, request_headers, request_body, response_headers, response_body, process_exe, content_type, body_hash ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( host, method, path, route_pattern, status_code, json.dumps(req.get("headers", {})), req_body, json.dumps(res.get("headers", {})), res_body, meta.get("process_exe", ""), res.get("content_type", ""), _hash_body(req_body) )) stored += 1 db.commit() db.close() return { "captured": len(events), "stored": stored, "message": f"Stored {stored} requests in cache" } @mcp.tool() def clear_traffic(older_than: str = "24h") -> dict: """ Clear old traffic from the cache. Args: older_than: Time window to keep (e.g., "1h", "24h", "7d"). Traffic older than this is deleted. Returns: Number of records deleted. """ match = re.match(r'^(\d+)([hdm])$', older_than) if not match: return {"error": "Invalid time format. Use format like '1h', '24h', '7d'"} amount, unit = int(match.group(1)), match.group(2) if unit == 'h': delta = timedelta(hours=amount) elif unit == 'd': delta = timedelta(days=amount) elif unit == 'm': delta = timedelta(minutes=amount) else: return {"error": "Invalid time unit"} cutoff = datetime.now() - delta db = _get_db() cursor = db.execute("DELETE FROM traffic WHERE captured_at < ?", (cutoff.strftime('%Y-%m-%d %H:%M:%S'),)) deleted = cursor.rowcount db.commit() db.close() return { "deleted": deleted, "cutoff": cutoff.isoformat(), "message": f"Deleted {deleted} records older than {older_than}" } @mcp.tool() def get_traffic_stats() -> dict: """ Get statistics about the traffic cache. Returns: Total records, unique routes, hosts, time range, and size. """ db = _get_db() stats = {} row = db.execute("SELECT COUNT(*) as count FROM traffic").fetchone() stats["total_records"] = row["count"] row = db.execute("SELECT COUNT(DISTINCT route_pattern) as count FROM traffic").fetchone() stats["unique_routes"] = row["count"] row = db.execute("SELECT COUNT(DISTINCT host) as count FROM traffic").fetchone() stats["unique_hosts"] = row["count"] row = db.execute("SELECT MIN(captured_at) as min_ts, MAX(captured_at) as max_ts FROM traffic").fetchone() stats["oldest_record"] = row["min_ts"] stats["newest_record"] = row["max_ts"] rows = db.execute(""" SELECT route_pattern, host, COUNT(*) as count FROM traffic GROUP BY route_pattern, host ORDER BY count DESC LIMIT 10 """).fetchall() stats["top_routes"] = [{"route": r["route_pattern"], "host": r["host"], "count": r["count"]} for r in rows] if DB_PATH.exists(): stats["cache_size_mb"] = round(DB_PATH.stat().st_size / (1024 * 1024), 2) db.close() return stats # ============================================================================= # MCP Tools - Route Discovery # ============================================================================= @mcp.tool() def get_routes(host: str = None, min_count: int = 1) -> dict: """ Discover all routes in the traffic cache, grouped by normalized pattern. Best for: Understanding what endpoints exist in your API. Args: host: Optional filter by host min_count: Minimum request count to include (default 1) Returns: List of routes with request counts, methods, and status codes. """ db = _get_db() if host: rows = db.execute(""" SELECT route_pattern, host, COUNT(*) as request_count, GROUP_CONCAT(DISTINCT method) as methods, GROUP_CONCAT(DISTINCT status_code) as status_codes, MIN(captured_at) as first_seen, MAX(captured_at) as last_seen FROM traffic WHERE host LIKE ? GROUP BY route_pattern, host HAVING COUNT(*) >= ? ORDER BY request_count DESC """, (f"%{host}%", min_count)).fetchall() else: rows = db.execute(""" SELECT route_pattern, host, COUNT(*) as request_count, GROUP_CONCAT(DISTINCT method) as methods, GROUP_CONCAT(DISTINCT status_code) as status_codes, MIN(captured_at) as first_seen, MAX(captured_at) as last_seen FROM traffic GROUP BY route_pattern, host HAVING COUNT(*) >= ? ORDER BY request_count DESC """, (min_count,)).fetchall() routes = [] for row in rows: routes.append({ "route": row["route_pattern"], "host": row["host"], "request_count": row["request_count"], "methods": row["methods"].split(",") if row["methods"] else [], "status_codes": [int(s) for s in row["status_codes"].split(",") if s] if row["status_codes"] else [], "first_seen": row["first_seen"], "last_seen": row["last_seen"] }) db.close() return { "route_count": len(routes), "routes": routes } @mcp.tool() def get_route_examples( route_pattern: str, host: str = None, max_examples: int = 5, include_errors: bool = True, dedupe_by: str = "shape" ) -> dict: """ Get representative examples for a specific route pattern. Best for: Understanding request/response shapes for a specific endpoint. Args: route_pattern: The normalized route pattern (e.g., "/users/{id}") host: Optional filter by host max_examples: Maximum examples to return (default 5) include_errors: Include 4xx/5xx responses (default True) dedupe_by: Deduplication strategy - "shape" (unique request bodies by hash), "status" (one per status code), or "none" Returns: List of request/response examples. """ db = _get_db() query = "SELECT * FROM traffic WHERE route_pattern = ?" params = [route_pattern] if host: query += " AND host LIKE ?" params.append(f"%{host}%") if not include_errors: query += " AND (status_code < 400 OR status_code IS NULL)" query += " ORDER BY captured_at DESC LIMIT 100" rows = db.execute(query, params).fetchall() db.close() examples = [] seen_shapes = set() seen_statuses = set() for row in rows: if len(examples) >= max_examples: break if dedupe_by == "shape": shape_key = _hash_body(row["request_body"] or "") or "empty" if shape_key in seen_shapes: continue seen_shapes.add(shape_key) elif dedupe_by == "status": if row["status_code"] in seen_statuses: continue seen_statuses.add(row["status_code"]) example = { "method": row["method"], "path": row["path"], "host": row["host"], "status_code": row["status_code"], "captured_at": row["captured_at"], "request": { "headers": json.loads(row["request_headers"]) if row["request_headers"] else {}, "body": row["request_body"] }, "response": { "headers": json.loads(row["response_headers"]) if row["response_headers"] else {}, "body": row["response_body"] } } examples.append(example) return { "route_pattern": route_pattern, "host": host, "example_count": len(examples), "examples": examples } # ============================================================================= # MCP Tools - Schema Inference # ============================================================================= @mcp.tool() def infer_schema( route_pattern: str, host: str = None, target: str = "both" ) -> dict: """ Infer JSON Schema from observed traffic for a route. Best for: Understanding the data contract for an endpoint. Note: Only uses successful (2xx) responses, limited to 100 samples. Args: route_pattern: The normalized route pattern (e.g., "/users/{id}") host: Optional filter by host target: What to analyze - "request", "response", or "both" (default) Returns: JSON Schema for request and/or response bodies. """ db = _get_db() query = "SELECT request_body, response_body, status_code FROM traffic WHERE route_pattern = ?" params = [route_pattern] if host: query += " AND host LIKE ?" params.append(f"%{host}%") query += " AND status_code >= 200 AND status_code < 300" query += " LIMIT 100" rows = db.execute(query, params).fetchall() db.close() if not rows: return {"error": f"No traffic found for route {route_pattern}"} result = { "route_pattern": route_pattern, "samples_analyzed": len(rows) } if target in ("request", "both"): request_bodies = [row["request_body"] for row in rows if row["request_body"]] if request_bodies: result["request_schema"] = _infer_schema_from_bodies(request_bodies, "Request") else: result["request_schema"] = {"note": "No request bodies found"} if target in ("response", "both"): response_bodies = [row["response_body"] for row in rows if row["response_body"]] if response_bodies: result["response_schema"] = _infer_schema_from_bodies(response_bodies, "Response") else: result["response_schema"] = {"note": "No response bodies found"} return result # ============================================================================= # MCP Tools - Code Generation # ============================================================================= @mcp.tool() def generate_types( route_pattern: str, host: str = None, format: str = "pydantic_v2", class_name: str = None, target: str = "both" ) -> dict: """ Generate type definitions from observed traffic. Best for: Creating type annotations for your codebase based on real traffic. Args: route_pattern: The normalized route pattern (e.g., "/users/{id}") host: Optional filter by host format: Output format - "json_schema", "pydantic_v2", "pydantic_v1", "typeddict", "dataclass" class_name: Override the generated class name (default: derived from route) target: What to generate - "request", "response", or "both" (default) Returns: Generated type definitions as code strings. """ schema_result = infer_schema(route_pattern, host, target) if "error" in schema_result: return schema_result result = { "route_pattern": route_pattern, "format": format, "samples_analyzed": schema_result.get("samples_analyzed", 0) } if not class_name: parts = route_pattern.strip("/").split("/") parts = [p for p in parts if not p.startswith("{")] class_name = _to_class_name("_".join(parts[-2:]) if len(parts) >= 2 else parts[-1] if parts else "Model") generators = { "json_schema": lambda s, n: json.dumps(s, indent=2), "pydantic_v2": _generate_pydantic_v2, "pydantic_v1": _generate_pydantic_v1, "typeddict": _generate_typeddict, "dataclass": _generate_dataclass, } if format not in generators: return {"error": f"Unknown format: {format}. Available: {', '.join(generators.keys())}"} generator = generators[format] if target in ("request", "both") and "request_schema" in schema_result: schema = schema_result["request_schema"] if "error" not in schema and schema.get("type") == "object": result["request_types"] = generator(schema, f"{class_name}Request") if target in ("response", "both") and "response_schema" in schema_result: schema = schema_result["response_schema"] if "error" not in schema and schema.get("type") == "object": result["response_types"] = generator(schema, f"{class_name}Response") return result # ============================================================================= # MCP Tools - Test Generation # ============================================================================= @mcp.tool() def generate_test_data( route_pattern: str, host: str = None, format: str = "json", max_examples: int = 10 ) -> dict: """ Generate test data from observed traffic. Best for: Creating test fixtures based on real API interactions. Args: route_pattern: The normalized route pattern (e.g., "/users/{id}") host: Optional filter by host format: Output format - "json" (raw data) or "pytest" (fixture code) max_examples: Maximum examples to include (default 10) Returns: Test data in requested format. """ examples_result = get_route_examples(route_pattern, host, max_examples, include_errors=True, dedupe_by="status") if "error" in examples_result: return examples_result examples = examples_result.get("examples", []) if not examples: return {"error": f"No examples found for route {route_pattern}"} if format == "json": test_cases = [] for i, ex in enumerate(examples): test_case = { "id": f"test_case_{i + 1}", "method": ex["method"], "path": ex["path"], "status_code": ex["status_code"], "request_body": None, "response_body": None } if ex["request"]["body"]: try: test_case["request_body"] = json.loads(ex["request"]["body"]) except json.JSONDecodeError: test_case["request_body"] = ex["request"]["body"] if ex["response"]["body"]: try: test_case["response_body"] = json.loads(ex["response"]["body"]) except json.JSONDecodeError: test_case["response_body"] = ex["response"]["body"] test_cases.append(test_case) return { "route_pattern": route_pattern, "format": "json", "test_cases": test_cases } elif format == "pytest": fixture_name = _to_class_name(route_pattern.strip("/").replace("/", "_").replace("{", "").replace("}", "")).lower() lines = [ "import pytest", "from typing import Any", "", "", f"# Test data for route: {route_pattern}", f"# Host: {host or 'any'}", f"# Generated from {len(examples)} observed requests", "", "", f"@pytest.fixture", f"def {fixture_name}_test_cases() -> list[dict[str, Any]]:", ' """Test cases captured from real traffic."""', " return [" ] for i, ex in enumerate(examples): req_body = "None" res_body = "None" if ex["request"]["body"]: try: req_body = repr(json.loads(ex["request"]["body"])) except json.JSONDecodeError: req_body = repr(ex["request"]["body"]) if ex["response"]["body"]: try: res_body = repr(json.loads(ex["response"]["body"])) except json.JSONDecodeError: res_body = repr(ex["response"]["body"]) lines.append(" {") lines.append(f' "method": {repr(ex["method"])},') lines.append(f' "path": {repr(ex["path"])},') lines.append(f' "status_code": {ex["status_code"]},') lines.append(f' "request_body": {req_body},') lines.append(f' "response_body": {res_body},') lines.append(" },") lines.append(" ]") lines.append("") lines.append("") lines.append(f"@pytest.fixture(params=[") for i in range(len(examples)): lines.append(f' "case_{i + 1}",') lines.append("])") lines.append(f"def {fixture_name}_case(request, {fixture_name}_test_cases):") lines.append(' """Parameterized test case fixture."""') lines.append(f" idx = int(request.param.split('_')[1]) - 1") lines.append(f" return {fixture_name}_test_cases[idx]") return { "route_pattern": route_pattern, "format": "pytest", "code": "\n".join(lines) } else: return {"error": f"Unknown format: {format}. Available: json, pytest"} # ============================================================================= # MCP Tools - Traffic Comparison # ============================================================================= @mcp.tool() def compare_traffic( route_pattern: str, host: str = None, baseline_minutes: int = 60 ) -> dict: """ Compare recent traffic against a baseline to detect schema drift. Best for: Validating changes haven't broken API contracts. Note: Only uses successful (2xx) responses, limited to 50 samples per period. Args: route_pattern: The normalized route pattern (e.g., "/users/{id}") host: Optional filter by host baseline_minutes: Compare traffic from last N minutes against older traffic (default 60) Returns: Comparison report showing new fields, removed fields, type changes. """ db = _get_db() cutoff = (datetime.now() - timedelta(minutes=baseline_minutes)).strftime('%Y-%m-%d %H:%M:%S') query = "SELECT request_body, response_body FROM traffic WHERE route_pattern = ? AND captured_at < ?" params = [route_pattern, cutoff] if host: query += " AND host LIKE ?" params.append(f"%{host}%") query += " AND status_code >= 200 AND status_code < 300 LIMIT 50" baseline_rows = db.execute(query, params).fetchall() query = "SELECT request_body, response_body FROM traffic WHERE route_pattern = ? AND captured_at >= ?" params = [route_pattern, cutoff] if host: query += " AND host LIKE ?" params.append(f"%{host}%") query += " AND status_code >= 200 AND status_code < 300 LIMIT 50" recent_rows = db.execute(query, params).fetchall() db.close() if not baseline_rows: return {"error": "No baseline traffic found (traffic older than cutoff)"} if not recent_rows: return {"error": "No recent traffic found"} baseline_req = [r["request_body"] for r in baseline_rows if r["request_body"]] baseline_res = [r["response_body"] for r in baseline_rows if r["response_body"]] recent_req = [r["request_body"] for r in recent_rows if r["request_body"]] recent_res = [r["response_body"] for r in recent_rows if r["response_body"]] result = { "route_pattern": route_pattern, "baseline_samples": len(baseline_rows), "recent_samples": len(recent_rows), "cutoff": cutoff, "changes": [] } def compare_schemas(baseline_bodies, recent_bodies, label): if not baseline_bodies or not recent_bodies: return baseline_schema = _infer_schema_from_bodies(baseline_bodies, "Baseline") recent_schema = _infer_schema_from_bodies(recent_bodies, "Recent") baseline_props = set(baseline_schema.get("properties", {}).keys()) recent_props = set(recent_schema.get("properties", {}).keys()) new_fields = recent_props - baseline_props removed_fields = baseline_props - recent_props if new_fields: result["changes"].append({ "type": "new_fields", "location": label, "fields": list(new_fields) }) if removed_fields: result["changes"].append({ "type": "removed_fields", "location": label, "fields": list(removed_fields) }) common = baseline_props & recent_props for field in common: baseline_type = baseline_schema.get("properties", {}).get(field, {}).get("type") recent_type = recent_schema.get("properties", {}).get(field, {}).get("type") if baseline_type != recent_type: result["changes"].append({ "type": "type_change", "location": label, "field": field, "baseline_type": baseline_type, "recent_type": recent_type }) compare_schemas(baseline_req, recent_req, "request") compare_schemas(baseline_res, recent_res, "response") result["has_drift"] = len(result["changes"]) > 0 return result if __name__ == "__main__": mcp.run() ``` Make it executable: ```bash chmod +x traffic_analysis_mcp.py ``` *** ## Connect to AI Assistants ### Option 1: Codex CLI [Codex CLI](https://github.com/openai/codex) is OpenAI's terminal-based coding assistant that supports MCP servers natively. **Add the MCP server to your config:** ```bash codex mcp add traffic-analysis -- python3 /path/to/traffic_analysis_mcp.py ``` Or edit `~/.codex/config.toml` directly: ```toml [mcp_servers.traffic-analysis] command = "python3" args = ["/path/to/traffic_analysis_mcp.py"] ``` {% hint style="info" %} **Using a virtual environment?** Replace `python3` with your venv Python path: ```bash codex mcp add traffic-analysis -- /path/to/venv/bin/python /path/to/traffic_analysis_mcp.py ``` {% endhint %} **Use it:** ```bash codex > Capture 2 minutes of traffic and show me what routes exist ``` *** ### Option 2: ChatGPT Developer Mode ChatGPT can connect to MCP servers via Developer Mode connectors. Since ChatGPT requires HTTPS, you'll need to expose your local server. **1. Start the MCP server with HTTP transport:** Create a separate entrypoint file `traffic_analysis_http.py` to avoid modifying the main script (which would break Codex/Claude stdio mode): ```python #!/usr/bin/env python3 # traffic_analysis_http.py - HTTP entrypoint for ChatGPT from traffic_analysis_mcp import mcp if __name__ == "__main__": mcp.run(transport="http", host="0.0.0.0", port=8081) ``` Then run it: ```bash python traffic_analysis_http.py ``` **2. Expose your local server via HTTPS tunnel:** {% tabs %} {% tab title="Cloudflared (Recommended)" %} ```bash # One-liner, no account required cloudflared tunnel --url http://localhost:8081 ``` This creates a temporary `*.trycloudflare.com` URL you can use immediately. {% endtab %} {% tab title="ngrok" %} ```bash ngrok http 8081 ``` Requires a free ngrok account. {% endtab %} {% endtabs %} **3. In ChatGPT:** * Go to Settings > Connectors > Advanced Settings * Enable Developer Mode * Add a new connector with your tunnel URL *** ### Option 3: Claude Code Claude Code supports MCP servers natively via the CLI. **Add the MCP server:** ```bash claude mcp add --transport stdio traffic-analysis -- python3 /path/to/traffic_analysis_mcp.py ``` {% hint style="info" %} **Using a virtual environment?** Replace `python3` with your venv Python path: ```bash claude mcp add --transport stdio traffic-analysis -- /path/to/venv/bin/python /path/to/traffic_analysis_mcp.py ``` {% endhint %} **Verify it's connected:** ```bash claude mcp list # Should show: traffic-analysis: ... - ✓ Connected ``` **Use it in Claude Code:** ``` Capture traffic while I run my tests, then generate types for the user API ``` *** ## Example Workflow ### Step 1: Start Traffic Generation Run your application, test suite, or any code that makes HTTP requests: ```bash # Option A: Run your app python my_app.py # Option B: Run your test suite pytest tests/ -v # Option C: Use curl/scripts ./run_api_tests.sh ``` ### Step 2: Capture Traffic While your application is running, ask your AI assistant to capture: ``` "Capture 2 minutes of traffic" ``` The assistant will call `store_traffic(seconds=120)` and report what was captured. {% hint style="info" %} **Note:** Traffic capture blocks for the specified duration. During a 2-minute capture, the assistant waits until capture completes before responding. {% endhint %} ### Step 3: Discover Routes ``` "What routes did we capture?" ``` The assistant calls `get_routes()` and shows you all endpoints grouped by pattern. ### Step 4: Examine Examples ``` "Show me examples for the /v1/charges endpoint" ``` The assistant calls `get_route_examples("/v1/charges")` to show request/response pairs. ### Step 5: Generate Types ``` "Generate Pydantic models for that endpoint" ``` The assistant calls `generate_types("/v1/charges", format="pydantic_v2")` to create type definitions. ### Step 6: Create Test Fixtures ``` "Create pytest fixtures from this traffic" ``` The assistant calls `generate_test_data("/v1/charges", format="pytest")` to generate test code. *** ## Example Prompts **Route Discovery:** * "What endpoints are we hitting?" * "Show me all routes for api.stripe.com" * "Which routes have error responses?" **Type Generation:** * "Generate types for the /users endpoint" * "Create Pydantic v1 models for backwards compatibility" * "Generate TypedDicts instead of Pydantic models" * "What does the response schema look like for /v1/charges?" **Test Generation:** * "Create test fixtures for the payment API" * "Generate JSON test data I can use in my tests" * "Build pytest fixtures with examples of error responses" **Schema Validation:** * "Has the user API schema changed since yesterday?" * "Compare recent traffic against the baseline" * "Are there any breaking changes in the response format?" **Cache Management:** * "How much traffic have we captured?" * "Clear traffic older than 1 hour" * "What's in the cache?" *** ## Example Output ### get\_routes() ```json { "route_count": 5, "routes": [ { "route": "/v1/charges", "host": "api.stripe.com", "request_count": 23, "methods": ["POST"], "status_codes": [200, 402], "first_seen": "2024-01-15 10:30:00", "last_seen": "2024-01-15 10:45:00" }, { "route": "/v1/customers/{id}", "host": "api.stripe.com", "request_count": 18, "methods": ["GET"], "status_codes": [200, 404], "first_seen": "2024-01-15 10:30:15", "last_seen": "2024-01-15 10:44:30" } ] } ``` ### infer\_schema() ```json { "route_pattern": "/v1/charges", "samples_analyzed": 23, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "Response", "type": "object", "properties": { "id": {"type": "string"}, "amount": {"type": "integer"}, "currency": {"type": "string"}, "status": {"type": "string"}, "created": {"type": "integer"}, "billing_details": { "type": "object", "properties": { "name": {"type": "string"}, "email": {"type": "string", "format": "email"} } } }, "required": ["id", "amount", "currency", "status", "created"] } } ``` ### generate\_types() - Pydantic v2 ```python from pydantic import BaseModel, Field from typing import Any class BillingDetails(BaseModel): name: str email: str | None = Field(default=None) class ChargesResponse(BaseModel): id: str amount: int currency: str status: str created: int billing_details: BillingDetails | None = Field(default=None) ``` *** ## Path Normalization Routes are automatically normalized to group similar paths: | Raw Path | Normalized Pattern | | ------------------------------------ | ----------------------- | | `/users/123` | `/users/{id}` | | `/users/456` | `/users/{id}` | | `/orders/abc-def-123-456-789` | `/orders/{uuid}` | | `/events/2024-01-15` | `/events/{date}` | | `/sessions/abc123def456789012345678` | `/sessions/{object_id}` | | `/tokens/sk_test_abc123...` | `/tokens/{token}` | This grouping ensures you get meaningful aggregates even with high-volume traffic containing many unique IDs. *** ## Supported Type Formats | Format | Description | Use Case | | ------------- | ------------------------------ | -------------------------- | | `json_schema` | Raw JSON Schema (RFC draft-07) | Language-agnostic, OpenAPI | | `pydantic_v2` | Pydantic v2 BaseModel | Modern Python | | `pydantic_v1` | Pydantic v1 compatible | Legacy Python | | `typeddict` | typing.TypedDict | Lightweight typing | | `dataclass` | @dataclass | Simple data containers | *** ## Troubleshooting ### "Cannot connect to DevTools API" Ensure Qtap is running with DevTools enabled: {% tabs %} {% tab title="Binary" %} ```bash sudo qtap --enable-dev-tools ``` {% endtab %} {% tab title="Docker" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` {% endtab %} {% endtabs %} Verify it's accessible: ```bash curl -s http://localhost:10001/devtools/api/events | head -3 ``` ### "No traffic captured" Traffic capture is **live and on-demand**, not persistent background collection. * Ensure traffic is flowing **while** `store_traffic()` runs * Run your app/tests in one terminal, capture in another * Try longer capture times: `store_traffic(seconds=120)` ### "No valid JSON bodies to analyze" The endpoint may return non-JSON responses (HTML, XML, binary). * Check `get_route_examples()` to see actual response bodies * Schema inference only works with JSON content ### "MCP server not found" Check your configuration path: ```bash # Codex cat ~/.codex/config.toml # Claude Code claude mcp list ``` Ensure the Python path is absolute and the script exists. *** ## Next Steps * [DevTools MCP Server](/guides/devtools-guides/devtools-mcp-server) - Real-time traffic debugging * [DevTools API Reference](/guides/devtools-guides/devtools-api) - Full API documentation * [DevTools Interface Guide](/guides/devtools-guides/devtools-interface-guide) - Browser UI walkthrough # Server-Side Event Filtering Server-side filtering lets you subscribe to specific event types with rulekit expressions, reducing bandwidth and eliminating client-side filtering. Instead of receiving all events and discarding most, tell Qtap exactly what you want. {% hint style="info" %} **New in v0.14.0**: Server-side filtering requires Qtap v0.14.0 or later. Earlier versions stream all events without filtering. {% endhint %} *** ## Why Server-Side Filtering? | Without Filtering | With Filtering | | -------------------------------------- | -------------------------------- | | Receive all events, filter client-side | Receive only matching events | | High bandwidth in busy environments | Minimal bandwidth usage | | Client processes every event | Server handles filtering | | Same data for all consumers | Different filters per connection | Use cases: * **High-traffic environments**: Subscribe only to errors instead of all HTTP transactions * **Targeted monitoring**: Watch specific containers or processes * **Multi-consumer scenarios**: Different dashboards subscribe to different event types *** ## Quick Start Subscribe to HTTP errors only (status >= 400): ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "res.status >= 400" } }' ``` The response stream starts with `system.connected` confirming your subscription: ``` event: system.connected data: {"data":{"topics":{"http":"res.status >= 400"}},"ts":"2026-01-20T19:36:50.978Z"} ``` *** ## API Reference ### Endpoint ``` POST http://localhost:10001/devtools/api/events Content-Type: application/json ``` ### Subscription Payload The JSON payload uses a `topics` object to specify subscriptions: ```json { "topics": { "process": "*", "connection": "*", "http": "res.status >= 400" } } ``` {% hint style="warning" %} **Topic names must be lowercase**: Use `http`, `process`, `connection` - not capitalized versions. {% endhint %} ### Subscription Semantics | Payload | Behavior | | ------------------ | ------------------------------------ | | `"*"` | Subscribe to all events of that type | | Filter expression | Subscribe to matching events only | | Omit topic | No events of that type | | Empty payload `{}` | Default: all topics with `"*"` | ### Examples ```bash # All events (default behavior) curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{}' # HTTP only, no process or connection events curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "*" } }' # Processes only curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "process": "*" } }' ``` *** ## Filter Fields by Topic Native properties (belonging to the event type) have no prefix. Relational properties (from associated entities) are prefixed. {% hint style="info" %} **Verified Fields**: Fields marked with ✓ have been tested and verified to work with server-side filtering. Fields in the "Not Available" section are present in event data but cannot be used for server-side filtering. {% endhint %} ### Process Events Fields available for `process.started` and `process.stopped` events: | Field | Description | Example | | ------------------------ | --------------------------------- | ----------------------------- | | `binary` ✓ | Executable name | `python3.11`, `node`, `nginx` | | `path` ✓ | Full path to binary | `/usr/local/bin/python3.11` | | `hostname` ✓ | Process hostname | `payment-service` | | `user.name` ✓ | Username | `root`, `www-data` | | `user.id` ✓ | User ID | `0`, `1000` | | `container.name` ✓ | Container name | `my-app` | | `container.id` ✓ | Container ID | `1781322a598f` | | `container.image` ✓ | Container image | `python:3.11-slim` | | `container.labels.` | Container label value | `container.labels.app` | | `pod.name` | Pod name (Kubernetes only) | `payment-service-7d8f9` | | `pod.namespace` | Pod namespace (Kubernetes only) | `production` | | `pod.labels.` | Pod label value (Kubernetes only) | `pod.labels.app` | ### Connection Events Native connection properties + relational `process.*` fields: | Field | Description | Example | | ------------------------ | --------------------------------- | ----------------------------------------------- | | `direction` ✓ | Connection direction | `egress-external`, `egress-internal`, `ingress` | | `protocol` ✓ | L7 protocol | `http1`, `http2`, `other` | | `type` ✓ | Socket type | `tcp`, `udp` | | `src.ip` ✓ | Source IP address | `172.18.0.7` | | `src.port` ✓ | Source port | `34438` | | `dst.ip` ✓ | Destination IP address | `52.4.128.73` | | `dst.port` ✓ | Destination port | `443`, `80` | | `process.binary` ✓ | Process executable name | `python3.11` | | `process.path` ✓ | Full executable path | `/usr/local/bin/python3.11` | | `process.hostname` ✓ | Process hostname | `payment-service` | | `process.user.name` ✓ | Username | `root` | | `process.user.id` ✓ | User ID | `0` | | `container.name` ✓ | Container name | `payment-service` | | `container.id` ✓ | Container ID | `1781322a598f` | | `container.image` ✓ | Container image | `python:3.11-slim` | | `container.labels.` | Container label value | `container.labels.app` | | `pod.name` | Pod name (Kubernetes only) | `payment-service-7d8f9` | | `pod.namespace` | Pod namespace (Kubernetes only) | `production` | | `pod.labels.` | Pod label value (Kubernetes only) | `pod.labels.app` | **Not available for server-side filtering** (present in event data only): * `dst.domain` - Use `req.host` on HTTP topic instead * `tls.enabled`, `tls.version`, `tls.sni` - TLS info available in event data ### HTTP Events Native HTTP properties (`req.*`/`res.*`) + relational connection/process fields: | Field | Description | Example | | ------------------------ | --------------------------------- | ----------------------------------------------- | | `req.method` ✓ | HTTP method | `GET`, `POST`, `PUT`, `DELETE` | | `req.host` ✓ | Request host | `api.stripe.com` | | `res.status` ✓ | Response status code | `200`, `404`, `500` | | `direction` ✓ | Connection direction | `egress-external`, `egress-internal`, `ingress` | | `protocol` ✓ | L7 protocol | `http1`, `http2` | | `src.ip` ✓ | Source IP address | `172.18.0.7` | | `src.port` ✓ | Source port | `34438` | | `dst.ip` ✓ | Destination IP address | `52.4.128.73` | | `dst.port` ✓ | Destination port | `443`, `80` | | `process.binary` ✓ | Process executable name | `python3.11` | | `process.path` ✓ | Full executable path | `/usr/local/bin/python3.11` | | `process.hostname` ✓ | Process hostname | `payment-service` | | `process.user.name` ✓ | Username | `root` | | `container.name` ✓ | Container name | `payment-service` | | `container.id` ✓ | Container ID | `1781322a598f` | | `container.image` ✓ | Container image | `python:3.11-slim` | | `container.labels.` | Container label value | `container.labels.app` | | `pod.name` | Pod name (Kubernetes only) | `payment-service-7d8f9` | | `pod.namespace` | Pod namespace (Kubernetes only) | `production` | | `pod.labels.` | Pod label value (Kubernetes only) | `pod.labels.app` | **Not available for server-side filtering** (present in event data only): * `req.path`, `req.url` - Full path/URL available in decoded event payload * `dst.domain` - Use `req.host` instead * `tls.enabled`, `tls.version`, `tls.sni` - TLS info available in event data * `process.user.id` - Use `process.user.name` instead {% hint style="info" %} **Cross-Entity Filtering**: Filters can apply across related entities. For example, filtering HTTP events by `container.name == "nginx"` will match HTTP requests from that container via connection cache lookup. {% endhint %} *** ## Operators ### Comparison Operators | Operator | Description | Example | | -------- | ---------------- | --------------------- | | `==` | Equal | `res.status == 200` | | `!=` | Not equal | `req.method != "GET"` | | `>` | Greater than | `res.status > 399` | | `>=` | Greater or equal | `res.status >= 400` | | `<` | Less than | `res.status < 400` | | `<=` | Less or equal | `src.port <= 1024` | ### String Operators | Operator | Description | Example | | ---------- | --------------- | ------------------------------------- | | `contains` | Substring match | `req.host contains "api."` | | `matches` | Regex match | `req.host matches /api\.[a-z]+\.com/` | ### Logical Operators | Operator | Description | Example | | -------- | ---------------- | -------------------------------------------- | | `AND` | Both conditions | `res.status >= 400 AND req.method == "POST"` | | `OR` | Either condition | `res.status == 500 OR res.status == 503` | | `NOT` | Negate condition | `NOT req.method == "GET"` | ### Special Operators | Operator | Description | Example | | -------- | ------------- | ----------------------------------------- | | `in` | Value in list | `req.method in ["POST", "PUT", "DELETE"]` | | `in` | IP in CIDR | `dst.ip in "10.0.0.0/8"` | *** ## Custom Functions ### in\_zone(domain, zone) Check if a domain belongs to a DNS zone: ``` in_zone(req.host, "stripe.com") ``` Matches `api.stripe.com`, `dashboard.stripe.com`, `stripe.com`. ### zone(domain) Extract the zone from a domain for comparison: ``` zone(req.host) == "stripe.com" ``` *** ## Examples ### Subscribe to All Events (Default) ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{}' ``` Response confirms subscription to all topics: ``` event: system.connected data: {"data":{"topics":{"connection":"*","http":"*","process":"*"}}} ``` ### HTTP Errors Only ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "res.status >= 400" } }' ``` ### Specific Container Traffic ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "container.name == \"stripe-payment-service\"", "connection": "container.name == \"stripe-payment-service\"" } }' ``` ### HTTPS Connections Only (Port 443) ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "connection": "dst.port == 443" } }' ``` ### Specific Process Binary ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "process": "binary == \"python3.11\"" } }' ``` ### Multiple Topics with Different Filters ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "process": "*", "connection": "dst.port == 443", "http": "res.status >= 400" } }' ``` ### Cross-Entity Filtering (HTTP by Process) Filter HTTP transactions by the originating process: ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "process.binary == \"python3.11\"" } }' ``` ### Complex Filters Combine multiple conditions: ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{ "topics": { "http": "res.status >= 500 AND req.method in [\"POST\", \"PUT\", \"DELETE\"]" } }' ``` *** ## Best Practices ### Start Broad, Narrow Down Begin with permissive filters to verify events are flowing, then add constraints: ```bash # Step 1: Verify HTTP events are captured curl -sN -X POST ... -d '{"topics": {"http": "*"}}' # Step 2: Add status filter curl -sN -X POST ... -d '{"topics": {"http": "res.status >= 400"}}' # Step 3: Add container filter curl -sN -X POST ... -d '{"topics": {"http": "res.status >= 400 AND container.name == \"my-app\""}}' ``` ### Verify Subscriptions Always check the `system.connected` event to confirm your filters were accepted: ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{"topics": {"http": "res.status >= 400"}}' | head -3 ``` Expected output: ``` event: system.connected data: {"data":{"topics":{"http":"res.status >= 400"}}} ``` ### Use Specific Topics Only subscribe to event types you need: ```bash # Good: Only HTTP events curl -sN -X POST ... -d '{"topics": {"http": "*"}}' # Wasteful: All events when you only need HTTP curl -sN -X POST ... -d '{}' ``` *** ## Troubleshooting ### No Events Received 1. **Check subscription**: Verify `system.connected` shows your topics 2. **Generate traffic**: The filter might be too restrictive 3. **Test with `"*"`**: Remove filters to confirm events are flowing 4. **Check Qtap version**: Server-side filtering requires v0.14.0+ ### Filter Syntax Errors Invalid filters cause the connection to fail. Test expressions incrementally: ```bash # Start simple curl -sN -X POST ... -d '{"topics": {"http": "res.status >= 400"}}' # Then add complexity curl -sN -X POST ... -d '{"topics": {"http": "res.status >= 400 AND process.binary == \"python3.11\""}}' ``` ### Common Mistakes | Mistake | Fix | | -------------------------------- | ---------------------------------------------------- | | `"HTTP"` instead of `"http"` | Use lowercase: `"http"`, `"process"`, `"connection"` | | Flat payload `{"http": "..."}` | Use topics wrapper: `{"topics": {"http": "..."}}` | | `status` instead of `res.status` | Use full field path: `res.status >= 400` | | `method` instead of `req.method` | Use full field path: `req.method == "POST"` | | Wrong port (10002) | Use port `10001` | ### Events Don't Match Filter 1. Verify field names match the schema (case-sensitive) 2. Check string values are quoted correctly 3. Use `"*"` first to see actual field values in events 4. Test filter logic with simpler conditions *** ## Related Documentation * [DevTools API](/guides/devtools-guides/devtools-api) - Full API reference and event schemas * [Getting Started with DevTools](/guides/devtools-guides/getting-started-with-devtools) - Initial setup guide # Enhanced MCP with Server-Side Filtering Upgrade your MCP server to use server-side event filtering for optimized AI-powered troubleshooting. Instead of receiving all events and filtering in Python, tell Qtap exactly what you want and receive only matching events. {% hint style="info" %} **Requires Qtap v0.14.0+**: Server-side filtering is available in Qtap v0.14.0 and later. See [Server-Side Event Filtering](/guides/devtools-guides/server-side-event-filtering) for the full filter reference. {% endhint %} **What you'll build:** ``` You: "Show me errors from the payment service" AI: [Calling get_container_errors with filter: res.status >= 400 AND container.name == "payment-service"] "Found 3 errors in payment-service over the last 60 seconds: - POST /v1/charges → 402 Payment Required (insufficient funds) - POST /v1/charges → 500 Internal Server Error (Stripe timeout) - GET /v1/customers/cus_xxx → 404 Not Found ..." ``` *** ## Why Server-Side Filtering? | Aspect | Standard MCP (Client-Side) | Enhanced MCP (Server-Side) | | ---------- | --------------------------- | ---------------------------- | | Network | Receive 1000s of events | Receive only matching events | | Processing | Decode/filter all in Python | Server handles filtering | | Latency | Wait for full capture | Faster results | | Bandwidth | High in busy environments | Minimal | | Use Case | General exploration | Targeted troubleshooting | **When to use this approach:** * High-traffic environments where bandwidth matters * Incident response requiring focused debugging * Container-scoped troubleshooting in microservices * Security audits targeting specific patterns *** ## Architecture Comparison ### Standard MCP Server ``` DevTools API ──GET──> All Events ──> Python MCP Server ──> Filter ──> AI (1000s) (decode all) (discard most) ``` ### Enhanced MCP Server ``` DevTools API ──POST + Filter──> Matching Events Only ──> Python MCP Server ──> AI (10s or 100s) (decode matches) ``` The server-side filter is applied at the Qtap level before events are streamed, dramatically reducing network traffic and processing overhead. *** ## Enhanced MCP Server Code Save this as `devtools_mcp_enhanced.py`: ```python #!/usr/bin/env python3 """ Enhanced MCP Server with Server-Side Filtering for Qtap DevTools. Uses POST with filter expressions (v0.14.0+) for targeted event capture. Works with Codex CLI, ChatGPT, and Claude Code. Tools: - get_errors: HTTP errors with server-side status filtering - get_errors_by_container: Errors scoped to specific container - get_container_traffic: All traffic for a specific container - get_process_traffic: All traffic from a specific process - get_external_traffic: External egress traffic for API auditing - get_traffic_filtered: Custom filter expression support - get_traffic_summary: Quick overview of patterns - get_hosts: External API dependency mapping - check_sensitive_data: Security audit for leaked secrets """ import base64 import json import re import time import httpx from mcp.server.fastmcp import FastMCP mcp = FastMCP("Qtap DevTools Enhanced") DEVTOOLS_URL = "http://localhost:10001/devtools/api/events" # Sensitive data patterns for security scanning SENSITIVE_PATTERNS = [ (r'[Aa]uthorization', 'Authorization header'), (r'[Bb]earer\s+[A-Za-z0-9\-_\.]+', 'Bearer token'), (r'[Aa]pi[-_]?[Kk]ey', 'API key'), (r'[Ss]ecret', 'Secret'), (r'[Pp]assword', 'Password'), (r'[Tt]oken', 'Token'), (r'[Cc]ookie', 'Cookie'), (r'[Ss]ession[-_]?[Ii]d', 'Session ID'), (r'[Cc]redential', 'Credential'), (r'[Pp]rivate[-_]?[Kk]ey', 'Private key'), (r'sk-[a-zA-Z0-9]{20,}', 'OpenAI API key'), (r'ghp_[a-zA-Z0-9]{36}', 'GitHub PAT'), (r'xox[baprs]-[a-zA-Z0-9-]+', 'Slack token'), ] def _capture_http_filtered( seconds: int = 30, max_http: int = 500, filter_expr: str = None ) -> list | dict: """ Capture HTTP transactions with optional server-side filtering (v0.14.0+). Args: seconds: Capture duration (max 300) max_http: Maximum HTTP transactions to return filter_expr: Rulekit filter expression for server-side filtering Returns: List of decoded HTTP transactions or error dict """ seconds = min(seconds, 300) try: with httpx.Client(timeout=seconds + 10) as client: events = [] first_event_time = None # Build request based on filter if filter_expr: # Server-side filtering via POST payload = {"topics": {"http": filter_expr}} response_ctx = client.stream( "POST", DEVTOOLS_URL, json=payload, timeout=seconds + 10 ) else: # No filter - GET all events response_ctx = client.stream( "GET", DEVTOOLS_URL, timeout=seconds + 10 ) with response_ctx as response: current_event_type = None start = time.time() for line in response.iter_lines(): if time.time() - start > seconds: break if line.startswith("event: "): current_event_type = line[7:] elif line.startswith("data: "): try: event_data = json.loads(line[6:]) event_ts = event_data.get("ts", "") if first_event_time is None: first_event_time = event_ts # Only process HTTP transactions if current_event_type == "request.http_transaction": decoded = _decode_http_transaction({ "type": current_event_type, "ts": event_ts, "data": event_data.get("data", {}) }) if decoded: events.append(decoded) except json.JSONDecodeError: pass if len(events) >= max_http: break return events except httpx.ConnectError: return { "error": "Cannot connect to DevTools API at localhost:10001", "hint": "Start Qtap with --enable-dev-tools or ENABLE_DEV_TOOLS=true" } except Exception as e: return {"error": str(e)} def _decode_http_transaction(event: dict) -> dict | None: """Decode a raw http_transaction event into structured data.""" inner = event.get("data", {}) if isinstance(inner, dict) and "data" in inner: payload = inner.get("data") if isinstance(payload, dict) and "data" in payload: payload = payload.get("data") if not isinstance(payload, str): return None try: decoded = json.loads(base64.b64decode(payload)) # Decode response body res = decoded.get("response", {}) if res.get("body"): try: res["body"] = base64.b64decode(res["body"]).decode('utf-8', errors='replace') except: res["body"] = "[binary data]" # Decode request body req = decoded.get("request", {}) if req.get("body"): try: req["body"] = base64.b64decode(req["body"]).decode('utf-8', errors='replace') except: req["body"] = "[binary data]" return { "ts": event.get("ts"), "metadata": decoded.get("metadata", {}), "request": req, "response": res } except: pass return None # ============================================================================= # Server-Side Filtered Tools # ============================================================================= @mcp.tool() def get_errors(seconds: int = 60) -> dict: """ Get HTTP errors (4xx/5xx) using server-side filtering. Best for: "What's failing?", "Show me all errors", "Debug 500s" Uses server-side filter: res.status >= 400 Args: seconds: Capture duration (default 60, max 300) Returns: Error requests with full details for debugging. """ events = _capture_http_filtered( seconds=seconds, max_http=500, filter_expr="res.status >= 400" ) if isinstance(events, dict) and "error" in events: return events errors = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) errors.append({ "ts": event.get("ts"), "host": req.get("authority"), "method": req.get("method"), "path": req.get("path"), "status": res.get("status"), "request_headers": req.get("headers", {}), "request_body": req.get("body", "")[:1000] if req.get("body") else None, "response_headers": res.get("headers", {}), "response_body": res.get("body", "")[:3000] if res.get("body") else None }) by_status = {} for err in errors: status = str(err["status"]) by_status[status] = by_status.get(status, 0) + 1 return { "filter_used": "res.status >= 400", "capture_seconds": seconds, "error_count": len(errors), "by_status": by_status, "errors": errors[:200] } @mcp.tool() def get_errors_by_container(container_name: str, seconds: int = 60) -> dict: """ Get HTTP errors from a specific container using server-side filtering. Best for: "What's failing in payment-service?", "Errors from my-app container" Uses server-side filter: res.status >= 400 AND container.name == "..." Args: container_name: Docker container name to filter by seconds: Capture duration (default 60, max 300) Returns: Errors from the specified container only. """ filter_expr = f'res.status >= 400 AND container.name == "{container_name}"' events = _capture_http_filtered( seconds=seconds, max_http=500, filter_expr=filter_expr ) if isinstance(events, dict) and "error" in events: return events errors = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) errors.append({ "ts": event.get("ts"), "host": req.get("authority"), "method": req.get("method"), "path": req.get("path"), "status": res.get("status"), "response_body": res.get("body", "")[:3000] if res.get("body") else None }) return { "filter_used": filter_expr, "container": container_name, "capture_seconds": seconds, "error_count": len(errors), "errors": errors[:100] } @mcp.tool() def get_process_traffic(process_name: str, seconds: int = 60, max_requests: int = 200) -> dict: """ Get all HTTP traffic from a specific process using server-side filtering. Best for: "What's python doing?", "Traffic from node", "Process debugging" Uses server-side filter: process.binary == "..." Args: process_name: Process binary name to filter by (e.g., "python3.11", "node") seconds: Capture duration (default 60, max 300) max_requests: Maximum requests to return (default 200) Returns: All HTTP traffic from the specified process. """ filter_expr = f'process.binary == "{process_name}"' events = _capture_http_filtered( seconds=seconds, max_http=max_requests, filter_expr=filter_expr ) if isinstance(events, dict) and "error" in events: return events requests = [] hosts = {} methods = {} status_codes = {} for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") method = req.get("method", "?") status = res.get("status", 0) hosts[host] = hosts.get(host, 0) + 1 methods[method] = methods.get(method, 0) + 1 status_codes[str(status)] = status_codes.get(str(status), 0) + 1 requests.append({ "ts": event.get("ts"), "host": host, "method": method, "path": req.get("path"), "status": status }) return { "filter_used": filter_expr, "process": process_name, "capture_seconds": seconds, "total_requests": len(requests), "hosts": dict(sorted(hosts.items(), key=lambda x: -x[1])), "methods": methods, "status_codes": dict(sorted(status_codes.items())), "requests": requests } @mcp.tool() def get_container_traffic(container_name: str, seconds: int = 60, max_requests: int = 200) -> dict: """ Get all HTTP traffic from a specific container using server-side filtering. Best for: "What's payment-service doing?", "Traffic from my-app" Uses server-side filter: container.name == "..." Args: container_name: Docker container name to filter by seconds: Capture duration (default 60, max 300) max_requests: Maximum requests to return (default 200) Returns: All HTTP traffic from the specified container. """ filter_expr = f'container.name == "{container_name}"' events = _capture_http_filtered( seconds=seconds, max_http=max_requests, filter_expr=filter_expr ) if isinstance(events, dict) and "error" in events: return events requests = [] hosts = {} methods = {} status_codes = {} for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") method = req.get("method", "?") status = res.get("status", 0) hosts[host] = hosts.get(host, 0) + 1 methods[method] = methods.get(method, 0) + 1 status_codes[str(status)] = status_codes.get(str(status), 0) + 1 requests.append({ "ts": event.get("ts"), "host": host, "method": method, "path": req.get("path"), "status": status }) return { "filter_used": filter_expr, "container": container_name, "capture_seconds": seconds, "total_requests": len(requests), "hosts": dict(sorted(hosts.items(), key=lambda x: -x[1])), "methods": methods, "status_codes": dict(sorted(status_codes.items())), "requests": requests } @mcp.tool() def get_external_traffic(seconds: int = 60, max_requests: int = 200) -> dict: """ Get external egress HTTP traffic using server-side filtering. Best for: "What external APIs are we calling?", "Outbound traffic audit" Uses server-side filter: direction == "egress-external" Args: seconds: Capture duration (default 60, max 300) max_requests: Maximum requests to return (default 200) Returns: External HTTP requests for API dependency analysis. """ filter_expr = 'direction == "egress-external"' events = _capture_http_filtered( seconds=seconds, max_http=max_requests, filter_expr=filter_expr ) if isinstance(events, dict) and "error" in events: return events requests = [] by_host = {} for event in events: req = event.get("request", {}) res = event.get("response", {}) meta = event.get("metadata", {}) host = req.get("authority", "unknown") by_host[host] = by_host.get(host, 0) + 1 requests.append({ "ts": event.get("ts"), "host": host, "method": req.get("method"), "path": req.get("path"), "status": res.get("status"), "process": meta.get("process_exe") }) return { "filter_used": filter_expr, "capture_seconds": seconds, "external_requests": len(requests), "by_host": dict(sorted(by_host.items(), key=lambda x: -x[1])), "requests": requests } @mcp.tool() def get_traffic_filtered(filter_expr: str, seconds: int = 60, max_requests: int = 200) -> dict: """ Get HTTP traffic with a custom server-side filter expression. Best for: Complex queries requiring custom filter logic. Filter examples: - "res.status >= 500" - Server errors only - "req.method == \"POST\"" - POST requests only - "req.host == \"api.stripe.com\"" - Stripe API calls - "process.binary == \"python3.11\"" - Python process traffic - "res.status >= 400 AND container.name == \"my-app\"" - Container errors - "direction == \"egress-external\"" - External API traffic Args: filter_expr: Rulekit filter expression seconds: Capture duration (default 60, max 300) max_requests: Maximum requests to return (default 200) Returns: HTTP transactions matching the filter. """ events = _capture_http_filtered( seconds=seconds, max_http=max_requests, filter_expr=filter_expr ) if isinstance(events, dict) and "error" in events: return events requests = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) meta = event.get("metadata", {}) requests.append({ "ts": event.get("ts"), "host": req.get("authority"), "method": req.get("method"), "path": req.get("path"), "status": res.get("status"), "process": meta.get("process_exe"), "request_headers": req.get("headers", {}), "response_body": res.get("body", "")[:2000] if res.get("body") else None }) return { "filter_used": filter_expr, "capture_seconds": seconds, "match_count": len(requests), "requests": requests } # ============================================================================= # Standard Tools (No Server-Side Filtering) # ============================================================================= @mcp.tool() def get_traffic_summary(seconds: int = 60) -> dict: """ Quick overview of HTTP traffic patterns. Best for: "What's happening?", "Traffic overview", "Any errors?" Args: seconds: Capture duration (default 60, max 300) Returns: Host counts, method breakdown, status distribution. """ events = _capture_http_filtered(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events hosts = {} status_codes = {} methods = {} errors = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") method = req.get("method", "?") status = res.get("status", 0) hosts[host] = hosts.get(host, 0) + 1 methods[method] = methods.get(method, 0) + 1 status_codes[str(status)] = status_codes.get(str(status), 0) + 1 if status >= 400: errors.append({ "host": host, "method": method, "path": req.get("path", "/")[:60], "status": status }) return { "total_requests": len(events), "unique_hosts": len(hosts), "hosts": dict(sorted(hosts.items(), key=lambda x: -x[1])[:25]), "methods": methods, "status_codes": dict(sorted(status_codes.items())), "error_count": len(errors), "errors": errors[:30] } @mcp.tool() def get_hosts(seconds: int = 60) -> dict: """ Map all external hosts/APIs being called. Best for: "What APIs are we calling?", "External dependencies" Args: seconds: Capture duration (default 60, max 300) Returns: Hosts with request counts, methods, status codes, paths. """ events = _capture_http_filtered(seconds=seconds, max_http=1000) if isinstance(events, dict) and "error" in events: return events hosts = {} for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") if host not in hosts: hosts[host] = { "count": 0, "methods": set(), "status_codes": set(), "paths": [], "errors": 0 } hosts[host]["count"] += 1 hosts[host]["methods"].add(req.get("method", "?")) hosts[host]["status_codes"].add(res.get("status", 0)) if res.get("status", 0) >= 400: hosts[host]["errors"] += 1 if len(hosts[host]["paths"]) < 5: hosts[host]["paths"].append(req.get("path", "/")[:60]) result = {} for host, data in sorted(hosts.items(), key=lambda x: -x[1]["count"]): result[host] = { "requests": data["count"], "errors": data["errors"], "methods": sorted(data["methods"]), "status_codes": sorted(data["status_codes"]), "sample_paths": data["paths"] } return { "capture_seconds": seconds, "total_requests": len(events), "unique_hosts": len(result), "hosts": result } @mcp.tool() def check_sensitive_data(seconds: int = 60) -> dict: """ Scan traffic for sensitive data: API keys, tokens, passwords. Best for: "Are we leaking secrets?", "Security audit" Args: seconds: Capture duration (default 60, max 300) Returns: Report of sensitive patterns found. """ events = _capture_http_filtered(seconds=seconds, max_http=500) if isinstance(events, dict) and "error" in events: return events findings = [] for event in events: req = event.get("request", {}) res = event.get("response", {}) host = req.get("authority", "unknown") path = req.get("path", "/") event_findings = [] # Check request headers for header, value in req.get("headers", {}).items(): for pattern, label in SENSITIVE_PATTERNS: if re.search(pattern, header, re.IGNORECASE) or re.search(pattern, str(value)): event_findings.append({ "location": f"request header: {header}", "type": label, "preview": str(value)[:80] + ("..." if len(str(value)) > 80 else "") }) break # Check response headers for header, value in res.get("headers", {}).items(): for pattern, label in SENSITIVE_PATTERNS: if re.search(pattern, header, re.IGNORECASE): event_findings.append({ "location": f"response header: {header}", "type": label }) break # Check request body req_body = str(req.get("body", "")) if req_body: for pattern, label in SENSITIVE_PATTERNS: match = re.search(f'.{{0,30}}{pattern}.{{0,30}}', req_body, re.IGNORECASE) if match: event_findings.append({ "location": "request body", "type": label, "context": match.group(0) }) # Check response body res_body = str(res.get("body", "")) if res_body and len(res_body) < 50000: for pattern, label in SENSITIVE_PATTERNS: match = re.search(f'.{{0,30}}{pattern}.{{0,30}}', res_body, re.IGNORECASE) if match: event_findings.append({ "location": "response body", "type": label, "context": match.group(0) }) if event_findings: findings.append({ "host": host, "method": req.get("method"), "path": path[:100], "findings": event_findings }) return { "requests_scanned": len(events), "requests_with_sensitive_data": len(findings), "risk_level": "HIGH" if len(findings) > 5 else "MEDIUM" if len(findings) > 0 else "LOW", "findings": findings } if __name__ == "__main__": mcp.run() ``` Make it executable: ```bash chmod +x devtools_mcp_enhanced.py ``` *** ## Troubleshooting Workflows ### Workflow 1: Error-First Debugging **Scenario**: Production incident - users reporting failures in the payment service. **Filter**: `res.status >= 400 AND container.name == "payment-service"` **Example session**: ``` You: "What errors are happening in the payment service?" AI: [Calling get_errors_by_container("payment-service")] "Found 5 errors in payment-service over the last 60 seconds: 1. POST /v1/charges → 402 Payment Required Response: {"error": {"code": "card_declined", "message": "Your card has insufficient funds"}} 2. POST /v1/charges → 500 Internal Server Error Response: {"error": {"type": "api_error", "message": "Stripe timeout after 30s"}} 3. GET /v1/customers/cus_xxx → 404 Not Found Response: {"error": {"code": "resource_missing"}} The 500 error suggests a timeout issue with Stripe. Consider: - Checking Stripe's status page - Reviewing timeout configuration - Adding retry logic for transient failures" ``` *** ### Workflow 2: External API Audit **Scenario**: Audit what external APIs your services are calling. **Filter**: `direction == "egress-external"` **Example session**: ``` You: "What external APIs are we calling? Any credential leaks?" AI: [Calling get_external_traffic()] "Found 25 external requests over the last 60 seconds: By host: - api.stripe.com: 8 requests - api.github.com: 7 requests - api.openai.com: 5 requests - api.sendgrid.com: 5 requests Let me scan for sensitive data leaks..." AI: [Calling check_sensitive_data()] "Security scan complete: - 25 requests scanned - 0 sensitive data leaks detected - Risk level: LOW All credentials are properly contained in Authorization headers." ``` *** ### Workflow 3: Container-Scoped Debugging **Scenario**: A specific microservice is behaving unexpectedly. **Filter**: `container.name == "user-service"` **Example session**: ``` You: "What's the user-service container doing?" AI: [Calling get_container_traffic("user-service")] "Traffic from user-service over 60 seconds: Total: 45 requests Hosts: - postgres-primary:5432: 28 requests - redis-cache:6379: 12 requests - api.sendgrid.com: 5 requests Methods: GET (35), POST (10) Status codes: 200 (42), 201 (2), 500 (1) There's 1 error - let me get details..." AI: [Calling get_errors_by_container("user-service")] "1 error found: - POST to api.sendgrid.com/v3/mail/send → 500 Response: Rate limit exceeded The user-service is hitting SendGrid rate limits." ``` *** ### Workflow 4: Process-Scoped Debugging **Scenario**: Need to understand what a specific process is doing. **Filter**: `process.binary == "python3.11"` **Example session**: ``` You: "What HTTP calls is Python making?" AI: [Calling get_process_traffic("python3.11")] "Found 45 requests from python3.11 over 60 seconds: By host: - api.stripe.com: 15 requests - api.openai.com: 12 requests - postgres-primary:5432: 10 requests - redis-cache:6379: 8 requests Methods: GET (28), POST (17) Status codes: 200 (40), 201 (3), 500 (2) There are 2 errors - let me get details..." AI: [Calling get_traffic_filtered("process.binary == \"python3.11\" AND res.status >= 400")] "2 errors from Python: - POST to api.stripe.com/v1/charges → 402 (card declined) - POST to api.openai.com/v1/chat/completions → 500 (rate limit) The OpenAI 500 suggests you're hitting rate limits." ``` *** ## Filter Expression Quick Reference ### Common Filters | Use Case | Filter Expression | | -------------------- | ----------------------------------------- | | All errors | `res.status >= 400` | | Server errors only | `res.status >= 500` | | Client errors only | `res.status >= 400 AND res.status < 500` | | Specific container | `container.name == "my-app"` | | Specific process | `process.binary == "python3.11"` | | POST/PUT/DELETE only | `req.method in ["POST", "PUT", "DELETE"]` | | Specific host | `req.host == "api.stripe.com"` | | External egress | `direction == "egress-external"` | | HTTPS traffic | `dst.port == 443` | | HTTP/2 protocol | `protocol == "http2"` | ### Combining Filters ``` # Errors from a specific container res.status >= 400 AND container.name == "payment-service" # External API errors res.status >= 400 AND direction == "egress-external" # Stripe errors from Python req.host == "api.stripe.com" AND res.status >= 400 AND process.binary == "python3.11" # POST requests to external services req.method == "POST" AND direction == "egress-external" ``` See [Server-Side Event Filtering](/guides/devtools-guides/server-side-event-filtering) for the complete operator and field reference. *** ## Setup Instructions ### Prerequisites * **Qtap v0.14.0+** with DevTools enabled * **Python 3.10+** * **mcp** and **httpx** packages ### Installation ```bash # Install dependencies pip install mcp httpx # Save the enhanced MCP server code # (copy from above to devtools_mcp_enhanced.py) # Make executable chmod +x devtools_mcp_enhanced.py ``` ### Start Qtap with DevTools {% tabs %} {% tab title="Binary" %} ```bash sudo qtap --enable-dev-tools ``` {% endtab %} {% tab title="Docker" %} ```bash docker run -d --name qtap \ --user 0:0 --privileged \ --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \ --pid=host --network=host \ -v /sys:/sys \ -v /var/run/docker.sock:/var/run/docker.sock \ -e TINI_SUBREAPER=1 \ -e ENABLE_DEV_TOOLS=true \ --ulimit=memlock=-1 \ us-docker.pkg.dev/qpoint-edge/public/qtap:v0 ``` {% endtab %} {% endtabs %} ### Connect to AI Assistants {% tabs %} {% tab title="Claude Code" %} ```bash claude mcp add --transport stdio devtools-enhanced -- python3 /path/to/devtools_mcp_enhanced.py ``` Verify: ```bash claude mcp list # Should show: devtools-enhanced: ... - ✓ Connected ``` {% endtab %} {% tab title="Codex CLI" %} ```bash codex mcp add devtools-enhanced -- python3 /path/to/devtools_mcp_enhanced.py ``` Or edit `~/.codex/config.toml`: ```toml [mcp_servers.devtools-enhanced] command = "python3" args = ["/path/to/devtools_mcp_enhanced.py"] ``` {% endtab %} {% tab title="ChatGPT" %} Modify the server to use HTTP transport: ```python if __name__ == "__main__": mcp.run(transport="http", host="0.0.0.0", port=8080) ``` Expose via HTTPS tunnel: ```bash cloudflared tunnel --url http://localhost:8080 ``` Add the tunnel URL to ChatGPT Developer Mode connectors. {% endtab %} {% endtabs %} *** ## Available Tools | Tool | Server-Side Filter | Use Case | | ------------------------------- | ----------------------------------------------- | -------------------------- | | `get_errors()` | `res.status >= 400` | All HTTP errors | | `get_errors_by_container(name)` | `res.status >= 400 AND container.name == "..."` | Container-scoped errors | | `get_container_traffic(name)` | `container.name == "..."` | All traffic from container | | `get_process_traffic(name)` | `process.binary == "..."` | All traffic from process | | `get_external_traffic()` | `direction == "egress-external"` | External API calls | | `get_traffic_filtered(expr)` | Custom | Any filter expression | | `get_traffic_summary()` | None (all traffic) | Quick overview | | `get_hosts()` | None (all traffic) | API dependency mapping | | `check_sensitive_data()` | None (all traffic) | Security audit | *** ## Troubleshooting ### "No events received" with filter 1. **Verify filter syntax**: Test with `"*"` first to confirm events are flowing 2. **Check field names**: Use exact field names (`res.status`, not `status`) 3. **Verify Qtap version**: Server-side filtering requires v0.14.0+ 4. **Generate matching traffic**: Your filter might be too restrictive ### Verify subscription accepted The first event should be `system.connected` with your filter: ```bash curl -sN -X POST http://localhost:10001/devtools/api/events \ -H "Content-Type: application/json" \ -d '{"topics": {"http": "res.status >= 400"}}' | head -3 ``` Expected: ``` event: system.connected data: {"data":{"topics":{"http":"res.status >= 400"}}} ``` ### Check Qtap version ```bash qtap --version # Should be v0.14.0 or later ``` ### Common filter mistakes | Mistake | Fix | | ----------------------- | ------------------------------------------------ | | `status >= 400` | Use full path: `res.status >= 400` | | `method == "POST"` | Use full path: `req.method == "POST"` | | `container == "my-app"` | Use full path: `container.name == "my-app"` | | Unescaped quotes | Escape in JSON: `"container.name == \"my-app\""` | *** ## Next Steps * [Server-Side Event Filtering](/guides/devtools-guides/server-side-event-filtering) - Full filter reference * [DevTools MCP Server](/guides/devtools-guides/devtools-mcp-server) - Original MCP server (client-side filtering) * [DevTools API](/guides/devtools-guides/devtools-api) - Event schemas and API reference * [Traffic Analysis MCP Server](/guides/devtools-guides/traffic-analysis-mcp-server) - Type inference and test generation # QScan Guides Hands-on guides for deploying and using QScan, Qpoint's PII and sensitive data scanning engine. QScan analyzes HTTP traffic captured by Qtap to detect personally identifiable information flowing through your network. ## What You'll Learn These guides cover: * **End-to-end setup** - Standing up the full pipeline from object storage to PII detections * **Configuration** - Tuning monitors, sampling rates, and scan behavior * **Production deployment** - Running QScan in Kubernetes and Cloud Run environments * **Interpreting results** - Understanding PII findings in the Pulse dashboard ## Available Guides ### Getting Started #### [End-to-End Setup: PII Scanning from Scratch](https://github.com/qpoint-io/documentation/blob/main/guides/qscan-guides/end-to-end-setup.md) The "hello world" for QScan. Walk through the complete pipeline from nothing to seeing PII detections in the Pulse dashboard. **What you'll learn:** * Setting up local S3-compatible object storage with MinIO * Configuring Qtap to capture and store HTTP artifacts * Deploying QScan to scan captured traffic for PII * Generating test traffic and verifying detections **Perfect for:** * First-time QScan users * Evaluating PII scanning capabilities * Development and testing environments **Time to complete:** 30 minutes **Skill level:** Beginner-Intermediate *** #### [AWS Setup: PII Scanning on EKS](/guides/qscan-guides/aws-eks-setup) Deploy QScan on Amazon EKS with S3 for artifact storage and Qplane for centralized management. Covers migrating from standalone YAML to Helm-managed Qplane, configuring S3 object storage, deploying QScan, and enabling PII scanning. **What you'll learn:** * Verifying egress access to Qpoint services * Connecting Qtap to Qplane via Helm * Configuring AWS S3 object storage for artifact capture * Deploying QScan and enabling PII scanning in Qplane * Scaling and production considerations (IRSA, GPU, monitoring) **Perfect for:** * Teams running EKS with EC2 node groups * Production AWS deployments with Qplane * Migrating from standalone Qtap to cloud-managed **Time to complete:** 30 minutes **Skill level:** Intermediate *** ## How QScan Works QScan operates as an asynchronous scanning pipeline: 1. **Qtap** captures HTTP request and response bodies during normal traffic flow 2. Captured artifacts are stored in **S3-compatible object storage** 3. **Pulse** schedules scan jobs based on the captured artifacts 4. **QScan** polls for jobs, downloads artifacts, and runs PII detection models 5. Results appear in the **Pulse dashboard** with entity types and confidence scores This architecture means QScan never sits in the request path -- it processes captured data after the fact, with zero impact on application latency. *** ## Next Steps **New to QScan?** Start with the [End-to-End Setup](https://github.com/qpoint-io/documentation/blob/main/guides/qscan-guides/end-to-end-setup.md) guide. **Looking for installation docs?** See the [QScan Installation](/getting-started/qscan/installation) reference for Docker, Kubernetes, and Cloud Run. **Need to configure Qtap for scanning?** The end-to-end guide covers this, or see [Stacks and Plugins](/getting-started/qplane/configuration/stacks-and-plugins) for the qscan plugin reference. # Self-Hosted Setup: PII Scanning with Local Storage Set up a complete QScan pipeline on your local machine -- from object storage to PII detections in the Pulse dashboard. This is the fastest way to see QScan in action. ## Who This Is For **Use this guide if you want to:** * Try QScan for the first time * See the full scanning pipeline working end-to-end * Evaluate PII detection capabilities before a production rollout **Choose something else if you:** * Already have Qtap and object storage running -- see the [QScan Installation](/getting-started/qscan/installation) docs * Want production Kubernetes deployment -- see [Kubernetes Installation](/getting-started/qscan/installation/kubernetes) **Time to complete:** 30 minutes *** ## Prerequisites Before you begin, make sure you have: * **Docker and Docker Compose** installed and running * A **registration token** from [app.qpoint.io](https://app.qpoint.io) (Settings -> API Tokens) * Basic familiarity with **YAML** configuration {% hint style="info" %} The registration token connects Qtap and QScan to the Pulse service, which coordinates scan jobs and displays results. You can create one from the Settings page in the Qpoint dashboard. {% endhint %} *** ## Step 1: Set Up Object Storage QScan reads captured HTTP artifacts from S3-compatible object storage. For this guide, you will use MinIO as a lightweight local store. Create a project directory and add a `docker-compose.yml`: ```bash mkdir qscan-quickstart && cd qscan-quickstart ``` Add the MinIO service to `docker-compose.yml`: {% code title="docker-compose.yml" %} ```yaml services: minio: image: minio/minio:latest container_name: minio command: server /data --console-address ":9001" ports: - "9000:9000" - "9001:9001" environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin volumes: - minio-data:/data volumes: minio-data: ``` {% endcode %} Start MinIO: ```bash docker compose up -d minio ``` Create the bucket that Qtap will write artifacts to: ```bash docker run --rm --network host \ --entrypoint sh minio/mc -c " mc alias set local http://localhost:9000 minioadmin minioadmin && mc mb local/qpoint --ignore-existing " ``` You now have an S3-compatible store running at `http://localhost:9000` with a `qpoint` bucket. *** ## Step 2: Configure Qtap for Artifact Capture Create a Qtap configuration that captures HTTP traffic and stores artifacts in MinIO with the QScan plugin enabled. {% code title="qtap.yaml" %} ```yaml version: 2 services: object_stores: - id: minio type: s3 endpoint: localhost:9000 bucket: qpoint region: us-east-1 insecure: true access_key: type: env value: AWS_ACCESS_KEY_ID secret_key: type: env value: AWS_SECRET_ACCESS_KEY stacks: scan_stack: plugins: - type: http_capture config: level: full format: json - type: qscan config: record_document: true sample_baseline: 10 sample_rate: 1.0 monitors: - type: PERSON record_value: false - type: EMAIL_ADDRESS record_value: false - type: PHONE_NUMBER record_value: false - type: CREDIT_CARD record_value: false - type: US_SSN record_value: false tap: direction: egress http: stack: scan_stack ``` {% endcode %} Key configuration choices: * **`sample_rate: 1.0`** scans every captured request. In production, lower this to reduce cost and load. * **`sample_baseline: 10`** ensures the first 10 requests to each endpoint are always scanned, regardless of sample rate. * **`record_value: false`** on each monitor means QScan reports that PII was found, but does not store the actual sensitive values. {% hint style="info" %} To use Qpoint's hosted QScan cloud service instead of self-hosted, set `qscan_cloud: true` in the plugin config. When enabled, Qtap sends artifacts to Qpoint's cloud for scanning -- no QScan deployment needed on your side. {% endhint %} *** ## Step 3: Deploy Qtap Add the Qtap service to your `docker-compose.yml`: {% code title="docker-compose.yml (add to services)" %} ```yaml qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap restart: unless-stopped privileged: true pid: host network_mode: host command: >- --log-level=info --log-encoding=console --config=/app/config/qpoint.yaml volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qpoint.yaml:ro environment: - REGISTRATION_TOKEN=${REGISTRATION_TOKEN} - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - TINI_SUBREAPER=1 ulimits: memlock: soft: -1 hard: -1 ``` {% endcode %} Start Qtap: ```bash docker compose up -d qtap ``` Check that it starts cleanly: ```bash docker logs qtap --tail 20 ``` You should see log lines indicating that Qtap has loaded the configuration and is tapping traffic. *** ## Step 4: Deploy QScan Add the QScan service to your `docker-compose.yml`: {% code title="docker-compose.yml (add to services)" %} ```yaml qscan: image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest container_name: qscan restart: unless-stopped depends_on: - minio deploy: resources: limits: cpus: '2.0' memory: 12G reservations: cpus: '0.5' memory: 256M environment: - REGISTRATION_TOKEN=${REGISTRATION_TOKEN} - S3_ENDPOINT_URL=http://minio:9000 - S3_BUCKET_NAME=qpoint - S3_REGION_NAME=us-east-1 - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - METRICS_PORT=8080 - NUM_POLLERS=1 - NUM_SCANNERS=1 - LOG_LEVEL=info ``` {% endcode %} {% hint style="warning" %} QScan loads PII detection models into memory. The 12 GB memory limit is recommended for reliable operation. Machines with less available memory may experience out-of-memory errors during scanning. {% endhint %} Start QScan: ```bash docker compose up -d qscan ``` Verify it connects and begins polling for scan jobs: ```bash docker logs qscan --tail 20 ``` Look for log messages indicating that QScan has connected to Pulse and is polling for work. *** ## Step 5: Generate Test Traffic With the pipeline running, generate some HTTP requests containing fake PII. Qtap will capture these, store the artifacts in MinIO, and Pulse will schedule scan jobs for QScan. Run a few curl commands from a container on the host (so Qtap can observe the traffic): ```bash # Request with fake PII in the body curl -X POST https://httpbin.org/post \ -H "Content-Type: application/json" \ -d '{ "name": "Jane Smith", "email": "jane.smith@example.com", "phone": "555-867-5309", "ssn": "123-45-6789", "card": "4111-1111-1111-1111" }' # A few more to build up data curl -X POST https://httpbin.org/post \ -H "Content-Type: application/json" \ -d '{"customer": "John Doe", "contact": "john@example.org"}' curl -X POST https://httpbin.org/post \ -H "Content-Type: application/json" \ -d '{"patient": "Alice Johnson", "phone": "415-555-0142", "ssn": "987-65-4321"}' ``` {% hint style="info" %} httpbin.org echoes your request body back in the response, so both the request and response will contain PII for QScan to detect. {% endhint %} Wait a minute or two for the pipeline to process: 1. Qtap captures the requests and writes artifacts to MinIO 2. Pulse receives metadata and schedules scan jobs 3. QScan picks up the jobs, downloads artifacts, and runs PII detection *** ## Step 6: Verify PII Detections ### Check QScan Logs ```bash docker logs qscan --tail 50 ``` Look for log entries showing scan activity -- messages about downloading artifacts, running detection, and reporting results. ### Check the Pulse Dashboard Open [app.qpoint.io](https://app.qpoint.io) and navigate to your environment. You should see: * **PII findings** associated with the `httpbin.org` endpoint * **Detected entity types** such as PERSON, EMAIL\_ADDRESS, PHONE\_NUMBER, US\_SSN, and CREDIT\_CARD * **Confidence scores** for each detection The dashboard shows which endpoints are transmitting sensitive data, what types of PII were found, and how frequently it appears. ### Check MinIO (Optional) You can browse the stored artifacts through the MinIO console at (login with `minioadmin` / `minioadmin`). Look in the `qpoint` bucket for captured request and response data. *** ## Step 7: Next Steps You now have a working QScan pipeline. Here are paths forward: * **Tune sampling rates** -- Lower `sample_rate` for production traffic and rely on `sample_baseline` to ensure coverage of new endpoints. See [QScan Configuration](https://github.com/qpoint-io/documentation/blob/main/getting-started/qscan/configuration/README.md). * **Deploy to production** -- Run QScan on [Kubernetes](/getting-started/qscan/installation/kubernetes) or [Cloud Run](/getting-started/qscan/installation/cloud-run) for production workloads. * **Add more monitors** -- QScan supports additional entity types like LOCATION, STREET\_ADDRESS, US\_BANK\_NUMBER, and US\_DRIVER\_LICENSE. * **Enable GPU acceleration** -- For higher throughput scanning, QScan can use GPU resources to accelerate PII detection models. * **Target specific endpoints** -- Use Qtap's endpoint configuration to apply the `qscan` plugin only to specific domains (e.g., AI providers, external APIs). *** ## Complete Docker Compose File For convenience, here is the full `docker-compose.yml` with all three services ready to run: {% code title="docker-compose.yml" %} ```yaml services: minio: image: minio/minio:latest container_name: minio command: server /data --console-address ":9001" ports: - "9000:9000" - "9001:9001" environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin volumes: - minio-data:/data qtap: image: us-docker.pkg.dev/qpoint-edge/public/qtap:v0 container_name: qtap restart: unless-stopped privileged: true pid: host network_mode: host depends_on: - minio command: >- --log-level=info --log-encoding=console --config=/app/config/qpoint.yaml volumes: - /sys:/sys - /var/run/docker.sock:/var/run/docker.sock - ./qtap.yaml:/app/config/qpoint.yaml:ro environment: - REGISTRATION_TOKEN=${REGISTRATION_TOKEN} - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - TINI_SUBREAPER=1 ulimits: memlock: soft: -1 hard: -1 qscan: image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest container_name: qscan restart: unless-stopped depends_on: - minio deploy: resources: limits: cpus: '2.0' memory: 12G reservations: cpus: '0.5' memory: 256M environment: - REGISTRATION_TOKEN=${REGISTRATION_TOKEN} - S3_ENDPOINT_URL=http://minio:9000 - S3_BUCKET_NAME=qpoint - S3_REGION_NAME=us-east-1 - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - METRICS_PORT=8080 - NUM_POLLERS=1 - NUM_SCANNERS=1 - LOG_LEVEL=info volumes: minio-data: ``` {% endcode %} To run the full stack: ```bash # Set your registration token export REGISTRATION_TOKEN=your-token-from-app-qpoint-io # Create the bucket and start everything docker compose up -d minio docker run --rm --network host \ --entrypoint sh minio/mc -c " mc alias set local http://localhost:9000 minioadmin minioadmin && mc mb local/qpoint --ignore-existing " docker compose up -d ``` Then generate test traffic with the curl commands from Step 5 and check results in the [Pulse dashboard](https://app.qpoint.io). # AWS Setup: PII Scanning on EKS Add PII scanning to your EKS cluster using Qtap, Qplane, AWS S3, and QScan. By the end of this guide, QScan will be scanning HTTP traffic captured by Qtap and reporting PII findings to the Pulse dashboard. ## Who This Is For **Use this guide if you:** * Have an EKS cluster (new or existing) where you want PII scanning * Want to use Qplane for centralized management * Are storing artifacts in AWS S3 **Choose something else if you:** * Want a local development setup -- see [Self-Hosted Setup](/guides/qscan-guides/self-hosted-setup) * Need GCP Cloud Run deployment -- see [Cloud Run Installation](/getting-started/qscan/installation/cloud-run) **Time to complete:** 30 minutes *** ## Prerequisites * EKS cluster with EC2 node groups (at least one node with 12 GB+ available memory for QScan) * `kubectl` and `helm` configured for the cluster * `aws` CLI configured * A Qpoint account at [app.qpoint.io](https://app.qpoint.io) *** ## Set Your Variables Replace these values throughout the guide: ```bash export AWS_REGION="us-east-1" export S3_BUCKET="your-org-qpoint-artifacts" ``` All commands below use these variables. Substitute your actual values before running. {% hint style="info" %} Shell variables work in CLI commands but not in YAML or JSON files. Before applying manifests, either replace the placeholder values manually or pipe through `envsubst`: ```bash envsubst < qscan-deployment.yaml | kubectl apply -f - ``` {% endhint %} ### Create the Namespace ```bash kubectl create namespace qpoint ``` *** ## Step 1: Verify Outbound Access Confirm the cluster can reach all required endpoints: | Endpoint | Used By | Purpose | | ------------------------------- | ------------ | ---------------------------------------- | | `api.qpoint.io:443` | Qtap | Control plane API (registration, config) | | `pulse.qpoint.io:443` | Qtap | Event and telemetry ingestion | | `api-pulse.qpoint.io:443` | QScan | Scan job polling and result reporting | | `s3..amazonaws.com:443` | Qtap + QScan | Object storage | Run a quick egress test: {% code overflow="wrap" %} ```bash kubectl run --rm -it egress-test --image=curlimages/curl -- sh -c \ "curl -s https://api.qpoint.io/health && \ curl -s https://pulse.qpoint.io/health && \ curl -s https://api-pulse.qpoint.io/health && \ curl -s -o /dev/null -w '%{http_code}' https://s3.${AWS_REGION}.amazonaws.com" ``` {% endcode %} {% hint style="warning" %} If egress is restricted, all four endpoints must be allowlisted before proceeding. {% endhint %} *** ## Step 2: Create AWS S3 Bucket and IAM Credentials Set up object storage before connecting any agents to Qplane. This ensures captured payloads go directly to your S3 bucket and never to Qpoint Cloud storage. ### Create the S3 Bucket ```bash aws s3 mb s3://${S3_BUCKET} --region ${AWS_REGION} ``` {% hint style="info" %} S3 bucket names are globally unique across all AWS accounts. Choose a name with your organization's prefix (e.g., `acme-qpoint-artifacts`) to avoid conflicts. {% endhint %} ### Harden the Bucket Block all public access: ```bash aws s3api put-public-access-block \ --bucket ${S3_BUCKET} \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true ``` Enable default encryption (SSE-S3): ```bash aws s3api put-bucket-encryption \ --bucket ${S3_BUCKET} \ --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}' ``` Enforce TLS-only access: ```bash aws s3api put-bucket-policy --bucket ${S3_BUCKET} --policy '{ "Version": "2012-10-17", "Statement": [ { "Sid": "DenyInsecureTransport", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws:s3:::'"${S3_BUCKET}"'", "arn:aws:s3:::'"${S3_BUCKET}"'/*" ], "Condition": { "Bool": { "aws:SecureTransport": "false" } } } ] }' ``` {% hint style="info" %} For stricter encryption requirements, use SSE-KMS instead of SSE-S3. {% endhint %} ### Create IAM Credentials Create an IAM user with the minimum required permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::your-org-qpoint-artifacts" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::your-org-qpoint-artifacts/*" } ] } ``` Includes `DeleteObject` for lifecycle policy cleanup. Remove if your retention policy handles cleanup externally. {% hint style="info" %} **Key rotation guidance:** Since Qtap requires static IAM keys, rotate them regularly. The process: create a new access key → update the `qpoint-s3-creds` Kubernetes secret → restart Qtap and QScan pods → verify they reconnect → delete the old access key. {% endhint %} ### Store Credentials in Kubernetes Create the secret now. Both Qtap and QScan will reference it in later steps. ```bash kubectl create secret generic qpoint-s3-creds \ --from-literal=AWS_ACCESS_KEY_ID='' \ --from-literal=AWS_SECRET_ACCESS_KEY='' \ -n qpoint ``` *** ## Step 3: Configure Qplane Configure your Qplane environment with object storage **before** deploying Qtap agents. This prevents any payloads from being sent to Qpoint Cloud storage. {% hint style="warning" %} New Qplane accounts include a default "Basic Reporting and Error Detection" stack that begins capturing error payloads immediately when an agent connects. If object storage is not configured first, those payloads will be sent to Qpoint Cloud. See [Object Storage Configuration](/getting-started/qplane/configuration/object-storage) for details. {% endhint %} ### Create an Environment 1. Go to [app.qpoint.io](https://app.qpoint.io) 2. Navigate to Settings -> Deploy -> Environments 3. Create a new environment and copy the registration token ### Add Object Storage 1. Navigate to Settings -> Deploy -> Services -> Object Stores 2. Click **"+ Add Object Store"** and configure: | Field | Value | | ---------- | ------------------------------------------------------------------------- | | Endpoint | `s3.us-east-1.amazonaws.com` | | Bucket | `your-org-qpoint-artifacts` | | Region | Your AWS region (e.g., `us-east-1`) | | Access URL | `https://your-org-qpoint-artifacts.s3.us-east-1.amazonaws.com/{{DIGEST}}` | {% hint style="info" %} Use the regionalized S3 endpoint (`s3..amazonaws.com`) to avoid redirect issues with buckets outside `us-east-1`. {% endhint %} ### Verify in Snapshot YAML Go to Settings -> Deploy -> Snapshot and confirm your object store appears in the `services.object_stores` section with the correct endpoint, bucket, and region. *** ## Step 4: Deploy Qtap Now that Qplane has object storage configured, deploy Qtap. Choose the option that matches your deployment. ### Option A: Deploy with Helm (recommended) ```bash # Store the registration token (key must be named "token" for the Helm chart) kubectl create secret generic qtap-registration \ --from-literal=token='' \ -n qpoint # Add the Qpoint Helm repo helm repo add qpoint https://helm.qpoint.io helm repo update # Deploy Qtap with S3 credentials helm install qtap qpoint/qtap \ -n qpoint --create-namespace \ --set registrationTokenSecretRefName="qtap-registration" \ --set extraEnv[0].name="AWS_ACCESS_KEY_ID" \ --set extraEnv[0].valueFrom.secretKeyRef.name="qpoint-s3-creds" \ --set extraEnv[0].valueFrom.secretKeyRef.key="AWS_ACCESS_KEY_ID" \ --set extraEnv[1].name="AWS_SECRET_ACCESS_KEY" \ --set extraEnv[1].valueFrom.secretKeyRef.name="qpoint-s3-creds" \ --set extraEnv[1].valueFrom.secretKeyRef.key="AWS_SECRET_ACCESS_KEY" ``` ### Option B: Add Qplane to an existing manifest If you manage Qtap through your own Kubernetes manifests, add the registration token and S3 credentials to your existing deployment. Qtap reads the `REGISTRATION_TOKEN` environment variable directly: ```bash kubectl create secret generic qtap-registration \ --from-literal=REGISTRATION_TOKEN='' \ -n qpoint ``` Add `envFrom` to your existing Qtap container spec to inject both the registration token and S3 credentials: ```yaml envFrom: - secretRef: name: qtap-registration - secretRef: name: qpoint-s3-creds ``` Remove any local configuration (config file mounts, `--config` args) -- Qplane pushes configuration to the agent once it connects. Apply the updated manifest. {% hint style="info" %} If your cluster has restricted namespaces or security policies, you may need to adjust the namespace or add appropriate pod security labels. The Qtap pod requires privileged access for eBPF operations. {% endhint %} ### Verify 1. Confirm the agent appears in the Qplane dashboard 2. Generate some HTTP traffic through a Qtap-monitored service 3. Check that objects appear in your S3 bucket: ```bash aws s3 ls s3://${S3_BUCKET}/ --recursive | head ``` *** ## Step 5: Deploy QScan QScan is a separate container that runs PII detection models. It polls Pulse for scan jobs, pulls artifacts from S3, scans them, and reports findings back. Payload artifacts and scan processing stay in your AWS account. Only anonymized metadata and authentication flow to Qpoint Cloud over TLS. ### QScan Credentials QScan needs AWS credentials for S3 access and a registration token for Pulse. Create a secret for the registration token: ```bash kubectl create secret generic qscan-registration \ --from-literal=REGISTRATION_TOKEN='' \ -n qpoint ``` QScan uses the `qpoint-s3-creds` secret created in Step 2 for S3 access. {% hint style="info" %} Both Qtap and QScan require explicit static credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`). Neither supports IRSA or ambient credential chains. {% endhint %} {% code title="qscan-deployment.yaml" %} ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: qscan namespace: qpoint labels: app: qscan spec: replicas: 1 selector: matchLabels: app: qscan template: metadata: labels: app: qscan spec: containers: - name: qscan image: us-docker.pkg.dev/qpoint-edge/public/qscan:latest env: - name: REGISTRATION_TOKEN valueFrom: secretKeyRef: name: qscan-registration key: REGISTRATION_TOKEN - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: qpoint-s3-creds key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: qpoint-s3-creds key: AWS_SECRET_ACCESS_KEY - name: S3_ENDPOINT_URL value: "https://s3.us-east-1.amazonaws.com" # Replace with your region - name: S3_BUCKET_NAME value: "your-org-qpoint-artifacts" # Replace with your bucket - name: S3_REGION_NAME value: "us-east-1" # Replace with your region - name: NUM_POLLERS value: "1" - name: NUM_SCANNERS value: "1" - name: LOG_LEVEL value: "info" - name: METRICS_PORT value: "8080" ports: - containerPort: 8080 name: metrics resources: requests: cpu: "2" memory: "12Gi" limits: cpu: "6" memory: "24Gi" livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 10 startupProbe: httpGet: path: / port: 8080 initialDelaySeconds: 5 periodSeconds: 30 failureThreshold: 10 ``` {% endcode %} ### Apply and Verify ```bash kubectl apply -f qscan-deployment.yaml ``` {% hint style="warning" %} QScan loads approximately 8.4 GB of ML models at startup. 12 GB memory is the absolute minimum; 24 GB is recommended. Initial model loading takes 1-2 minutes -- the startup probe accounts for this. {% endhint %} {% hint style="info" %} `S3_ENDPOINT_URL` must be set explicitly for customer S3 buckets. Without it, QScan defaults to an internal Qpoint endpoint that will not work with your AWS credentials. {% endhint %} ```bash kubectl get pods -n qpoint -l app=qscan kubectl logs -n qpoint deployment/qscan --tail 30 ``` Look for log messages indicating that QScan has loaded models and is polling Pulse for scan jobs. *** ## Step 6: Enable PII Scanning in Qplane Add the QScan plugin to your stack in the Qplane UI (Stacks and Plugins section). ### Recommended Settings | Setting | Recommended | Description | | ----------------- | ----------- | ---------------------------------------------------- | | `sample_baseline` | `10` | Always scan first 10 requests per endpoint | | `sample_rate` | `0.1` | Then scan 10% of subsequent requests | | `cache_ttl` | `24h` | Cache results for 24 hours to reduce duplicate scans | | `record_document` | `false` | Do not store full document content with findings | ### Monitor Types Enable the PII types you want to detect: `PERSON`, `EMAIL_ADDRESS`, `PHONE_NUMBER`, `US_SSN`, `CREDIT_CARD`, `STREET_ADDRESS`, `US_BANK_NUMBER`, `US_DRIVER_LICENSE`, `LOCATION`, `ORGANIZATION` Set `record_value: false` on all monitors to report detections without storing the actual sensitive values. {% hint style="info" %} For initial testing, set `sample_rate: 1.0` to scan every request. Once you've confirmed detections are working, lower to `0.1` for production traffic. {% endhint %} *** ## Step 7: Verify PII Detections Once the plugin is active, QScan begins processing captured HTTP traffic: 1. **Qtap** captures HTTP payloads and stores artifacts in S3 2. **Pulse** schedules scan jobs for new artifacts 3. **QScan** polls for jobs, downloads artifacts, scans with ML models, and reports findings 4. **Findings** surface in the Qplane dashboard Check QScan logs to confirm scanning activity: ```bash kubectl logs -n qpoint deployment/qscan --tail 50 ``` Open [app.qpoint.io](https://app.qpoint.io) and check the dashboards for PII findings associated with your endpoints. *** ## Scaling If the scan queue backs up, scale QScan horizontally: ```bash kubectl scale deployment qscan -n qpoint --replicas=3 ``` Each replica polls Pulse independently. You can also increase `NUM_POLLERS` and `NUM_SCANNERS` per replica for more concurrency within a single pod, though this requires proportionally more memory. *** ## Production Considerations | Consideration | Detail | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Capture level | QScan requires full HTTP capture (request and response bodies). Summary-only capture will not produce artifacts for scanning. | | S3 must match | Qtap and QScan must point to the same S3 bucket and endpoint. | | Data sovereignty | Payload artifacts and scan processing stay in your AWS account. Only anonymized metadata and authentication flow to Qpoint Cloud over TLS. | | S3 credentials | Both Qtap and QScan require explicit `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. Neither supports IRSA or ambient credential chains. Rotate keys regularly: create new key → update Kubernetes secret → restart pods → delete old key. | | S3 endpoint | Always set `S3_ENDPOINT_URL` explicitly (e.g., `https://s3.us-east-1.amazonaws.com`). The default endpoint is an internal Qpoint service, not AWS S3. | | QScan resources | Plan node capacity for 12 GB+ memory per QScan pod. Use dedicated node groups if needed. Resource values in this guide are starting points -- tune based on your workload. | | Monitoring | QScan exposes Prometheus metrics on the configured metrics port. Use `kubectl port-forward` to verify: `kubectl port-forward -n qpoint deployment/qscan 8080:8080` then `curl localhost:8080/metrics`. | *** ## Order of Operations ``` Egress -> S3 Bucket -> Configure -> Deploy -> Deploy -> Enable -> Scanning Test + IAM Qplane Qtap QScan Plugin Active ``` 1. Verify egress to Qpoint and S3 endpoints 2. Create S3 bucket and IAM credentials 3. Configure Qplane: environment, object store, verify in Snapshot 4. Deploy Qtap via Helm or manifest (S3 is already configured, so no payloads leak to Qpoint Cloud) 5. Deploy QScan (12 GB+ RAM, ML models load at startup) 6. Add QScan plugin in Qplane (monitors, sampling) 7. PII findings appear in Qplane dashboards --- [Next Page](/llms-full.txt/1)