Traffic Processing with Plugins

Understanding Stacks and Plugins

In Qtap's configuration, traffic processing is organized using two key concepts:

• Stacks: Named collections of plugins that work together to process traffic

• Plugins: Individual components that perform specific functions on captured traffic

This structure allows you to create different processing configurations for different types of traffic.

New to plugins? Follow the Complete Guide for hands-on examples progressing through all 4 levels of plugin configuration (from basic to production-ready).

Want centralized management? Qplane provides visual configuration for stacks and plugins with automatic propagation to all agents. See the POC Kick Off Guide to get started.

Stack Configuration

Stacks are defined in the stacks section of your qpoint.yaml file. Each stack has a unique name and contains one or more plugins:

stacks:
  default_stack:   # Stack name
    plugins:       # List of plugins in this stack
      - type: http_capture
        config:
          # Plugin-specific configuration

You can create multiple stacks for different purposes, each with its own set of plugins and configurations.

Available Plugins

Qtap includes several plugins that provide different processing capabilities. Rules use Rulekit - Qpoint's flexible expression-based rules engine for evaluating conditions against HTTP traffic.

HTTP Capture Plugin

The http_capture plugin provides comprehensive HTTP traffic capture with flexible logging levels and the ability to upload payloads to object storage.

Basic Configuration

- type: http_capture
  config:
    level: summary        # Default capture level (none|summary|headers|full)
    format: text          # Output format (text|json)

Level Options

• none: No capture (effectively disables the plugin)

• summary: Basic information (method, path, status code)

• headers: Includes headers

• full: Complete information including request/response bodies

Example with Rules

- type: http_capture
  config:
    level: summary        # Default level for most traffic
    format: text          # Human-readable format
    rules:
      - name: "Full capture for httpbin.org"
        expr: http.req.host == "httpbin.org"
        level: full

      - name: "Headers logging for server errors"
        expr: http.res.status >= 500
        level: headers

      - name: "Full capture for client errors"
        expr: http.res.status >= 400 && http.res.status < 500
        level: full

This configuration:

1. Uses summary level by default

2. Captures full details for all traffic to httpbin.org

3. Captures detailed information for server errors (5xx)

4. Captures full information for client errors (4xx)

Container and Pod-based Filtering

You can also create rules based on container or Kubernetes pod attributes:

- type: http_capture
  config:
    level: summary
    format: text
    rules:
      - name: "Debug specific container"
        expr: src.container.name == "my-app-container"
        level: full
      
      - name: "Debug by container label"
        expr: src.container.labels.qpoint_debug_level == "debug"
        level: full
      
      - name: "Debug specific pod"
        expr: src.pod.name == "frontend-deployment-abc123"
        level: full
      
      - name: "Debug by pod label"
        expr: src.pod.labels.app == "frontend" && src.pod.labels.debug == "true"
        level: full

The http_capture plugin uploads captured data to the configured object store at all capture levels. At summary level, the stored object contains metadata only (method, URL, status, duration). At headers level, request and response headers are included. At full level, both headers and bodies are captured and stored.

Access Logs Plugin

The access_logs plugin provides formatted logging of HTTP traffic to stdout. It does not upload to the object store.

Basic Configuration

- type: access_logs
  config:
    mode: details        # Default logging level (summary|details|full)
    format: console      # Output format (console|json)

Mode Options

• summary: Basic information (method, path, status code)

• details: Includes headers and timing information

• full: Complete information including request/response bodies

Example with Rules

- type: access_logs
  config:
    mode: summary        # Default mode for most traffic
    format: console      # Human-readable format
    rules:
      - name: "Detailed API Logging"
        expr: http.req.host == "api.example.com"
        mode: details

      - name: "Full Error Logging"
        expr: http.res.status >= 400
        mode: full

HTTP Metrics Plugin

The http_metrics plugin exposes Prometheus-style metrics for HTTP traffic. It tracks request and response counts, durations, and payload sizes — useful for dashboards, alerting, and capacity planning.

Configuration

- type: http_metrics

No additional configuration is needed. Once included in a stack, the plugin automatically records the following metrics:

Exported Metrics

Metric	Type	Description
requests_total	Counter	Total number of requests
requests_duration_ms	Histogram	Request duration in milliseconds
requests_size_bytes	Histogram	Request payload size in bytes
responses_total	Counter	Total number of responses
responses_duration_ms	Histogram	Response duration in milliseconds
responses_size_bytes	Histogram	Response payload size in bytes
duration_ms	Histogram	Combined request + response duration

All counter and histogram metrics include labels for method, host, status_code, and protocol.

Example

stacks:
  monitored:
    plugins:
      - type: http_metrics
      - type: access_logs
        config:
          mode: summary
          format: console

This gives you both Prometheus metrics and console logging in the same stack.

Report Usage Plugin

The report_usage plugin sends anonymized usage metrics to Pulse. This is mainly useful when using Qplane. It works alongside other plugins and doesn't affect traffic capture.

- type: report_usage

You can also use report_usage to promote specific HTTP headers into your OTel event store as custom attributes. Add a headers list to extract headers from observed traffic:

- type: report_usage
  config:
    headers:
      - Cf-Access-Authenticated-User-Email
      - X-Forwarded-For
      - X-Request-Id

Promoted headers appear as customHeaders.req.<name> and customHeaders.res.<name> attributes in your OTel backend. See the full guide: Exporting HTTP Headers to OpenTelemetry.

This plugin is optional and can be included in any stack.

Error Detection Plugin

The detect_errors plugin captures detailed information when HTTP responses meet specific error criteria. It provides granular control over what gets recorded — you can independently toggle request headers, request body, response headers, and response body per rule.

This is especially useful when you need to capture request payloads without storing potentially massive response bodies (e.g., API calls that return large datasets).

Basic Configuration

- type: detect_errors
  config:
    rules:
      - name: "Server Errors"
        trigger_status_codes:
          - '5xx'
        report_as_issue: true
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: true

Rule Options

Field	Type	Description
name	string	Rule name for identification
trigger_status_codes	list	Status code patterns to match (e.g., '5xx', '404', '4xx')
trigger_empty_body	bool	Trigger when response body is empty
trigger_duration	duration	Trigger when request exceeds this duration (e.g., '5s', '500ms')
trigger_contains	string	Trigger when response body contains this string
only_categories	list	Only match specific MIME categories
only_urls	list	Only match specific URL patterns
exclude_urls	list	Exclude specific URL patterns
with_tags	list	Only match connections with specific tags
report_as_issue	bool	Flag matched requests as issues in the event store
record_req_headers	bool	Record request headers
record_req_body	bool	Record request body
record_res_headers	bool	Record response headers
record_res_body	bool	Record response body

Example: Capture requests but skip large response bodies

- type: detect_errors
  config:
    rules:
      - name: "API request capture"
        trigger_status_codes:
          - '2xx'
          - '4xx'
          - '5xx'
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: false    # Skip potentially massive responses

Example: Slow query detection with full capture

- type: detect_errors
  config:
    rules:
      - name: "Slow requests"
        trigger_duration: '5s'
        report_as_issue: true
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: true
      
      - name: "Errors without response body"
        trigger_status_codes:
          - '5xx'
        report_as_issue: true
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: false

Debug Plugin (Deprecated)

DEPRECATED: The debug plugin is deprecated and will be removed in a future version.

Migration: Use access_logs plugin for console output, or http_capture for more flexible capture.

Why deprecated: The access_logs plugin provides better formatting and more features, making this plugin redundant.

The debug plugin provides basic logging of HTTP traffic.

Example of OLD (deprecated) approach:

- type: debug
  config:
    mode: summary  # or "details" for more information

NEW approach using access_logs:

- type: access_logs
  config:
    mode: details       # summary, details, or full
    format: console     # Human-readable console output

Migration benefits:

• Better formatted output (Apache-style logs)

• Support for both console and JSON formats

• Rule-based selective logging

• Consistent with other plugins

Rule Expressions with Rulekit

Both the http_capture and access_logs plugins use Rulekit for rule evaluation. Rulekit is Qpoint's expression-based rules engine that evaluates conditions against key-value data from HTTP traffic.

Expression Syntax

Rule expressions follow a straightforward pattern:

<field> <operator> <value>

Multiple conditions can be combined using logical operators:

<field1> <operator1> <value1> and <field2> <operator2> <value2>

Available Fields for Rule Expressions

Request Fields

Field	Example Usage	Description
http.req.method	http.req.method == "POST"	HTTP method (GET, POST, etc.)
http.req.path	http.req.path contains "/api/"	Request path
http.req.host	http.req.host == "api.example.com"	Host header value
http.req.url	http.req.url contains "search"	Full URL
http.req.headers.<name>	http.req.headers.content-type == "application/json"	Request header by name

Response Fields

Field	Example Usage	Description
http.res.status	http.res.status >= 400	HTTP status code
http.res.headers.<name>	http.res.headers.content-type contains "json"	Response header by name

Source Context Fields

Field	Example Usage	Description
src.container.name	src.container.name == "my-app"	Container name
src.container.labels.<key>	src.container.labels.app == "frontend"	Container label value
src.pod.name	src.pod.name == "frontend-abc123"	Kubernetes pod name
src.pod.labels.<key>	src.pod.labels.version == "v2"	Pod label value

Operators for Rule Expressions

Rulekit supports various operators for building expressions:

Comparison Operators

Operator	Aliases	Description	Example
==	eq	Equal to	http.req.method == "GET"
!=	ne	Not equal to	http.req.host != "internal.example.com"
>	gt	Greater than	http.res.status > 200
>=	ge	Greater than or equal to	http.res.status >= 400
<	lt	Less than	http.res.status < 300
<=	le	Less than or equal to	http.res.status <= 399
=~	matches	Matches regex pattern	http.req.path matches /^\/api\/v\d+\//
contains		Contains substring	http.req.url contains "search"
in		Is contained in array	http.req.method in ["GET", "HEAD"]

Logical Operators

Operator	Aliases	Description	Example
and	&&	Logical AND	http.req.method == "POST" and http.res.status >= 400
or	||	Logical OR	http.req.path contains "/admin" or http.req.path contains "/auth"
not	!	Logical NOT	not http.req.host == "public.example.com"

Value Types

Rulekit expressions support various value types:

• Boolean: true, false

• Number: Integer or floating-point values (e.g., 200, 1.5)

• String: Text enclosed in double quotes (e.g., "example.com")

• Regex Pattern: Patterns enclosed in slashes /pattern/ or vertical bars |pattern|

• Array: Values in square brackets (e.g., [200, 201, 204])

Using Rulekit Macros

You can define reusable expression macros in the rulekit section of your configuration:

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400 && http.res.status < 600
    - name: is_api_request
      expr: http.req.path matches /^\/api\//
    - name: is_production
      expr: src.pod.labels.env == "production" || src.container.labels.env == "production"

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: text
          rules:
            - name: "Capture production API errors"
              expr: is_production() && is_api_request() && is_error()
              level: full

            - name: "Debug all errors"
              expr: is_error()
              level: headers

Known Limitation: The not operator does not currently work in rule expressions or macros. Use explicit negation with != and && operators instead.

For example, instead of:

- name: is_not_production
  expr: not (src.pod.labels.env == "production")

Use:

- name: is_not_production
  expr: src.pod.labels.env != "production"

Complete Configuration Example

Here's a comprehensive example that demonstrates various plugin features:

version: 2

# Define reusable macros
rulekit:
  macros:
    - name: is_4xx
      expr: http.res.status >= 400 && http.res.status < 500
    - name: is_5xx
      expr: http.res.status >= 500 && http.res.status < 600
    - name: is_debug_enabled
      expr: src.container.labels.debug == "true" || src.pod.labels.debug == "true"

services:
  event_stores:
    - id: console_stdout
      type: stdout
  object_stores:
    - id: minio
      type: s3
      endpoint: minio.internal:9000
      bucket: qpoint-objects
      region: us-east-1
      access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

stacks:
  production_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: text
          rules:
            # Debug containers with debug label
            - name: "Debug mode capture"
              expr: is_debug_enabled()
              level: full
            
            # Capture errors from specific service
            - name: "Payment service errors"
              expr: http.req.host == "payment.api.com" && (is_4xx() || is_5xx())
              level: full

            # Monitor external API calls
            - name: "External API monitoring"
              expr: http.req.host matches /\.(googleapis\.com|amazonaws\.com)$/ && http.req.method == "POST"
              level: headers

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: true
  http:
    stack: production_stack

Additional Resources

• Rulekit Documentation: For detailed information about rule expressions, operators, and advanced features, visit the Rulekit GitHub repository

• Rulekit Examples: The repository includes an interactive CLI demo tool for testing rule expressions