Traffic Processing with Plugins

Understanding Stacks and Plugins

In Qtap's configuration, traffic processing is organized using two key concepts:

Stacks: Named collections of plugins that work together to process traffic
Plugins: Individual components that perform specific functions on captured traffic

This structure allows you to create different processing configurations for different types of traffic.

Stack Configuration

Stacks are defined in the stacks section of your qpoint.yaml file. Each stack has a unique name and contains one or more plugins:

stacks:
  default_stack:   # Stack name
    plugins:       # List of plugins in this stack
      - type: access_logs
        config:
          # Plugin-specific configuration

You can create multiple stacks for different purposes, each with its own set of plugins and configurations.

Available Plugins

Qtap includes several plugins that provide different processing capabilities.

Access Logs Plugin

The access_logs plugin provides formatted logging of HTTP traffic to stdout. It does not upload to the object store, though this functionality is coming soon.

Only the Error Detection plugin uploads data into the object storage

Basic Configuration

- type: access_logs
  config:
    mode: details        # Default logging level (summary|details|full)
    format: console      # Output format (console|json)

Mode Options

summary: Basic information (method, path, status code)
details: Includes headers and timing information
full: Complete information including request/response bodies

Example with Rules

- type: access_logs
  config:
    mode: summary        # Default mode for most traffic
    format: console      # Human-readable format
    rules:
      - name: "Detailed API Logging"
        expr: request.host == "api.example.com"
        mode: details
      
      - name: "Full Error Logging"
        expr: response.status >= 400
        mode: full

This configuration:

Uses summary mode by default
Applies details mode to api.example.com traffic
Prints to stdout full details for any response with a status code ≥ 400

Error Detection Plugin

The detect_errors plugin captures detailed information when responses meet specific error criteria and optionally uploads the specified headers and/or bodies to the object store.

- type: detect_errors
  config:
    rules:
      - name: "Server Errors"
        trigger_status_codes:
          - '5xx'              # Status code pattern
        report_as_issue: true  # Flag as issue
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: true
      
      - name: "Not Found Errors"
        trigger_status_codes:
          - '404'
        report_as_issue: false
        record_req_headers: true
        record_req_body: false

Status codes can be specified as:

Individual codes: '404', '500'
Code ranges: '4xx', '5xx'

Debug Plugin

Note: This plugin is deprecated. Consider using the access_logs plugin instead.

The debug plugin provides basic logging of HTTP traffic:

- type: debug
  config:
    mode: summary  # or "details" for more information

Rule Expressions in the Access Logs Plugin

The access_logs plugin is the only plugin that supports rule expressions, allowing you to selectively apply different logging modes based on request and response characteristics.

Expression Syntax

Rule expressions follow a straightforward pattern:

<field> <operator> <value>

Multiple conditions can be combined using logical operators:

<field1> <operator1> <value1> and <field2> <operator2> <value2>

Available Fields for Rule Expressions

Based on the HeaderMap RulePairs, the following fields are available:

Request Fields

Field

Example Usage

request.method

request.method == "POST"

request.path

request.path contains "/api/"

request.host

request.host == "api.example.com"

request.url

request.url contains "search"

request.header.<name>

request.header.content-type == "application/json"

Response Fields

Field

Example Usage

response.status

response.status >= 400

response.host

response.host != "api.example.com"

response.url

response.url contains "redirect"

response.header.<name>

response.header.content-type contains "json"

Operators for Rule Expressions

The expressions support various operators based on the rulekit implementation:

Comparison Operators

Operator

Aliases

Description

Example

==

eq

Equal to

request.method == "GET"

!=

ne

Not equal to

request.host != "internal.example.com"

>

gt

Greater than

response.status > 200

>=

ge

Greater than or equal to

response.status >= 400

<

lt

Less than

response.status < 300

<=

le

Less than or equal to

response.status <= 399

=~

matches

Matches regex pattern

request.path matches /^\/api\/v\d+\//

contains

Contains substring

request.url contains "search"

in

Is contained in array

request.method in ["GET", "HEAD"]

Logical Operators

Operator

Aliases

Description

Example

and

&&

Logical AND

request.method == "POST" and response.status >= 400

or

||

Logical OR

request.path contains "/admin" or request.path contains "/auth"

not

!

Logical NOT

not request.host == "public.example.com"

Value Types

The expressions support various value types:

Boolean: true, false
Number: Integer or floating-point values
String: Text enclosed in quotes
Regex Pattern: Patterns enclosed in slashes /pattern/ or vertical bars |pattern|
Array: Values in square brackets, e.g., [value1, value2, value3]

Common Plugin Configurations

Basic Monitoring

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console

Detailed Error Tracking

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
          rules:
            - name: "Log Client Errors"
              expr: response.status >= 400 and response.status < 500
              mode: details
            
            - name: "Log Server Errors"
              expr: response.status >= 500
              mode: full

API-Specific Monitoring

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
  
  api_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          format: json
          rules:
            - name: "Log Authentication Issues"
              expr: response.status == 401 or response.status == 403
              mode: full

tap:
  http:
    stack: default_stack
  endpoints:
    - domain: api.example.com
      http:
        stack: api_stack

This configuration:

Uses the default_stack for most traffic
Applies the api_stack to traffic matching api.example.com
Captures detailed information for all API traffic
Records full details for authentication issues

Creating Multiple Stacks

You can create multiple stacks for different traffic types and apply them selectively:

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
  
  payment_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          rules:
            - name: "Payment Errors"
              expr: response.status >= 400
              mode: full

tap:
  http:
    stack: default_stack
  endpoints:
    - domain: payments.example.com
      http:
        stack: payment_stack

The endpoints section in the tap configuration determines which stack is used for which traffic.

Combining Plugins

You can combine multiple plugins in a single stack:

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
      
      - type: detect_errors
        config:
          rules:
            - name: "Critical Errors"
              trigger_status_codes:
                - '5xx'
              report_as_issue: true
              record_req_headers: true
              record_req_body: true
              record_res_headers: true
              record_res_body: true

Plugins are processed in the order they appear in the configuration.

Complete Example

version: 2

services:
  event_stores:
    - id: console_stdout
      type: stdout
  
  object_stores:
    - id: minio
      type: s3
      endpoint: minio.internal:9000
      bucket: qpoint-objects
      region: us-east-1
      access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
          rules:
            - name: "Detailed Error Logging"
              expr: response.status >= 400
              mode: details
  
  api_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          format: json
          rules:
            - name: "Full API Error Logging"
              expr: response.status >= 400
              mode: full
            
            - name: "Auth Endpoints"
              expr: request.path contains "/auth" and request.method == "POST"
              mode: details
              exclude_req_body: true  # Don't log auth credentials

tap:
  direction: egress
  ignore_loopback: false
  audit_include_dns: true
  http:
    stack: default_stack
  endpoints:
    - domain: api.example.com
      http:
        stack: api_stack

This configuration provides:

Basic summary logging for most traffic
Detailed logging for all errors
Specialized handling for api.example.com traffic
Full logging of API errors
Privacy protection for authentication endpoints

By understanding and utilizing plugins effectively, you can create a configuration that provides the visibility you need while respecting performance and privacy considerations.

PreviousStorage Configuration NextTraffic Capture Settings

Last updated 1 month ago