Traffic Processing with Plugins

Understanding Stacks and Plugins

In Qtap's configuration, traffic processing is organized using two key concepts:

• Stacks: Named collections of plugins that work together to process traffic

• Plugins: Individual components that perform specific functions on captured traffic

This structure allows you to create different processing configurations for different types of traffic.

Stack Configuration

Stacks are defined in the stacks section of your qpoint.yaml file. Each stack has a unique name and contains one or more plugins:

stacks:
  default_stack:   # Stack name
    plugins:       # List of plugins in this stack
      - type: access_logs
        config:
          # Plugin-specific configuration

You can create multiple stacks for different purposes, each with its own set of plugins and configurations.

Available Plugins

Qtap includes several plugins that provide different processing capabilities.

Access Logs Plugin

The access_logs plugin provides formatted logging of HTTP traffic to stdout. It does not upload to the object store, though this functionality is coming soon.

Only the Error Detection plugin uploads data into the object storage

Basic Configuration

- type: access_logs
  config:
    mode: details        # Default logging level (summary|details|full)
    format: console      # Output format (console|json)

Mode Options

• summary: Basic information (method, path, status code)

• details: Includes headers and timing information

• full: Complete information including request/response bodies

Example with Rules

- type: access_logs
  config:
    mode: summary        # Default mode for most traffic
    format: console      # Human-readable format
    rules:
      - name: "Detailed API Logging"
        expr: request.host == "api.example.com"
        mode: details
      
      - name: "Full Error Logging"
        expr: response.status >= 400
        mode: full

This configuration:

1. Uses summary mode by default

2. Applies details mode to api.example.com traffic

3. Prints to stdout full details for any response with a status code ≥ 400

Error Detection Plugin

The detect_errors plugin captures detailed information when responses meet specific error criteria and optionally uploads the specified headers and/or bodies to the object store.

- type: detect_errors
  config:
    rules:
      - name: "Server Errors"
        trigger_status_codes:
          - '5xx'              # Status code pattern
        report_as_issue: true  # Flag as issue
        record_req_headers: true
        record_req_body: true
        record_res_headers: true
        record_res_body: true
      
      - name: "Not Found Errors"
        trigger_status_codes:
          - '404'
        report_as_issue: false
        record_req_headers: true
        record_req_body: false

Status codes can be specified as:

• Individual codes: '404', '500'

• Code ranges: '4xx', '5xx'

Debug Plugin

Note: This plugin is deprecated. Consider using the access_logs plugin instead.

The debug plugin provides basic logging of HTTP traffic:

- type: debug
  config:
    mode: summary  # or "details" for more information

Rule Expressions in the Access Logs Plugin

The access_logs plugin is the only plugin that supports rule expressions, allowing you to selectively apply different logging modes based on request and response characteristics.

Expression Syntax

Rule expressions follow a straightforward pattern:

<field> <operator> <value>

Multiple conditions can be combined using logical operators:

<field1> <operator1> <value1> and <field2> <operator2> <value2>

Available Fields for Rule Expressions

Based on the HeaderMap RulePairs, the following fields are available:

Request Fields

Field	Example Usage
request.method	request.method == "POST"
request.path	request.path contains "/api/"
request.host	request.host == "api.example.com"
request.url	request.url contains "search"
request.header.<name>	request.header.content-type == "application/json"

Response Fields

Field	Example Usage
response.status	response.status >= 400
response.host	response.host != "api.example.com"
response.url	response.url contains "redirect"
response.header.<name>	response.header.content-type contains "json"

Operators for Rule Expressions

The expressions support various operators based on the rulekit implementation:

Comparison Operators

Operator	Aliases	Description	Example
==	eq	Equal to	request.method == "GET"
!=	ne	Not equal to	request.host != "internal.example.com"
>	gt	Greater than	response.status > 200
>=	ge	Greater than or equal to	response.status >= 400
<	lt	Less than	response.status < 300
<=	le	Less than or equal to	response.status <= 399
=~	matches	Matches regex pattern	request.path matches /^\/api\/v\d+\//
contains		Contains substring	request.url contains "search"
in		Is contained in array	request.method in ["GET", "HEAD"]

Logical Operators

Operator	Aliases	Description	Example
and	&&	Logical AND	request.method == "POST" and response.status >= 400
or	||	Logical OR	request.path contains "/admin" or request.path contains "/auth"
not	!	Logical NOT	not request.host == "public.example.com"

Value Types

The expressions support various value types:

• Boolean: true, false

• Number: Integer or floating-point values

• String: Text enclosed in quotes

• Regex Pattern: Patterns enclosed in slashes /pattern/ or vertical bars |pattern|

• Array: Values in square brackets, e.g., [value1, value2, value3]

Common Plugin Configurations

Basic Monitoring

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console

Detailed Error Tracking

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
          rules:
            - name: "Log Client Errors"
              expr: response.status >= 400 and response.status < 500
              mode: details
            
            - name: "Log Server Errors"
              expr: response.status >= 500
              mode: full

API-Specific Monitoring

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
  
  api_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          format: json
          rules:
            - name: "Log Authentication Issues"
              expr: response.status == 401 or response.status == 403
              mode: full

tap:
  http:
    stack: default_stack
  endpoints:
    - domain: api.example.com
      http:
        stack: api_stack

This configuration:

1. Uses the default_stack for most traffic

2. Applies the api_stack to traffic matching api.example.com

3. Captures detailed information for all API traffic

4. Records full details for authentication issues

Creating Multiple Stacks

You can create multiple stacks for different traffic types and apply them selectively:

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
  
  payment_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          rules:
            - name: "Payment Errors"
              expr: response.status >= 400
              mode: full

tap:
  http:
    stack: default_stack
  endpoints:
    - domain: payments.example.com
      http:
        stack: payment_stack

The endpoints section in the tap configuration determines which stack is used for which traffic.

Combining Plugins

You can combine multiple plugins in a single stack:

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
      
      - type: detect_errors
        config:
          rules:
            - name: "Critical Errors"
              trigger_status_codes:
                - '5xx'
              report_as_issue: true
              record_req_headers: true
              record_req_body: true
              record_res_headers: true
              record_res_body: true

Plugins are processed in the order they appear in the configuration.

Complete Example

version: 2

services:
  event_stores:
    - id: console_stdout
      type: stdout
  
  object_stores:
    - id: minio
      type: s3
      endpoint: minio.internal:9000
      bucket: qpoint-objects
      region: us-east-1
      access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
      insecure: false
      access_key:
        type: env
        value: S3_ACCESS_KEY
      secret_key:
        type: env
        value: S3_SECRET_KEY

stacks:
  default_stack:
    plugins:
      - type: access_logs
        config:
          mode: summary
          format: console
          rules:
            - name: "Detailed Error Logging"
              expr: response.status >= 400
              mode: details
  
  api_stack:
    plugins:
      - type: access_logs
        config:
          mode: details
          format: json
          rules:
            - name: "Full API Error Logging"
              expr: response.status >= 400
              mode: full
            
            - name: "Auth Endpoints"
              expr: request.path contains "/auth" and request.method == "POST"
              mode: details
              exclude_req_body: true  # Don't log auth credentials

tap:
  direction: egress
  ignore_loopback: false
  audit_include_dns: true
  http:
    stack: default_stack
  endpoints:
    - domain: api.example.com
      http:
        stack: api_stack

This configuration provides:

1. Basic summary logging for most traffic

2. Detailed logging for all errors

3. Specialized handling for api.example.com traffic

4. Full logging of API errors

5. Privacy protection for authentication endpoints

By understanding and utilizing plugins effectively, you can create a configuration that provides the visibility you need while respecting performance and privacy considerations.