Traffic Processing with Plugins
Understanding Stacks and Plugins
In Qtap's configuration, traffic processing is organized using two key concepts:
Stacks: Named collections of plugins that work together to process traffic
Plugins: Individual components that perform specific functions on captured traffic
This structure allows you to create different processing configurations for different types of traffic.
Stack Configuration
Stacks are defined in the stacks
section of your qpoint.yaml
file. Each stack has a unique name and contains one or more plugins:
stacks:
default_stack: # Stack name
plugins: # List of plugins in this stack
- type: http_capture
config:
# Plugin-specific configuration
You can create multiple stacks for different purposes, each with its own set of plugins and configurations.
Available Plugins
Qtap includes several plugins that provide different processing capabilities. Rules use Rulekit - Qpoint's flexible expression-based rules engine for evaluating conditions against HTTP traffic.
HTTP Capture Plugin
The http_capture
plugin provides comprehensive HTTP traffic capture with flexible logging levels and the ability to upload payloads to object storage.
Basic Configuration
- type: http_capture
config:
level: summary # Default capture level (none|summary|details|full)
format: text # Output format (text|json)
Level Options
none
: No capture (effectively disables the plugin)summary
: Basic information (method, path, status code)details
: Includes headersfull
: Complete information including request/response bodies
Example with Rules
- type: http_capture
config:
level: summary # Default level for most traffic
format: text # Human-readable format
rules:
- name: "Full capture for httpbin.org"
expr: req.host == "httpbin.org"
level: full
- name: "Headers logging for server errors"
expr: res.status >= 500
level: details
- name: "Full capture for client errors"
expr: res.status >= 400 && res.status < 500
level: full
This configuration:
Uses summary level by default
Captures full details for all traffic to
httpbin.org
Captures detailed information for server errors (5xx)
Captures full information for client errors (4xx)
Container and Pod-based Filtering
You can also create rules based on container or Kubernetes pod attributes:
- type: http_capture
config:
level: summary
format: text
rules:
- name: "Debug specific container"
expr: src.container.name == "my-app-container"
level: full
- name: "Debug by container label"
expr: src.container.labels.qpoint_debug_level == "debug"
level: full
- name: "Debug specific pod"
expr: src.pod.name == "frontend-deployment-abc123"
level: full
- name: "Debug by pod label"
expr: src.pod.labels.app == "frontend" && src.pod.labels.debug == "true"
level: full
The http_capture
plugin uploads captured payloads to the configured object store when capture levels include body content. When using full
level, both headers and bodies are captured and stored. When using details
level, only headers are captured (no body storage).
Access Logs Plugin
The access_logs
plugin provides formatted logging of HTTP traffic to stdout
. It does not upload to the object store.
Basic Configuration
- type: access_logs
config:
mode: details # Default logging level (summary|details|full)
format: console # Output format (console|json)
Mode Options
summary
: Basic information (method, path, status code)details
: Includes headers and timing informationfull
: Complete information including request/response bodies
Example with Rules
- type: access_logs
config:
mode: summary # Default mode for most traffic
format: console # Human-readable format
rules:
- name: "Detailed API Logging"
expr: req.host == "api.example.com"
mode: details
- name: "Full Error Logging"
expr: res.status >= 400
mode: full
Report Usage Plugin
The report_usage
plugin sends anonymized usage metrics to Pulse. This is mainly useful when using Qplane. It works alongside other plugins and doesn't affect traffic capture.
- type: report_usage
This plugin is optional and can be included in any stack.
Error Detection Plugin
The detect_errors
plugin captures detailed information when responses meet specific error criteria and optionally uploads the specified headers and/or bodies to the object store.
- type: detect_errors
config:
rules:
- name: "Server Errors"
trigger_status_codes:
- '5xx' # Status code pattern
report_as_issue: true # Flag as issue
record_req_headers: true
record_req_body: true
record_res_headers: true
record_res_body: true
- name: "Not Found Errors"
trigger_status_codes:
- '404'
report_as_issue: false
record_req_headers: true
record_req_body: false
Status codes can be specified as:
Individual codes:
'404'
,'500'
Code ranges:
'4xx'
,'5xx'
Debug Plugin
The debug
plugin provides basic logging of HTTP traffic:
- type: debug
config:
mode: summary # or "details" for more information
Rule Expressions with Rulekit
Both the http_capture
and access_logs
plugins use Rulekit for rule evaluation. Rulekit is Qpoint's expression-based rules engine that evaluates conditions against key-value data from HTTP traffic.
Expression Syntax
Rule expressions follow a straightforward pattern:
<field> <operator> <value>
Multiple conditions can be combined using logical operators:
<field1> <operator1> <value1> and <field2> <operator2> <value2>
Available Fields for Rule Expressions
Request Fields
req.method
req.method == "POST"
HTTP method (GET, POST, etc.)
req.path
req.path contains "/api/"
Request path
req.host
req.host == "api.example.com"
Host header value
req.url
req.url contains "search"
Full URL
req.header.<name>
req.header.content-type == "application/json"
Request header by name
Response Fields
res.status
res.status >= 400
HTTP status code
res.header.<name>
res.header.content-type contains "json"
Response header by name
Source Context Fields
src.container.name
src.container.name == "my-app"
Container name
src.container.labels.<key>
src.container.labels.app == "frontend"
Container label value
src.pod.name
src.pod.name == "frontend-abc123"
Kubernetes pod name
src.pod.labels.<key>
src.pod.labels.version == "v2"
Pod label value
Operators for Rule Expressions
Rulekit supports various operators for building expressions:
Comparison Operators
==
eq
Equal to
req.method == "GET"
!=
ne
Not equal to
req.host != "internal.example.com"
>
gt
Greater than
res.status > 200
>=
ge
Greater than or equal to
res.status >= 400
<
lt
Less than
res.status < 300
<=
le
Less than or equal to
res.status <= 399
=~
matches
Matches regex pattern
req.path matches /^\/api\/v\d+\//
contains
Contains substring
req.url contains "search"
in
Is contained in array
req.method in ["GET", "HEAD"]
Logical Operators
and
&&
Logical AND
req.method == "POST" and res.status >= 400
or
||
Logical OR
req.path contains "/admin" or req.path contains "/auth"
not
!
Logical NOT
not req.host == "public.example.com"
Value Types
Rulekit expressions support various value types:
Boolean:
true
,false
Number: Integer or floating-point values (e.g.,
200
,1.5
)String: Text enclosed in double quotes (e.g.,
"example.com"
)Regex Pattern: Patterns enclosed in slashes
/pattern/
or vertical bars|pattern|
Array: Values in square brackets (e.g.,
[200, 201, 204]
)
Using Rulekit Macros
You can define reusable expression macros in the rulekit
section of your configuration:
rulekit:
macros:
- name: is_error
expr: res.status >= 400 && res.status < 600
- name: is_api_request
expr: req.path matches /^\/api\//
- name: is_production
expr: src.pod.labels.env == "production" || src.container.labels.env == "production"
stacks:
default_stack:
plugins:
- type: http_capture
config:
level: summary
format: text
rules:
- name: "Capture production API errors"
expr: is_production() && is_api_request() && is_error()
level: full
- name: "Debug all errors"
expr: is_error()
level: details
Complete Configuration Example
Here's a comprehensive example that demonstrates various plugin features:
version: 2
# Define reusable macros
rulekit:
macros:
- name: is_4xx
expr: res.status >= 400 && res.status < 500
- name: is_5xx
expr: res.status >= 500 && res.status < 600
- name: is_debug_enabled
expr: src.container.labels.debug == "true" || src.pod.labels.debug == "true"
services:
event_stores:
- id: console_stdout
type: stdout
object_stores:
- id: minio
type: s3
endpoint: minio.internal:9000
bucket: qpoint-objects
region: us-east-1
access_url: https://minio.internal:9000/{{BUCKET}}/{{DIGEST}}
insecure: false
access_key:
type: env
value: S3_ACCESS_KEY
secret_key:
type: env
value: S3_SECRET_KEY
stacks:
production_stack:
plugins:
- type: http_capture
config:
level: summary
format: text
rules:
# Debug containers with debug label
- name: "Debug mode capture"
expr: is_debug_enabled()
level: full
# Capture errors from specific service
- name: "Payment service errors"
expr: req.host == "payment.api.com" && (is_4xx() || is_5xx())
level: full
# Monitor external API calls
- name: "External API monitoring"
expr: req.host matches /\.(googleapis\.com|amazonaws\.com)$/ && req.method == "POST"
level: details
tap:
direction: egress
ignore_loopback: true
audit_include_dns: true
http:
stack: production_stack
Additional Resources
Rulekit Documentation: For detailed information about rule expressions, operators, and advanced features, visit the Rulekit GitHub repository
Rulekit Examples: The repository includes an interactive CLI demo tool for testing rule expressions
Last updated