Metrics

Qtap exposes Prometheus-compatible metrics endpoints for monitoring HTTP traffic activity and agent health. These can be used to build your own dashboards with Grafana. We have a sample Grafana dashboard available here.

REQUIRED: HTTP-level metrics (requests, responses, latency) require the http_metrics plugin in your stack configuration. Without this plugin, only connection-level metrics will be available. See Configuration below.

Endpoints

Qtap provides two metrics endpoints on localhost:10001:

/metrics

Application metrics for monitoring HTTP traffic observed by qtap:

• Request rates and volumes

• Error rates by status code

• Response times and latency distributions

• Request/response payload sizes

• Traffic patterns by host, method, and protocol

/system/metrics

System metrics for monitoring qtap agent health:

• eBPF program performance

• Memory and CPU usage

• Event processing rates

• Internal error counts

/devtools/ (Optional)

Interactive browser-based interface for debugging HTTP/S traffic in real-time. Requires ENABLE_DEV_TOOLS=true environment variable.

See DevTools - Interactive Traffic Inspection for the complete guide.

Shared Port: Metrics endpoints and DevTools all run on port 10001:

• /metrics - Application metrics (Prometheus)

• /system/metrics - Agent health metrics

• /devtools/ - Interactive debugging UI (requires ENABLE_DEV_TOOLS=true)

Configuration Requirement

CRITICAL: To enable HTTP-level Prometheus metrics, you must add the http_metrics plugin to your stack configuration.

Required Configuration

version: 2

# Storage Configuration - Everything to console
services:
  # Event metadata goes to stdout
  event_stores:
    - type: stdout
  
  # Object data (headers) goes to stdout
  object_stores:
    - type: stdout

# Processing Stack - Simple HTTP capture
stacks:
  starter_stack:
    plugins:
      # REQUIRED for prometheus endpoint
      - type: http_metrics
      # HTTP Capture plugin - outputs to console
      - type: http_capture
        config:
          level: headers  # Capture headers (use 'full' for bodies too)
          format: text    # Human-readable format (use 'json' for structured)

# Traffic Capture Settings
tap:
  direction: egress  # Capture outgoing traffic
  ignore_loopback: false  # Skip localhost traffic
  audit_include_dns: false  # Skip DNS queries for cleaner output
  http:
    stack: starter_stack

Without the http_metrics plugin, you will only see connection-level metrics like qtap_connection_open_total. The HTTP metrics like qtap_http_requests_total will not be available.

Accessing Metrics

From Host

curl http://localhost:10001/metrics
curl http://localhost:10001/system/metrics

From Docker

When using --network=host, access directly from the host:

curl http://localhost:10001/metrics

If NOT using host networking, expose port 10001:

docker run -d --name qtap \
  -p 10001:10001 \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --config="/app/config/qtap.yaml"

Then access:

curl http://localhost:10001/metrics

From Kubernetes

Forward the metrics port:

kubectl port-forward pod/qtap-xxxxx 10001:10001

Access locally:

curl http://localhost:10001/metrics

Available Metrics

HTTP Request Metrics

Metric	Type	Description	Labels
qtap_http_requests_total	Counter	Total HTTP requests observed	host, method, protocol, status_code
qtap_http_requests_duration_ms_bucket	Histogram	Request duration distribution	host, method, protocol, status_code, le
qtap_http_requests_duration_ms_sum	Counter	Total request processing time (ms)	host, method, protocol, status_code
qtap_http_requests_duration_ms_count	Counter	Number of requests measured	host, method, protocol, status_code
qtap_http_requests_size_bytes_bucket	Histogram	Request payload size distribution	host, method, protocol, status_code, le
qtap_http_requests_size_bytes_sum	Counter	Total request payload size (bytes)	host, method, protocol, status_code
qtap_http_requests_size_bytes_count	Counter	Number of requests measured for size	host, method, protocol, status_code

HTTP Response Metrics

Metric	Type	Description	Labels
qtap_http_responses_total	Counter	Total HTTP responses observed	host, method, protocol, status_code
qtap_http_responses_duration_ms_bucket	Histogram	Response duration distribution (TTFB)	host, method, protocol, status_code, le
qtap_http_responses_duration_ms_sum	Counter	Total response time/TTFB (ms)	host, method, protocol, status_code
qtap_http_responses_duration_ms_count	Counter	Number of responses measured	host, method, protocol, status_code
qtap_http_responses_size_bytes_bucket	Histogram	Response payload size distribution	host, method, protocol, status_code, le
qtap_http_responses_size_bytes_sum	Counter	Total response payload size (bytes)	host, method, protocol, status_code
qtap_http_responses_size_bytes_count	Counter	Number of responses measured for size	host, method, protocol, status_code

Combined Metrics

Metric	Type	Description	Labels
qtap_http_duration_ms_bucket	Histogram	Combined request+response duration distribution	None (aggregates all traffic)
qtap_http_duration_ms_sum	Counter	Total combined duration (ms)	None
qtap_http_duration_ms_count	Counter	Number of completed transactions	None

Connection-Level Metrics

These are always available (no plugin required):

Metric	Type	Description	Labels
qtap_connection_open_total	Counter	Total connections opened	direction, remote_addr, remote_port
qtap_connection_close_total	Counter	Total connections closed	direction, remote_addr, remote_port
qtap_connection_active_total	Gauge	Number of currently active connections	direction, remote_addr, remote_port
qtap_connection_protocol_total	Counter	Connections by protocol	protocol
qtap_connection_bytes_sent_total	Counter	Total bytes sent	direction, remote_addr, remote_port
qtap_connection_bytes_recv_total	Counter	Total bytes received	direction, remote_addr, remote_port
qtap_connection_duration_ms	Histogram	Connection duration distribution	direction, remote_addr, remote_port, le
qtap_connection_tls_handshake_total	Counter	Total TLS handshakes completed	direction, remote_addr, remote_port, sni, version

Common Queries

Request Rate

rate(qtap_http_requests_total[5m])

Error Rate

sum(rate(qtap_http_requests_total{status_code=~"4..|5.."}[5m]))
/
sum(rate(qtap_http_requests_total[5m]))

Average Response Time

rate(qtap_http_responses_duration_ms_sum[5m])
/
rate(qtap_http_responses_duration_ms_count[5m])

95th Percentile Latency

histogram_quantile(0.95,
  rate(qtap_http_requests_duration_ms_bucket[5m])
)

Traffic by Host

topk(10, sum by (host) (rate(qtap_http_requests_total[5m])))

HTTP/2 Traffic

sum(rate(qtap_http_requests_total{protocol="http2"}[5m]))

Average Request Size

rate(qtap_http_requests_size_bytes_sum[5m])
/
rate(qtap_http_requests_size_bytes_count[5m])

Overall Transaction Duration

# Average combined request+response time
rate(qtap_http_duration_ms_sum[5m])
/
rate(qtap_http_duration_ms_count[5m])

Active Connections

# Total active connections across all destinations
sum(qtap_connection_active_total)

TLS Handshake Rate

# Overall handshake rate
sum(rate(qtap_connection_tls_handshake_total[5m]))

Label Cardinality

Metrics include labels for host, method, protocol, and status_code. This enables powerful filtering but can increase cardinality in high-traffic environments.

To reduce cardinality:

1. Filter noisy processes using Traffic Capture Settings:

filters:
  groups:
    - qpoint
  custom:
    - exe: /usr/bin/noisy-app
      strategy: exact

1. Use domain-specific capture to limit metrics to important services:

tap:
  endpoints:
    - domain: 'important-api.example.com'
      http:
        stack: monitored_stack

1. Implement Prometheus recording rules to pre-aggregate metrics

Troubleshooting

No HTTP Metrics Appearing

If you don't see qtap_http_requests_total or other HTTP metrics:

1. Verify the http_metrics plugin is configured:

# Check your qtap config includes http_metrics plugin
cat config.yaml | grep -A 5 "plugins:"

1. Restart qtap after adding the plugin:

docker restart qtap
# Or for Kubernetes:
kubectl rollout restart daemonset/qtap

1. Verify metrics are now available:

curl http://localhost:10001/metrics | grep "qtap_http_requests_total"

Metrics Show Zero Values

If metrics exist but show zero counts despite HTTP traffic:

1. Verify qtap is capturing traffic - check logs for HTTP transactions

2. Check capture direction - ensure tap.direction matches your traffic flow

3. Verify filters - ensure you're not filtering out all traffic

Only Connection Metrics Available

If you only see qtap_connection_* metrics but no qtap_http_* metrics, the http_metrics plugin is missing from your configuration.

Integration

See the Monitoring Qtap with Prometheus and Grafana guide for step-by-step setup instructions.

For a comprehensive metrics reference including alerting examples, see Prometheus Metrics in the Appendix.