> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/appendix/prometheus-metrics.md).

# Prometheus Metrics

Qtap exposes Prometheus-compatible metrics endpoints for monitoring traffic activity and agent health. These metrics enable you to build dashboards, set up alerts, and track system performance.

{% hint style="warning" %}
**REQUIRED**: HTTP-level metrics require the `http_metrics` plugin in your stack configuration. See [Configuration Requirement](#configuration-requirement) below.
{% endhint %}

## Configuration Requirement

To enable HTTP-level Prometheus metrics, you **must** add the `http_metrics` plugin to your stack:

```yaml
stacks:
  my_stack:
    plugins:
      - type: http_capture  # Optional
        config:
          level: summary
      - type: http_metrics  # REQUIRED for Prometheus metrics
```

Without this plugin, only connection-level metrics will be available.

## Metrics Endpoints

Qtap provides two metrics endpoints on `localhost:10001`:

### `/metrics`

Application metrics for monitoring HTTP traffic activity on your system. Use these metrics to build dashboards tracking:

* Request rates and volumes
* Error rates and status code distributions
* Response times and latency percentiles
* Request/response payload sizes
* Traffic patterns by host, method, and protocol

### `/system/metrics`

System metrics related to the health and performance of the qtap agent itself. These include:

* eBPF program performance
* Memory and CPU usage
* Event processing rates
* Internal error counts

## Accessing Metrics

### From the Host

```bash
curl http://localhost:10001/metrics
curl http://localhost:10001/system/metrics
```

### From Docker Container

When running qtap in Docker, expose port 10001:

```bash
docker run -d --name qtap \
  -p 10001:10001 \
  # ... other flags ...
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0
```

Then access from the host:

```bash
curl http://localhost:10001/metrics
```

### From Kubernetes

Forward the metrics port to your local machine:

```bash
kubectl port-forward pod/qtap-xxxxx 10001:10001
```

Then access locally:

```bash
curl http://localhost:10001/metrics
```

## Key Application Metrics

The `/metrics` endpoint provides these HTTP traffic metrics:

### Request Metrics

**`qtap_http_requests_total`**

* **Type**: Counter
* **Description**: Total number of HTTP requests observed
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Calculate request rates, error rates, and traffic distribution

**`qtap_http_requests_duration_ms_bucket`**

* **Type**: Histogram
* **Description**: Request duration distribution in milliseconds
* **Labels**: `host`, `method`, `protocol`, `status_code`, `le` (histogram bucket)
* **Use**: Calculate latency percentiles (p50, p95, p99)

**`qtap_http_requests_duration_ms_sum`**

* **Type**: Counter
* **Description**: Total request processing time in milliseconds
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Calculate average request duration

**`qtap_http_requests_duration_ms_count`**

* **Type**: Counter
* **Description**: Total number of requests measured for duration
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Paired with `_sum` for average calculations

**`qtap_http_requests_size_bytes_bucket`**

* **Type**: Histogram
* **Description**: Request payload size distribution in bytes
* **Labels**: `host`, `method`, `protocol`, `status_code`, `le`
* **Use**: Calculate payload size percentiles

**`qtap_http_requests_size_bytes_sum`**

* **Type**: Counter
* **Description**: Total size of HTTP request payloads in bytes
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Calculate average request size

**`qtap_http_requests_size_bytes_count`**

* **Type**: Counter
* **Description**: Number of requests measured for size
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Paired with `_sum` for average calculations

### Response Metrics

**`qtap_http_responses_total`**

* **Type**: Counter
* **Description**: Total number of HTTP responses observed
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Track response counts

**`qtap_http_responses_duration_ms_bucket`**

* **Type**: Histogram
* **Description**: Response duration distribution (TTFB) in milliseconds
* **Labels**: `host`, `method`, `protocol`, `status_code`, `le`
* **Use**: Calculate response time percentiles

**`qtap_http_responses_duration_ms_sum`**

* **Type**: Counter
* **Description**: Total response time in milliseconds (TTFB - Time To First Byte)
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Calculate average response times

**`qtap_http_responses_duration_ms_count`**

* **Type**: Counter
* **Description**: Number of responses measured for duration
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Paired with `_sum` for average calculations

**`qtap_http_responses_size_bytes_bucket`**

* **Type**: Histogram
* **Description**: Response payload size distribution in bytes
* **Labels**: `host`, `method`, `protocol`, `status_code`, `le`
* **Use**: Calculate payload size percentiles

**`qtap_http_responses_size_bytes_sum`**

* **Type**: Counter
* **Description**: Total size of HTTP response payloads in bytes
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Calculate average response size

**`qtap_http_responses_size_bytes_count`**

* **Type**: Counter
* **Description**: Number of responses measured for size
* **Labels**: `host`, `method`, `protocol`, `status_code`
* **Use**: Paired with `_sum` for average calculations

### Combined Metrics

**`qtap_http_duration_ms_bucket`**

* **Type**: Histogram
* **Description**: Combined request+response duration distribution in milliseconds
* **Labels**: None (aggregates all traffic)
* **Use**: Calculate overall latency percentiles across all hosts

**`qtap_http_duration_ms_sum`**

* **Type**: Counter
* **Description**: Total combined request+response time in milliseconds
* **Labels**: None
* **Use**: Calculate overall average transaction duration

**`qtap_http_duration_ms_count`**

* **Type**: Counter
* **Description**: Total number of completed HTTP transactions
* **Labels**: None
* **Use**: Paired with `_sum` for average calculations

### Connection-Level Metrics

These metrics are always available without the `http_metrics` plugin:

**`qtap_connection_open_total`**

* **Type**: Counter
* **Description**: Total number of connections opened
* **Labels**: `direction`, `remote_addr`, `remote_port`

**`qtap_connection_close_total`**

* **Type**: Counter
* **Description**: Total number of connections closed
* **Labels**: `direction`, `remote_addr`, `remote_port`

**`qtap_connection_active_total`**

* **Type**: Gauge
* **Description**: Number of currently active connections
* **Labels**: `direction`, `remote_addr`, `remote_port`
* **Use**: Monitor real-time connection usage by destination

**`qtap_connection_protocol_total`**

* **Type**: Counter
* **Description**: Total connections by protocol type
* **Labels**: `protocol`
* **Use**: Track distribution of connection protocols (http1, http2, etc.)

**`qtap_connection_bytes_sent_total`**

* **Type**: Counter
* **Description**: Total bytes sent across all connections
* **Labels**: `direction`, `remote_addr`, `remote_port`

**`qtap_connection_bytes_recv_total`**

* **Type**: Counter
* **Description**: Total bytes received across all connections
* **Labels**: `direction`, `remote_addr`, `remote_port`

**`qtap_connection_duration_ms`**

* **Type**: Histogram
* **Description**: Connection duration distribution in milliseconds
* **Labels**: `direction`, `remote_addr`, `remote_port`, `le`

**`qtap_connection_tls_handshake_total`**

* **Type**: Counter
* **Description**: Total number of TLS handshakes completed
* **Labels**: `direction`, `remote_addr`, `remote_port`, `sni`, `version`
* **Use**: Track TLS/HTTPS connection establishment by SNI and TLS version

## Common Queries

### Request Rate (per second)

```promql
rate(qtap_http_requests_total[5m])
```

### Error Rate (4xx and 5xx responses)

```promql
sum(rate(qtap_http_requests_total{status_code=~"4..|5.."}[5m]))
/
sum(rate(qtap_http_requests_total[5m]))
```

### Average Response Time

```promql
rate(qtap_http_responses_duration_ms_sum[5m])
/
rate(qtap_http_responses_duration_ms_count[5m])
```

### 95th Percentile Latency

```promql
histogram_quantile(0.95,
  rate(qtap_http_requests_duration_ms_bucket[5m])
)
```

### Top Hosts by Request Volume

```promql
topk(10, sum by (host) (rate(qtap_http_requests_total[5m])))
```

### Request Rate by Method

```promql
sum by (method) (rate(qtap_http_requests_total[5m]))
```

### Overall Transaction Latency (Combined)

```promql
# Average overall transaction time
rate(qtap_http_duration_ms_sum[5m])
/
rate(qtap_http_duration_ms_count[5m])

# 95th percentile overall latency
histogram_quantile(0.95,
  rate(qtap_http_duration_ms_bucket[5m])
)
```

### Active Connections

```promql
# Total active connections
sum(qtap_connection_active_total)

# Active connections by destination
topk(10, qtap_connection_active_total)

# Active connections by direction
sum by (direction) (qtap_connection_active_total)
```

### TLS Handshake Rate

```promql
# Overall handshake rate
sum(rate(qtap_connection_tls_handshake_total[5m]))

# Handshake rate by SNI (domain)
sum by (sni) (rate(qtap_connection_tls_handshake_total[5m]))

# Handshake rate by TLS version
sum by (version) (rate(qtap_connection_tls_handshake_total[5m]))
```

### Connection Throughput

```promql
# Bytes sent per second
rate(qtap_connection_bytes_sent_total[5m])

# Bytes received per second
rate(qtap_connection_bytes_recv_total[5m])
```

## Grafana Dashboard

Qpoint provides a sample Grafana dashboard that visualizes these metrics:

**Dashboard**: [qtap-http-overview.json](https://github.com/qpoint-io/qtap/blob/main/examples/dashboards/qtap-http-overview.json)

The dashboard includes:

* **Request Rate**: Overall HTTP request volume over time
* **Error Rate**: Percentage of failed requests (4xx/5xx)
* **Response Time**: Average response duration and TTFB
* **Latency Percentiles**: p50, p95, p99 request duration
* **Request/Response Sizes**: Average payload sizes
* **Traffic by Host**: Request distribution across hosts
* **Status Code Distribution**: Breakdown of HTTP status codes

### Importing the Dashboard

1. Download the dashboard JSON from the link above
2. In Grafana, navigate to **Dashboards** → **Import**
3. Upload the JSON file or paste its contents
4. Configure your Prometheus datasource
5. Save the dashboard

## Prometheus Configuration

Add qtap as a scrape target in your `prometheus.yml`:

```yaml
scrape_configs:
  - job_name: 'qtap'
    static_configs:
      - targets: ['localhost:10001']
    metrics_path: '/metrics'
    scrape_interval: 15s

  - job_name: 'qtap-system'
    static_configs:
      - targets: ['localhost:10001']
    metrics_path: '/system/metrics'
    scrape_interval: 30s
```

For Kubernetes deployments, use ServiceMonitor (Prometheus Operator) or pod annotations for automatic discovery:

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "10001"
  prometheus.io/path: "/metrics"
```

## Alerting Examples

### High Error Rate Alert

```yaml
groups:
  - name: qtap
    rules:
      - alert: HighHTTPErrorRate
        expr: |
          (
            sum(rate(qtap_http_requests_total{status_code=~"5.."}[5m]))
            /
            sum(rate(qtap_http_requests_total[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High HTTP 5xx error rate"
          description: "Error rate is {{ $value | humanizePercentage }}"
```

### High Latency Alert

```yaml
- alert: HighResponseLatency
  expr: |
    histogram_quantile(0.95,
      rate(qtap_http_requests_duration_ms_bucket[5m])
    ) > 1000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "High response latency detected"
    description: "95th percentile latency is {{ $value }}ms"
```

## Best Practices

1. **Use labels wisely**: The `host`, `method`, `protocol`, and `status_code` labels enable powerful filtering but can increase cardinality. Filter noisy traffic if needed.
2. **Set appropriate scrape intervals**: 15-30 seconds is typical for application metrics. System metrics can be scraped less frequently.
3. **Use recording rules**: Pre-compute expensive queries like percentiles to improve dashboard performance.
4. **Monitor agent health**: Track `/system/metrics` to ensure qtap itself is healthy and not experiencing backpressure.
5. **Set up alerts**: Don't just collect metrics - alert on anomalies like error rate spikes or latency increases.

## Troubleshooting

### Metrics endpoint not accessible

Check that qtap is running and port 10001 is accessible:

```bash
docker logs qtap
netstat -tlnp | grep 10001
```

### No metrics appearing

Verify qtap is capturing traffic:

```bash
# Check qtap logs for capture activity
docker logs qtap | grep -i http

# Generate test traffic
curl https://httpbin.org/get

# Check metrics again
curl http://localhost:10001/metrics | grep qtap_http_requests_total
```

### High cardinality issues

If you have too many unique label combinations:

1. Use filters to exclude noisy processes (see [Traffic Capture Settings](/getting-started/qtap/configuration/traffic-capture-settings.md))
2. Aggregate metrics in Prometheus recording rules
3. Limit capture to specific hosts with endpoints configuration