# OpenTelemetry Integration

This guide shows you how to send qtap observability data to OpenTelemetry-compatible backends using OTLP (OpenTelemetry Protocol).

## Understanding Qtap's OpenTelemetry Integration

Qtap exports two types of data as **OpenTelemetry Logs** via the OTLP protocol:

| Data Type   | Configuration             | What It Contains                                                                                     | Use Case                                                                             |
| ----------- | ------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| **Events**  | qtap.yaml `event_stores`  | Connection metadata, HTTP transaction summaries (method, URL, status, duration, process attribution) | Analytics, monitoring, troubleshooting — know WHAT happened and WHICH process did it |
| **Objects** | qtap.yaml `object_stores` | Full request/response headers and bodies                                                             | Compliance, security investigation, debugging — see the actual payload content       |

Both can be sent through the same OTel collector endpoint, giving you a **unified pipeline** for all captured data — no separate S3 setup required.

{% hint style="info" %}
**Events vs Objects:** Events are lightweight metadata about every connection and HTTP transaction. Objects are the full request/response content, captured selectively based on your [plugin configuration](https://docs.qpoint.io/getting-started/qtap/configuration/traffic-processing-with-plugins). You can send both through OTel, or use OTel for events and S3 for objects — see [Storage Configuration](https://docs.qpoint.io/getting-started/qtap/configuration/storage-configuration) for all options.
{% endhint %}

## Prerequisites

* Qtap installed and running (see [Getting Started](https://docs.qpoint.io/getting-started/qtap/getting-started))
* An OpenTelemetry-compatible backend or collector

## Quick Start (5 minutes)

Send both events and artifacts to any OpenTelemetry-compatible backend.

### Step 1: Configure Qtap

Create `qtap-config.yaml`:

```yaml
version: 2

services:
  event_stores:
    - type: otel  # Send logs to OpenTelemetry
      endpoint: "localhost:4317"  # OTel Collector gRPC endpoint
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false  # Set to true for production

  object_stores:
    - type: otel  # Send artifacts through OTel (or use S3/stdout)
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

stacks:
  default:
    plugins:
      - type: http_capture
        config:
          level: summary  # (none|summary|headers|full)
          format: json    # (json|text)

tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)
  http:
    stack: default
```

### Step 2: Deploy OpenTelemetry Collector

{% hint style="info" %}
**Already have an OpenTelemetry Collector?** Update the `endpoint` in Step 1 to point to your collector and skip to [Step 3](#step-3-start-qtap-logs-only).
{% endhint %}

Create `otel-collector-config.yaml`:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints logs to console for testing
  debug:
    verbosity: detailed

  # Add your backend exporter here
  # otlphttp:
  #   endpoint: "https://your-backend.com/v1/logs"
  #   headers:
  #     api-key: "${API_KEY}"

service:
  pipelines:
    logs:  # Logs pipeline for qtap event_stores
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # Add your backend exporter
```

Create `docker-compose.yaml`:

```yaml
services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
```

Start the collector:

```bash
docker compose up -d
```

### Step 3: Start Qtap

```bash
docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"
```

{% hint style="info" %}
**Note**: No additional environment variables are needed — logs are configured entirely through qtap.yaml.
{% endhint %}

### Step 4: Generate Traffic and Verify

```bash
# Generate test traffic
curl https://httpbin.org/get

# Check OTel Collector logs
docker logs otel-collector
```

**You should see two types of log events:**

**1. Connection Event:**

```
LogRecord
Body: Connection: [egress-external via tcp] hostname → httpbin.org
Attributes:
  event.type: connection
  event.l7Protocol: http2
  event.tlsVersion: 772
  event.meta.connectionId: d3sggv07p3qrvfkj173g
  event.source.exe: /usr/bin/curl
```

**2. Artifact Record Event:**

```
LogRecord
Body: Artifact stored: http_transaction
Attributes:
  event.type: artifact_record
  event.summary.request_method: GET
  event.summary.request_host: httpbin.org
  event.summary.response_status: 200
  event.summary.duration_ms: 2023
  event.summary.connection_id: d3sggv07p3qrvfkj173g
```

**3. Object (Artifact Data):**

When using OTel object stores, the full request/response content is also sent as log records:

```
LogRecord
Body: Artifact: http_request (application/json, 1247 bytes)
Attributes:
  artifact.type: http_request
  artifact.content_type: application/json
  artifact.digest: sha256:a1b2c3d4...
  artifact.size_bytes: 1247
  artifact.data: <raw bytes — full request/response content>
  connection.id: d3sggv07p3qrvfkj173g
  connection.endpoint_id: httpbin.org
  connection.request_id: d3sggv07p3qrvfkj174g
```

Objects include the complete HTTP headers and bodies as captured by the `http_capture` plugin. The `connection.id` links objects back to their corresponding connection and artifact record events.

{% hint style="success" %}
**Success!** Qtap is sending events and artifacts to OpenTelemetry. Use `connection.id` to correlate connection metadata, HTTP transaction summaries, and full request/response content.
{% endhint %}

## Understanding the Data

### What Logs Contain

**Connection Events** (`event.type: connection`):

* TCP connection metadata
* Protocol detection (http1, http2, tcp)
* TLS version and inspection status
* Bytes sent/received
* Process information (exe path, user ID)

**Artifact Record Events** (`event.type: artifact_record`):

* HTTP method, URL, path, status code
* User agent and content type
* Duration in milliseconds
* Process information
* Link to stored artifact (OTel or S3 URL depending on object store type)

### What Objects Contain

When using OTel object stores (`object_stores` type: `otel`), artifact data is sent as additional log records:

* `artifact.type` — `http_request` or `http_response`
* `artifact.content_type` — MIME type (e.g., `application/json`)
* `artifact.data` — The raw bytes (full headers and body)
* `artifact.size_bytes` — Size of the artifact
* `artifact.digest` — Content hash for deduplication
* `connection.id` — Links back to the connection and artifact record events
* `artifact.summary.*` — Request/response metadata (method, host, status, etc.)

{% hint style="info" %}
**Events vs Objects:** Events (`event_stores`) are lightweight metadata sent for every connection and HTTP transaction. Objects (`object_stores`) are the full payload content, captured based on your `http_capture` plugin level setting (`summary`, `details`, or `full`).
{% endhint %}

**Example log query:**

```
event.type = "artifact_record" AND event.summary.response_status >= 500
```

## Kubernetes Deployment

### Using OpenTelemetry Operator

{% hint style="info" %}
**Already have OpenTelemetry Operator installed?** Skip to step 2.
{% endhint %}

1. Install the OpenTelemetry Operator:

```bash
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
```

2. Deploy an OpenTelemetry Collector:

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qtap-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 10s

    exporters:
      otlphttp:
        endpoint: "https://your-backend.com/v1/logs"
        headers:
          api-key: "${API_KEY}"

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]
```

3. Deploy qtap using Helm:

First, create your qtap configuration file `qtap-config.yaml`:

```yaml
version: 2

services:
  event_stores:
    - type: otel
      endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

  object_stores:
    - type: otel
      otel_endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: json

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack
```

Install qtap with Helm:

```bash
# Add Qpoint Helm repository
helm repo add qpoint https://helm.qpoint.io/
helm repo update

# Install qtap with config
helm install qtap qpoint/qtap \
  -n qpoint \
  --create-namespace \
  --set-file config=./qtap-config.yaml \
  --set logLevel=warn
```

Verify the deployment:

```bash
kubectl get pods -n qpoint
kubectl logs -n qpoint -l app.kubernetes.io/name=qtap --tail=50
```

## Backend-Specific Configurations

### SaaS Platforms

#### Datadog

Datadog supports OTLP ingestion directly:

```yaml
event_stores:
  - type: otel
    endpoint: "https://http-intake.logs.datadoghq.com/v2/logs"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      DD-API-KEY:
        type: env
        value: DATADOG_API_KEY
    tls:
      enabled: true
```

#### Honeycomb

```yaml
event_stores:
  - type: otel
    endpoint: "api.honeycomb.io:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      x-honeycomb-team:
        type: env
        value: HONEYCOMB_API_KEY
      x-honeycomb-dataset:
        type: text
        value: "qtap-logs"
    tls:
      enabled: true
```

#### New Relic

```yaml
event_stores:
  - type: otel
    endpoint: "otlp.nr-data.net:4317"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      api-key:
        type: env
        value: NEW_RELIC_LICENSE_KEY
    tls:
      enabled: true
```

#### Grafana Cloud

```yaml
event_stores:
  - type: otel
    endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      authorization:
        type: env
        value: GRAFANA_CLOUD_API_KEY  # Format: "Basic base64(instanceID:apiKey)"
    tls:
      enabled: true
```

#### Elastic

```yaml
event_stores:
  - type: otel
    endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      Authorization:
        type: env
        value: ELASTIC_APM_SECRET_TOKEN
    tls:
      enabled: true
```

### Self-Hosted

For self-hosted backends (Jaeger, Zipkin, Grafana Tempo, etc.), point qtap to your OpenTelemetry Collector, then configure the collector to export to your backend:

```yaml
# Qtap config - same for all self-hosted backends
event_stores:
  - type: otel
    endpoint: "otel-collector.monitoring.svc.cluster.local:4317"
    protocol: grpc
```

Then configure your OTel Collector with the appropriate exporter for your backend.

## Querying and Filtering

### Filter by Event Type

```
# Show only HTTP transaction summaries
event.type = "artifact_record"

# Show only TCP connections
event.type = "connection"
```

### Find Slow Requests

```
event.type = "artifact_record" AND event.summary.duration_ms > 1000
```

### Find Errors

```
# HTTP errors
event.type = "artifact_record" AND event.summary.response_status >= 500
```

### Track Specific Endpoints

```
# Specific host
event.type = "artifact_record" AND event.summary.request_host = "api.example.com"

# Specific path pattern
event.type = "artifact_record" AND event.summary.request_path LIKE "/api/users/%"
```

### Monitor HTTP/2 vs HTTP/1 Traffic

```
# HTTP/2 connections (logs)
event.type = "connection" AND event.l7Protocol = "http2"

# HTTP/1 connections (logs)
event.type = "connection" AND event.l7Protocol = "http1"
```

## Troubleshooting

### No Logs Appearing

**Check qtap logs for OTel connection:**

```bash
docker logs qtap | grep -i "otel\|error"
```

**Common issues:**

1. **Wrong endpoint** - Verify the endpoint address and port
   * gRPC default: `4317`
   * HTTP default: `4318`
   * With `--network=host`: use `localhost:4317`
2. **TLS mismatch** - If backend requires TLS, set `tls.enabled: true`
3. **Authentication** - Verify headers/API keys are correct
4. **Collector not running** - Check `docker ps | grep otel`

### Connection Refused Errors

```
rpc error: code = Unavailable desc = name resolver error
```

**Solutions:**

* Check if OTel Collector is running: `docker ps | grep otel`
* Verify network connectivity between qtap and collector
* Check firewall rules for port 4317/4318

### Debugging with stdout Protocol

For local debugging, use the `stdout` protocol to see OTLP data printed to qtap logs:

```yaml
event_stores:
  - type: otel
    protocol: stdout  # Prints OTLP data to console
    service_name: "qtap"
    environment: "development"
```

Then check qtap logs:

```bash
docker logs qtap
```

## Best Practices

1. **Use TLS in production** - Always enable `tls.enabled: true` for production
2. **Store artifacts securely** - Use OTel object stores for unified pipelines, or S3 for separate storage
3. **Filter appropriately** - Use qtap's filters to avoid capturing unnecessary traffic
4. **Set resource attributes** - Use `service_name` and `environment` for filtering
5. **Monitor qtap itself** - Set up alerts on qtap's Prometheus metrics
6. **Use batch processing** - OTel Collector's batch processor reduces API calls
7. **Use connection.id for correlation** - Link connection events with artifact records using `connection.id`

## Configuration Quick Reference

**qtap.yaml:**

```yaml
services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
```

## Next Steps

* [Storage Configuration](https://docs.qpoint.io/getting-started/qtap/configuration/storage-configuration) — OTel and S3 object storage options
* [Self-Hosted ClickStack Observability](https://docs.qpoint.io/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability) — Full setup with OTel events + objects
* [Set up Prometheus Metrics](https://docs.qpoint.io/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana) for qtap health monitoring
* [Configure Traffic Filters](https://docs.qpoint.io/getting-started/qtap/configuration/traffic-capture-settings) to reduce noise

## Additional Resources

* [Storage Configuration Reference](https://docs.qpoint.io/getting-started/qtap/configuration/storage-configuration)
* [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/)
* [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/)