> For the complete documentation index, see [llms.txt](https://docs.qpoint.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qpoint.io/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry.md).

# OpenTelemetry Integration

This guide shows you how to send qtap observability data to OpenTelemetry-compatible backends using OTLP (OpenTelemetry Protocol).

## Understanding Qtap's OpenTelemetry Integration

Qtap exports two types of data as **OpenTelemetry Logs** via the OTLP protocol:

| Data Type   | Configuration             | What It Contains                                                                                     | Use Case                                                                             |
| ----------- | ------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| **Events**  | qtap.yaml `event_stores`  | Connection metadata, HTTP transaction summaries (method, URL, status, duration, process attribution) | Analytics, monitoring, troubleshooting — know WHAT happened and WHICH process did it |
| **Objects** | qtap.yaml `object_stores` | Full request/response headers and bodies                                                             | Compliance, security investigation, debugging — see the actual payload content       |

Both can be sent through the same OTel collector endpoint, giving you a **unified pipeline** for all captured data — no separate S3 setup required.

{% hint style="info" %}
**Events vs Objects:** Events are lightweight metadata about every connection and HTTP transaction. Objects are the full request/response content, captured selectively based on your [plugin configuration](/getting-started/qtap/configuration/traffic-processing-with-plugins.md). You can send both through OTel, or use OTel for events and S3 for objects — see [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) for all options.
{% endhint %}

## Prerequisites

* Qtap installed and running (see [Getting Started](/getting-started/qtap/getting-started.md))
* An OpenTelemetry-compatible backend or collector

## Quick Start (5 minutes)

Send both events and artifacts to any OpenTelemetry-compatible backend.

### Step 1: Configure Qtap

Create `qtap-config.yaml`:

```yaml
version: 2

services:
  event_stores:
    - type: otel  # Send logs to OpenTelemetry
      endpoint: "localhost:4317"  # OTel Collector gRPC endpoint
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false  # Set to true for production

  object_stores:
    - type: otel  # Send artifacts through OTel (or use S3/stdout)
      otel_endpoint: "localhost:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"
      tls:
        enabled: false

stacks:
  default:
    plugins:
      - type: http_capture
        config:
          level: summary  # (none|summary|headers|full)
          format: json    # (json|text)

tap:
  direction: egress        # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: false   # (true|false)
  audit_include_dns: false # (true|false)
  http:
    stack: default
```

### Step 2: Deploy OpenTelemetry Collector

{% hint style="info" %}
**Already have an OpenTelemetry Collector?** Update the `endpoint` in Step 1 to point to your collector and skip to [Step 3](#step-3-start-qtap-logs-only).
{% endhint %}

Create `otel-collector-config.yaml`:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints logs to console for testing
  debug:
    verbosity: detailed

  # Add your backend exporter here
  # otlphttp:
  #   endpoint: "https://your-backend.com/v1/logs"
  #   headers:
  #     api-key: "${API_KEY}"

service:
  pipelines:
    logs:  # Logs pipeline for qtap event_stores
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # Add your backend exporter
```

Create `docker-compose.yaml`:

```yaml
services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
```

Start the collector:

```bash
docker compose up -d
```

### Step 3: Start Qtap

```bash
docker run -d --name qtap \
  --user 0:0 --privileged \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid=host --network=host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/qtap-config.yaml:/app/config/qtap.yaml" \
  -e TINI_SUBREAPER=1 \
  --ulimit=memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config="/app/config/qtap.yaml"
```

{% hint style="info" %}
**Note**: No additional environment variables are needed — logs are configured entirely through qtap.yaml.
{% endhint %}

### Step 4: Generate Traffic and Verify

```bash
# Generate test traffic
curl https://httpbin.org/get

# Check OTel Collector logs
docker logs otel-collector
```

**You should see two types of log events:**

**1. Connection Event:**

```
LogRecord
Body: Connection: [egress-external via tcp] hostname → httpbin.org
Attributes:
  event.type: connection
  event.l7Protocol: http2
  event.tlsVersion: 772
  event.meta.connectionId: d3sggv07p3qrvfkj173g
  event.source.exe: /usr/bin/curl
```

**2. Artifact Record Event:**

```
LogRecord
Body: Artifact stored: http_transaction
Attributes:
  event.type: artifact_record
  event.summary.request_method: GET
  event.summary.request_host: httpbin.org
  event.summary.response_status: 200
  event.summary.duration_ms: 2023
  event.summary.connection_id: d3sggv07p3qrvfkj173g
```

**3. Object (Artifact Data):**

When using OTel object stores, the full request/response content is also sent as log records:

```
LogRecord
Body: Artifact: http_request (application/json, 1247 bytes)
Attributes:
  artifact.type: http_request
  artifact.content_type: application/json
  artifact.digest: sha256:a1b2c3d4...
  artifact.size_bytes: 1247
  artifact.data: <raw bytes — full request/response content>
  connection.id: d3sggv07p3qrvfkj173g
  connection.endpoint_id: httpbin.org
  connection.request_id: d3sggv07p3qrvfkj174g
```

Objects include the complete HTTP headers and bodies as captured by the `http_capture` plugin. The `connection.id` links objects back to their corresponding connection and artifact record events.

{% hint style="success" %}
**Success!** Qtap is sending events and artifacts to OpenTelemetry. Use `connection.id` to correlate connection metadata, HTTP transaction summaries, and full request/response content.
{% endhint %}

## Understanding the Data

### What Logs Contain

**Connection Events** (`event.type: connection`):

* TCP connection metadata
* Protocol detection (http1, http2, tcp)
* TLS version and inspection status
* Bytes sent/received
* Process information (exe path, user ID)

**Artifact Record Events** (`event.type: artifact_record`):

* HTTP method, URL, path, status code
* User agent and content type
* Duration in milliseconds
* Process information
* Link to stored artifact (OTel or S3 URL depending on object store type)

### What Objects Contain

When using OTel object stores (`object_stores` type: `otel`), artifact data is sent as additional log records:

* `artifact.type` — `http_request` or `http_response`
* `artifact.content_type` — MIME type (e.g., `application/json`)
* `artifact.data` — The raw bytes (full headers and body)
* `artifact.size_bytes` — Size of the artifact
* `artifact.digest` — Content hash for deduplication
* `connection.id` — Links back to the connection and artifact record events
* `artifact.summary.*` — Request/response metadata (method, host, status, etc.)

{% hint style="info" %}
**Events vs Objects:** Events (`event_stores`) are lightweight metadata sent for every connection and HTTP transaction. Objects (`object_stores`) are the full payload content, captured based on your `http_capture` plugin level setting (`summary`, `details`, or `full`).
{% endhint %}

**Example log query:**

```
event.type = "artifact_record" AND event.summary.response_status >= 500
```

## Kubernetes Deployment

### Using OpenTelemetry Operator

{% hint style="info" %}
**Already have OpenTelemetry Operator installed?** Skip to step 2.
{% endhint %}

1. Install the OpenTelemetry Operator:

```bash
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
```

2. Deploy an OpenTelemetry Collector:

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qtap-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 10s

    exporters:
      otlphttp:
        endpoint: "https://your-backend.com/v1/logs"
        headers:
          api-key: "${API_KEY}"

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]
```

3. Deploy qtap using Helm:

First, create your qtap configuration file `qtap-config.yaml`:

```yaml
version: 2

services:
  event_stores:
    - type: otel
      endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

  object_stores:
    - type: otel
      otel_endpoint: "qtap-collector-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      service_name: "qtap"
      environment: "production"

stacks:
  default_stack:
    plugins:
      - type: http_capture
        config:
          level: summary
          format: json

tap:
  direction: egress
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: default_stack
```

Install qtap with Helm:

```bash
# Add Qpoint Helm repository
helm repo add qpoint https://helm.qpoint.io/
helm repo update

# Install qtap with config
helm install qtap qpoint/qtap \
  -n qpoint \
  --create-namespace \
  --set-file config=./qtap-config.yaml \
  --set logLevel=warn
```

Verify the deployment:

```bash
kubectl get pods -n qpoint
kubectl logs -n qpoint -l app.kubernetes.io/name=qtap --tail=50
```

## Backend-Specific Configurations

### SaaS Platforms

#### Datadog

Datadog supports OTLP ingestion directly:

```yaml
event_stores:
  - type: otel
    endpoint: "https://http-intake.logs.datadoghq.com/v2/logs"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      DD-API-KEY:
        type: env
        value: DATADOG_API_KEY
    tls:
      enabled: true
```

#### Honeycomb

```yaml
event_stores:
  - type: otel
    endpoint: "api.honeycomb.io:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      x-honeycomb-team:
        type: env
        value: HONEYCOMB_API_KEY
      x-honeycomb-dataset:
        type: text
        value: "qtap-logs"
    tls:
      enabled: true
```

#### New Relic

```yaml
event_stores:
  - type: otel
    endpoint: "otlp.nr-data.net:4317"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      api-key:
        type: env
        value: NEW_RELIC_LICENSE_KEY
    tls:
      enabled: true
```

#### Grafana Cloud

```yaml
event_stores:
  - type: otel
    endpoint: "otlp-gateway-prod-us-central-0.grafana.net:443"
    protocol: grpc
    service_name: "qtap"
    environment: "production"
    headers:
      authorization:
        type: env
        value: GRAFANA_CLOUD_API_KEY  # Format: "Basic base64(instanceID:apiKey)"
    tls:
      enabled: true
```

#### Elastic

```yaml
event_stores:
  - type: otel
    endpoint: "https://your-deployment.es.us-central1.gcp.cloud.es.io:443"
    protocol: http
    service_name: "qtap"
    environment: "production"
    headers:
      Authorization:
        type: env
        value: ELASTIC_APM_SECRET_TOKEN
    tls:
      enabled: true
```

### Self-Hosted

For self-hosted backends (Jaeger, Zipkin, Grafana Tempo, etc.), point qtap to your OpenTelemetry Collector, then configure the collector to export to your backend:

```yaml
# Qtap config - same for all self-hosted backends
event_stores:
  - type: otel
    endpoint: "otel-collector.monitoring.svc.cluster.local:4317"
    protocol: grpc
```

Then configure your OTel Collector with the appropriate exporter for your backend.

## Querying and Filtering

### Filter by Event Type

```
# Show only HTTP transaction summaries
event.type = "artifact_record"

# Show only TCP connections
event.type = "connection"
```

### Find Slow Requests

```
event.type = "artifact_record" AND event.summary.duration_ms > 1000
```

### Find Errors

```
# HTTP errors
event.type = "artifact_record" AND event.summary.response_status >= 500
```

### Track Specific Endpoints

```
# Specific host
event.type = "artifact_record" AND event.summary.request_host = "api.example.com"

# Specific path pattern
event.type = "artifact_record" AND event.summary.request_path LIKE "/api/users/%"
```

### Monitor HTTP/2 vs HTTP/1 Traffic

```
# HTTP/2 connections (logs)
event.type = "connection" AND event.l7Protocol = "http2"

# HTTP/1 connections (logs)
event.type = "connection" AND event.l7Protocol = "http1"
```

## Troubleshooting

### No Logs Appearing

**Check qtap logs for OTel connection:**

```bash
docker logs qtap | grep -i "otel\|error"
```

**Common issues:**

1. **Wrong endpoint** - Verify the endpoint address and port
   * gRPC default: `4317`
   * HTTP default: `4318`
   * With `--network=host`: use `localhost:4317`
2. **TLS mismatch** - If backend requires TLS, set `tls.enabled: true`
3. **Authentication** - Verify headers/API keys are correct
4. **Collector not running** - Check `docker ps | grep otel`

### Connection Refused Errors

```
rpc error: code = Unavailable desc = name resolver error
```

**Solutions:**

* Check if OTel Collector is running: `docker ps | grep otel`
* Verify network connectivity between qtap and collector
* Check firewall rules for port 4317/4318

### Debugging with stdout Protocol

For local debugging, use the `stdout` protocol to see OTLP data printed to qtap logs:

```yaml
event_stores:
  - type: otel
    protocol: stdout  # Prints OTLP data to console
    service_name: "qtap"
    environment: "development"
```

Then check qtap logs:

```bash
docker logs qtap
```

## Best Practices

1. **Use TLS in production** - Always enable `tls.enabled: true` for production
2. **Store artifacts securely** - Use OTel object stores for unified pipelines, or S3 for separate storage
3. **Filter appropriately** - Use qtap's filters to avoid capturing unnecessary traffic
4. **Set resource attributes** - Use `service_name` and `environment` for filtering
5. **Monitor qtap itself** - Set up alerts on qtap's Prometheus metrics
6. **Use batch processing** - OTel Collector's batch processor reduces API calls
7. **Use connection.id for correlation** - Link connection events with artifact records using `connection.id`

## Configuration Quick Reference

**qtap.yaml:**

```yaml
services:
  event_stores:
    - type: otel
      endpoint: "localhost:4317"
      protocol: grpc
```

## Next Steps

* [Storage Configuration](/getting-started/qtap/configuration/storage-configuration.md) — OTel and S3 object storage options
* [Self-Hosted ClickStack Observability](/guides/qtap-guides/observability-and-integration/self-hosted-clickstack-observability.md) — Full setup with OTel events + objects
* [Set up Prometheus Metrics](/guides/qtap-guides/observability-and-integration/monitoring-qtap-with-prometheus-and-grafana.md) for qtap health monitoring
* [Configure Traffic Filters](/getting-started/qtap/configuration/traffic-capture-settings.md) to reduce noise

## Additional Resources

* [Storage Configuration Reference](/getting-started/qtap/configuration/storage-configuration.md)
* [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/)
* [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.qpoint.io/guides/qtap-guides/observability-and-integration/sending-qtap-events-to-opentelemetry.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.