BPF Trace - Advanced Debugging

Qtap Version: v0.11.3+ Status: Advanced feature for syscall-level debugging

---

Overview

The --bpf-trace flag enables syscall-level tracing of network operations, providing deep visibility into how applications interact with the Linux kernel. This is an advanced debugging feature that shows:

• Individual syscall invocations (read, write, writev, recvfrom, accept4)

• File descriptor (FD) numbers for each operation

• Data transfer sizes

• TLS/SSL detection per-syscall

• Process IDs and executable paths

Use cases:

• Debugging complex proxy scenarios (e.g., HTTP→HTTPS reverse proxies)

• Understanding per-FD TLS state tracking issues

• Investigating missing or corrupted traffic captures

• Analyzing syscall-level data flow through applications

---

Syntax

--bpf-trace="mod:<module>,exe.contains:<executable_name>"

Parameters

Parameter	Description	Example
mod:<module>	BPF trace module to enable	mod:socket
exe.contains:<name>	Filter traces to executables containing this string	exe.contains:nginx

Available modules:

• socket - Socket lifecycle and syscall tracing ✅ WORKS

• openssl - OpenSSL library event tracking ✅ WORKS

Note: Use exe.contains (not bin.contains) for syscall filtering.

---

Basic Example

Configuration

docker run \
  --privileged --user 0:0 \
  --cap-add CAP_BPF --cap-add CAP_SYS_ADMIN \
  --pid host --network host \
  -v /sys:/sys \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $(pwd)/qtap.yaml:/app/config/qtap.yaml \
  -e TINI_SUBREAPER=1 \
  --ulimit memlock=-1 \
  us-docker.pkg.dev/qpoint-edge/public/qtap:v0 \
  --log-level=info \
  --log-encoding=console \
  --config=/app/config/qtap.yaml \
  --bpf-trace="mod:socket,exe.contains:nginx"

Expected Output

2025-10-23 00:38:43.341	INFO	eBPF trace	{"msg": "syscall/accept4", "caller": "syscall/accept4", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3}
2025-10-23 00:38:43.341	INFO	eBPF trace	{"msg": "syscall/recvfrom (init)", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "bytes": 81}
2025-10-23 00:38:43.341	INFO	eBPF trace	{"msg": "syscall/recvfrom", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "bytes": 81}
2025-10-23 00:38:43.341	INFO	eBPF trace	{"msg": "process_data (pre-protocol)", "caller": "syscall/recvfrom", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "direction": 0, "bytes": 81, "ssl": false, "protocol": 0}
2025-10-23 00:38:43.365	INFO	eBPF trace	{"msg": "syscall/write (init)", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "bytes": 1806}
2025-10-23 00:38:43.365	INFO	eBPF trace	{"msg": "syscall/write", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "bytes": 1806}
2025-10-23 00:38:43.365	INFO	eBPF trace	{"msg": "process_data (not ssl)", "caller": "syscall/write", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "direction": 1, "bytes": 1806, "ssl": false}
2025-10-23 00:38:43.391	INFO	eBPF trace	{"msg": "syscall/read (init)", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "bytes": 5}
2025-10-23 00:38:43.391	INFO	eBPF trace	{"msg": "syscall/read", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "bytes": 5}
2025-10-23 00:38:43.391	INFO	eBPF trace	{"msg": "process_data (not ssl)", "caller": "syscall/read", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 11, "direction": 0, "bytes": 5, "ssl": false}
2025-10-23 00:38:43.493	INFO	eBPF trace	{"msg": "syscall/writev (init)", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "bytes": 941}
2025-10-23 00:38:43.493	INFO	eBPF trace	{"msg": "syscall/writev", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "bytes": 941}
2025-10-23 00:38:43.493	INFO	eBPF trace	{"msg": "process_data (not ssl)", "caller": "syscall/writev", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3, "direction": 1, "bytes": 941, "ssl": false}

---

Understanding the Output

Syscall Types

Syscall	Description	Typical Usage
accept4	Accept incoming connection	Server accepting new client
recvfrom	Receive data from socket	Reading HTTP request
read	Read data from FD	Reading from SSL/TLS socket
write	Write data to FD	Writing to SSL/TLS socket
writev	Vectored write (scatter-gather)	Writing HTTP response with headers+body

Event Phases

Each syscall typically generates two events:

1. (init) event - Syscall entry (before execution)

2. Completion event - Syscall exit (after execution, with actual byte counts)

Field Reference

{
  "msg": "syscall/writev",        // Syscall name
  "caller": "syscall/writev",     // Source of the call
  "pid": 139929,                  // Process ID
  "exe": "/usr/sbin/nginx",       // Executable path
  "fd": 3,                        // File descriptor number
  "direction": 1,                 // 0=ingress (read), 1=egress (write)
  "bytes": 941,                   // Number of bytes transferred
  "ssl": false,                   // Whether SSL/TLS detected on this FD
  "protocol": 0,                  // Protocol detection status
  "open": false                   // Whether connection info is available
}

process_data Messages

These appear after syscall completion and show Qtap's data processing decisions:

• process_data (pre-protocol) - Data received before protocol detection

• process_data (not ssl) - Data on plaintext socket (HTTP)

• process_data (conn_info = NULL) - Socket not tracked (e.g., log files)

• process_data (conn_info->is_open = false) - Connection already closed

---

Real-World Example: Nginx HTTP→HTTPS Reverse Proxy

Scenario

Nginx reverse proxy configuration:

• Client → Nginx: HTTP on port 8000

• Nginx → Backend: HTTPS to api.treatmyocd.com

Traffic Flow Visualization

Client (curl)  →  Nginx (PID 139929)  →  Backend HTTPS
    ↓                    ↓                      ↓
  fd 3 (HTTP)        fd 11 (HTTPS)        api.treatmyocd.com:443

Step-by-Step Syscall Trace

1. Accept Client Connection

{"msg": "syscall/accept4", "pid": 139929, "exe": "/usr/sbin/nginx", "fd": 3}

Nginx accepts incoming HTTP connection from curl on fd 3.

2. Read HTTP Request from Client

{"msg": "syscall/recvfrom", "pid": 139929, "fd": 3, "bytes": 81, "direction": 0, "ssl": false}

Nginx reads 81 bytes of HTTP request from fd 3 (client socket).

3. Write HTTPS Request to Backend

{"msg": "syscall/write", "pid": 139929, "fd": 11, "bytes": 1806, "direction": 1, "ssl": false}

Nginx writes 1806 bytes to fd 11 (backend HTTPS connection).

Note: "ssl": false appears here even though fd 11 is an HTTPS connection. This indicates Qtap is tracking TLS state at the per-process level rather than per-FD.

4. Read HTTPS Response from Backend

{"msg": "syscall/read", "pid": 139929, "fd": 11, "bytes": 1210, "direction": 0, "ssl": false}
{"msg": "syscall/read", "pid": 139929, "fd": 11, "bytes": 2695, "direction": 0, "ssl": false}
{"msg": "syscall/read", "pid": 139929, "fd": 11, "bytes": 1367, "direction": 0, "ssl": false}

Nginx reads multiple chunks from fd 11 (backend response).

5. Write HTTP Response to Client

{"msg": "syscall/writev", "pid": 139929, "fd": 3, "bytes": 941, "direction": 1, "ssl": false}

Nginx writes 941 bytes back to fd 3 (client) using writev.

Critical: This is where the nginx HTTP→HTTPS bug would manifest. If Qtap incorrectly marks the entire process as "has SSL" after seeing fd 11's TLS traffic, it might discard this writev data expecting SSL_write instead.

6. Write Access Log

{"msg": "syscall/write", "pid": 139929, "fd": 4, "bytes": 89, "open": false}
{"msg": "process_data (conn_info = NULL)", "caller": "syscall/write", "fd": 4}

Nginx writes 89 bytes to fd 4 (access log file). Qtap shows conn_info = NULL because fd 4 is not a network socket.

---

Filtering Options

Filter by Executable Name

# Filter to nginx processes
--bpf-trace="mod:socket,exe.contains:nginx"

# Filter to curl processes
--bpf-trace="mod:socket,exe.contains:curl"

# Filter to any process with "python" in the path
--bpf-trace="mod:socket,exe.contains:python"

How exe.contains works:

• Matches substring anywhere in the executable path

• Example: exe.contains:nginx matches both /usr/sbin/nginx and /custom/path/nginx-debug

No Filter (All Processes)

# Trace all processes (VERY VERBOSE!)
--bpf-trace="mod:socket"

Warning: Tracing all processes generates massive log output and may impact system performance. Always use exe.contains filters in production.

---

Log Level Requirements

BPF trace output varies by log level:

Log Level	BPF Trace Output
warn	No syscall traces (silent)
info	Full syscall traces ✅ RECOMMENDED
debug	Full syscall traces + additional debug logs

Recommended configuration:

--log-level=info --log-encoding=console

---

Debugging Use Cases

1. Missing Response Data

Symptom: Qtap captures request but shows 0 bytes received in response.

Diagnosis with BPF trace:

--bpf-trace="mod:socket,exe.contains:nginx"

Look for:

1. writev or write syscalls that should contain response data

2. Check "ssl": false vs expected TLS state per-FD

3. Look for process_data messages showing why data was discarded

2. Per-FD TLS State Issues

Symptom: Mixed HTTP/HTTPS connections in same process cause capture failures.

Diagnosis:

Compare syscall traces for different file descriptors:

• Check if "ssl": true/false is accurate per-FD

• Look for process_data (not ssl) when SSL is expected (or vice versa)

Example buggy behavior:

// Correct: fd 3 is HTTP
{"msg": "syscall/writev", "fd": 3, "ssl": false}

// INCORRECT: fd 11 is HTTPS but shows ssl: false
{"msg": "syscall/write", "fd": 11, "ssl": false}

3. Understanding Data Flow

Trace complete request lifecycle:

docker logs qtap-container 2>&1 | grep "eBPF trace" | grep "pid: 139929"

Group by file descriptor to see data flow:

# Client-facing socket (fd 3)
grep "fd\": 3" | grep -E "recvfrom|writev"

# Backend socket (fd 11)
grep "fd\": 11" | grep -E "write|read"

---

Common Patterns

HTTP Server (Single FD)

// 1. Accept connection
{"msg": "syscall/accept4", "fd": 5}

// 2. Read HTTP request
{"msg": "syscall/recvfrom", "fd": 5, "bytes": 142, "direction": 0}

// 3. Write HTTP response
{"msg": "syscall/writev", "fd": 5, "bytes": 1024, "direction": 1}

HTTPS Client (Single FD with SSL)

// 1. Connect to server
{"msg": "syscall/connect", "fd": 8}

// 2. Write HTTPS request
{"msg": "syscall/write", "fd": 8, "bytes": 256, "ssl": true}

// 3. Read HTTPS response
{"msg": "syscall/read", "fd": 8, "bytes": 4096, "ssl": true}

Reverse Proxy (Two FDs - HTTP + HTTPS)

// Client-facing (HTTP)
{"msg": "syscall/recvfrom", "fd": 3, "ssl": false}  // Read request
{"msg": "syscall/writev", "fd": 3, "ssl": false}    // Write response

// Backend-facing (HTTPS)
{"msg": "syscall/write", "fd": 11, "ssl": true}     // Write request
{"msg": "syscall/read", "fd": 11, "ssl": true}      // Read response

---

Troubleshooting

No "eBPF trace" Output

Check 1: Log level

# Verify INFO or DEBUG level
docker logs qtap-container 2>&1 | head -20 | grep "Starting Qtap"

Check 2: Filter matches

# Verify executable name
docker exec nginx-container which nginx
# Should match exe.contains filter

Check 3: Traffic generated

# Ensure traffic is actually hitting the filtered process
curl http://localhost:8000/get

Check 4: Correct filter syntax

# CORRECT
--bpf-trace="mod:socket,exe.contains:nginx"

# WRONG
--bpf-trace="mod:socket,bin.contains:nginx"  # Wrong prefix!

Too Much Output

Problem: Tracing all processes floods logs.

Solution: Add executable filter:

# Before (too verbose)
--bpf-trace="mod:socket"

# After (filtered)
--bpf-trace="mod:socket,exe.contains:nginx"

Syscall Not Appearing

Some syscalls may not appear if:

1. Different syscall variant used - e.g., readv instead of read

2. Buffered I/O - Application uses buffering, syscalls occur later

3. Connection pooling - FD reused, syscalls mixed with other requests

---

Performance Considerations

BPF trace adds overhead to every syscall:

Traffic Level	Overhead	Recommendation
< 10 req/sec	Negligible	Safe for production debugging
10-100 req/sec	Low (~5%)	Use with caution, monitor CPU
> 100 req/sec	Moderate (10-20%)	Only use in isolated environments

Best practices:

• Use exe.contains filters to limit scope

• Enable only during active debugging sessions

• Monitor disk I/O (logs can grow rapidly)

• Use log rotation if enabled long-term

---

Comparison: Regular Logs vs BPF Trace

Regular Qtap Logs (--log-level=info)

INFO	HTTP Transaction

Metadata:
  Direction: egress-external
  Bytes Sent: 78
  Bytes Received: 941

Request:
  Method: GET
  URL: http://httpbin.org/get

Response:
  Status: 302

Pros: Human-readable, concise, shows final results Cons: No syscall details, can't debug per-FD issues

BPF Trace (--bpf-trace="mod:socket,exe.contains:nginx")

{"msg": "syscall/recvfrom", "pid": 139929, "fd": 3, "bytes": 81, "direction": 0, "ssl": false}
{"msg": "syscall/write", "pid": 139929, "fd": 11, "bytes": 1806, "direction": 1, "ssl": false}
{"msg": "syscall/read", "pid": 139929, "fd": 11, "bytes": 1210, "direction": 0, "ssl": false}
{"msg": "syscall/writev", "pid": 139929, "fd": 3, "bytes": 941, "direction": 1, "ssl": false}

Pros: Syscall-level detail, per-FD visibility, shows TLS state decisions Cons: Verbose, requires interpretation, not user-friendly

When to use BPF trace:

• Debugging complex proxy scenarios

• Investigating missing captures

• Understanding per-FD TLS state tracking

• Analyzing syscall-level data flow

When to use regular logs:

• Normal operation

• User-facing traffic analysis

• Compliance/audit logging

---

Related Configuration

BPF trace works alongside standard Qtap configuration:

# qtap.yaml
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  capture_all:
    plugins:
      - type: http_capture
        config:
          level: full           # Capture full HTTP data
          format: json

tap:
  direction: all                # Capture all directions
  ignore_loopback: false        # Include localhost traffic
  http:
    stack: capture_all

Run with:

docker run ... \
  --config=/app/config/qtap.yaml \
  --log-level=info \
  --bpf-trace="mod:socket,exe.contains:nginx"

---

Known Limitations

1. No SSL_write/SSL_read visibility BPF trace shows syscalls (read/write) but not OpenSSL library calls (SSL_read/SSL_write).

2. Per-process TLS state The "ssl": true/false field may reflect process-level state rather than per-FD state in some qtap versions.

3. Filter syntax constraints

   • Must use exe.contains (not bin.contains)

   • Only substring matching (no regex)

4. Log volume Without filters, trace output can be 100x larger than regular logs.

---

Summary

Feature	Value
Flag	--bpf-trace="mod:socket,exe.contains:nginx"
Log Level	info or debug
Output	Syscall-level traces (read, write, writev, accept4, recvfrom)
Filtering	By executable name substring
Use Cases	Debugging proxies, per-FD TLS issues, missing captures
Performance	Low overhead with filters, high without

Key Takeaway: BPF trace is the most powerful debugging tool for understanding Qtap's eBPF-level behavior, especially for complex scenarios like mixed HTTP/HTTPS proxies.