# Production Debugging with HTTPS Visibility

Get HTTPS traffic visibility in 30 seconds for urgent production debugging - no permanent installation needed.

## Who This Is For

**Use this guide when you:**

* Have a production issue RIGHT NOW that needs immediate investigation
* Need to see inside HTTPS traffic without certificates or proxies
* Want to debug authentication failures (401/403), API errors, or rate limiting
* Need temporary visibility without permanent installation

**This IS for:**

* Emergency troubleshooting and incident response
* Short-term traffic capture (hours to days)
* One-off debugging sessions
* Proof that Qtap works in your environment

**This is NOT for:**

* Long-term production monitoring (use [Complete Guide](https://docs.qpoint.io/guides/qtap-guides/getting-started/getting-started-complete-guide) for that)
* Selective capture with complex rules (use [Complete Guide](https://docs.qpoint.io/guides/qtap-guides/getting-started/getting-started-complete-guide) for rulekit)
* Multi-environment visibility (consider [Qplane](https://docs.qpoint.io/guides/qplane-guides/poc-kick-off-guide))

**Time to results:** 30 seconds (demo mode) or 5 minutes (full install)

> **🔒 Your Data Stays Local**
>
> This guide uses standalone Qtap with stdout output—**all captured traffic stays on your server**. No data is sent to Qpoint or any external service. Everything you capture remains in your terminal or log files under your control.
>
> Want centralized visibility across a fleet with alerting? That's [Qplane](https://docs.qpoint.io/guides/qplane-guides/poc-kick-off-guide) (our cloud control plane), but this guide keeps everything local and private.

***

## Temporary Installation

**The fastest way to see HTTPS traffic.** No installation, no configuration file—just one command that downloads qtap to `/tmp` and starts capturing immediately.

### One-Line Demo

SSH into your production server and run:

```bash
curl -s https://get.qpoint.io/demo | sudo LOG_LEVEL=warn sh
```

> **Prefer not to pipe curl to sudo?** We get it. See [manual installation instructions](https://docs.qpoint.io/getting-started/qtap/installation/linux-binary#manual-installation) to download and verify the binary yourself before running.

Qtap will:

1. Download the latest qtap binary to `/tmp` (no permanent installation)
2. Start capturing all HTTP/HTTPS traffic with full visibility
3. Display readable console output in real-time (access\_logs plugin, full mode)

### What You'll See

Captured traffic appears immediately:

{% hint style="success" %}
**Visual Alternative**: For interactive inspection with a Chrome DevTools-style interface, see the [DevTools UI](https://docs.qpoint.io/getting-started/qtap/configuration/devtools) guide. Perfect for exploring traffic visually or troubleshooting via SSH tunnel from your laptop.
{% endhint %}

```
=============================================================================================================
■ /usr/bin/curl → GET https://httpbin.org/get 200 OK
=============================================================================================================

------------------ META ------------------
PID: 12345
Exe: /usr/bin/curl
Direction: egress-external
Bytes Sent: 41
Bytes Received: 395

------------------ REQUEST ------------------
GET httpbin.org http2
:method: GET
:path: /get
:scheme: https
User-Agent: curl/8.10.1
Accept: */*
:authority: httpbin.org

------------------ REQUEST BODY ------------------
(empty)

------------------ RESPONSE ------------------
200 OK
Content-Type: application/json
Content-Length: 255
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
:status: 200
Date: Wed, 29 Oct 2025 19:51:23 GMT

------------------ RESPONSE BODY ------------------
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.10.1"
  },
  "origin": "73.71.138.108",
  "url": "https://httpbin.org/get"
}
```

### Stop the Demo

Press **Ctrl+C** to stop qtap. The binary remains in `/tmp` and will be cleaned up on system reboot.

**Pro tip:** The demo script uses a default configuration (access\_logs, full mode, all traffic). For production-safe filtering and custom rules, continue to the sections below.

***

## Full Installation

This option gives you full control with custom configuration. Perfect for production-safe debugging with filters and rules.

### Prerequisites

* SSH access to production server (Linux with kernel 5.10+)
* Root/sudo privileges
* Running application to debug

### Step 1: Create Minimal Configuration

SSH into your production server and create `/tmp/qtap-debug.yaml`:

```yaml
# Minimal production debugging configuration
# Captures all HTTP traffic with readable console output
version: 2

services:
  event_stores:
    - type: stdout  # Connection metadata to console
  object_stores:
    - type: stdout  # Request/response bodies to console

stacks:
  debug:
    plugins:
      - type: access_logs
        config:
          mode: full      # (summary|details|full) - Capture everything
          format: console # (console|json) - Human-readable output

tap:
  direction: all          # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true   # Skip localhost traffic
  audit_include_dns: false  # Skip DNS queries
  http:
    stack: debug
```

### Step 2: Install and Start Qtap

Install qtap with a single command:

```bash
# Install qtap
curl -s https://get.qpoint.io/install | sudo sh

# Verify installation
qtap --version
```

> **Prefer not to pipe curl to sudo?** See [manual installation instructions](https://docs.qpoint.io/getting-started/qtap/installation/linux-binary#manual-installation) to download, extract, and verify the binary yourself.

Start qtap (runs in foreground, shows captured traffic in real-time):

```bash
sudo qtap --config=/tmp/qtap-debug.yaml --log-level=info --log-encoding=console | tee /tmp/qtap-output.log
```

Qtap displays captured HTTPS traffic in real-time. Output is also saved to `/tmp/qtap-output.log` for later analysis.

### Step 3: Generate Test Traffic

**Open a second SSH session** to your server (keep qtap running in the first session), then generate test traffic:

```bash
# Test authentication failure (returns 401)
curl https://httpbin.org/status/401

# Test successful request (returns 200)
curl https://httpbin.org/get

# Test error response (returns 500)
curl https://httpbin.org/status/500
```

Or simply wait for your production application to generate traffic—you'll see it appear in real-time in your first SSH session.

### Step 4: What You'll See

**Readable console output** (example 401 error from httpbin):

```
=========================================================================
■ /usr/bin/curl → GET https://httpbin.org/status/401 401 Unauthorized
=========================================================================

------------------ META ------------------
PID: 12345
Exe: /usr/bin/curl
Direction: egress-external
Bytes Sent: 54
Bytes Received: 0

------------------ REQUEST ------------------
GET httpbin.org http2
User-Agent: curl/8.10.1
Accept: */*
:authority: httpbin.org
:method: GET
:path: /status/401
:scheme: https

------------------ RESPONSE ------------------
401 Unauthorized
:status: 401
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Wed, 29 Oct 2025 19:30:00 GMT
Server: gunicorn/19.9.0

------------------ RESPONSE BODY ------------------
(empty)
```

**Key indicators of successful capture:**

* ✅ Process identified (`/usr/bin/curl`)
* ✅ Request headers visible (including Authorization token!)
* ✅ Response body readable (no base64 encoding)
* ✅ Complete HTTPS traffic despite encryption

### Step 5: Cleanup

When debugging is complete, press **Ctrl+C** in the qtap session to stop it, then:

```bash
# Remove configuration and logs
rm /tmp/qtap-debug.yaml /tmp/qtap-output.log
```

***

## Production Debugging Configuration

The quick start captures **everything** at full detail. For production use, this single configuration works for all common debugging scenarios—just adjust what you look for in the output.

> **Need advanced capture options?** This guide uses the `access_logs` plugin for simplicity. For selective capture with rules, custom filtering, or other plugins (like `http_capture`), see [Traffic Processing with Plugins](https://docs.qpoint.io/getting-started/qtap/configuration/traffic-processing-with-plugins).

### Base Configuration

Create `/tmp/qtap-debug.yaml`:

```yaml
# Production debugging configuration
# Captures all HTTP/HTTPS traffic with full details
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  debug:
    plugins:
      - type: access_logs
        config:
          mode: full        # (summary|details|full) - Full capture for debugging
          format: console   # (console|json) - Readable output

tap:
  direction: all            # (egress|egress-external|egress-internal|ingress|all)
  ignore_loopback: true     # Skip localhost traffic
  audit_include_dns: false  # Skip DNS queries
  http:
    stack: debug
```

**Start capturing**:

```bash
sudo qtap --config=/tmp/qtap-debug.yaml --log-level=info --log-encoding=console | tee /tmp/qtap-output.log
```

Now use a second SSH session to generate traffic (or just wait for your production app to make or receive requests).

***

## Common Debugging Scenarios

Same config, different things to look for in the captured output.

### Scenario 1: Authentication Failures

**Problem**: Third-party API returning 401/403 errors

**Generate test traffic**:

```bash
curl https://api.stripe.com/v1/customers -H "Authorization: Bearer sk_test_invalid"
curl https://httpbin.org/status/403
```

**What to look for in qtap output**:

* **Status codes**: `401 Unauthorized` or `403 Forbidden`
* **Authorization header**: Check if token is present and formatted correctly
* **Response body**: Error message often explains the issue (expired token, wrong scope, etc.)
* **Process**: Which service is making the failing request (`exe: /usr/bin/python3`, container name)

***

### Scenario 2: Slow API Responses

**Problem**: External APIs taking too long to respond

**What to look for in qtap output**:

* **Duration**: Response time shown in milliseconds at top of each capture
* **Bytes Sent/Received**: Large payloads can cause slowness
* **Process**: Which service is making slow requests
* **Host**: Which external API is slow

**Pro tip**: Use `mode: summary` instead of `full` if you don't need bodies—just timing data.

***

### Scenario 3: Rate Limiting

**Problem**: Hitting API rate limits (429 responses)

**Generate test traffic**:

```bash
curl https://httpbin.org/status/429
```

**What to look for in qtap output**:

* **Status code**: `429 Too Many Requests`
* **Rate-limit headers**: `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After`
* **Request frequency**: Count how many requests to the same host in a short time window
* **Process**: Which service is making too many requests

***

### Scenario 4: External API Errors

**Problem**: Third-party API integration suddenly failing

**What to look for in qtap output**:

* **Status codes**: `500`, `502`, `503`, `504` (server errors)
* **Error response bodies**: Often contain specific error messages or codes
* **Request bodies**: Verify payload format is correct
* **Headers**: Check required headers are present (`Content-Type`, API version headers)
* **Process**: Which service is affected

***

## Reducing Noise with Filters

In busy production environments, you'll capture a lot of traffic. Use filters to focus on what matters.

### Filter by Process

Ignore noisy health checks, monitoring agents, or specific services:

```yaml
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  debug:
    plugins:
      - type: access_logs
        config:
          mode: full
          format: console

tap:
  direction: all
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: debug

  filters:
    groups:
      - qpoint              # Don't capture qtap's own traffic
    custom:
      - exe: /usr/bin/health-check
        strategy: exact     # Exact path match
      - exe: /usr/bin/
        strategy: prefix    # All executables in /usr/bin/
      - exe: .*node.*
        strategy: regex     # Any path matching pattern
```

**Common filters**:

* Health checks: `/usr/bin/health-check`, `/usr/bin/curl` (if used for monitoring)
* Monitoring agents: `.*datadog.*`, `.*newrelic.*`
* Internal tools: `/usr/local/bin/metrics-collector`

### Filter by Domain

Capture only specific domains by setting the default stack to `none` and overriding for specific endpoints:

```yaml
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  debug:
    plugins:
      - type: access_logs
        config:
          mode: full
          format: console

tap:
  direction: all
  ignore_loopback: true
  audit_include_dns: false

  # Default: capture nothing
  http:
    stack: none

  # Override: capture only these specific domains
  endpoints:
    - domain: 'api.stripe.com'
      http:
        stack: debug
    - domain: 'api.github.com'
      http:
        stack: debug
```

**How it works**: The default `http.stack` applies to all traffic. Specific domains under `endpoints` override the default. With `stack: none` as the default, only the listed domains are captured.

**When to use domain filtering**:

* Debugging a specific third-party integration
* High-volume environments where you need to focus
* Investigating issues with a particular external service

### Adjust Capture Level

Start with less detail, increase as needed by changing the `mode` value in your config:

```yaml
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

stacks:
  debug:
    plugins:
      - type: access_logs
        config:
          mode: summary     # Change this: summary | details | full
          format: console

tap:
  direction: all
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: debug
```

**Available modes**:

* `summary`: Method, URL, status, timing - no headers or bodies (minimal overhead)
* `details`: Summary + all headers - no bodies (auth/rate-limit debugging)
* `full`: Everything including headers and bodies (complete debugging)

**Tip**: Start with `summary`, look at the list of requests, then restart qtap with `full` mode to dig into specific issues.

### Filter by Response Code

Focus on errors, authentication failures, or rate limiting by capturing only specific HTTP status codes:

```yaml
version: 2

services:
  event_stores:
    - type: stdout
  object_stores:
    - type: stdout

rulekit:
  macros:
    - name: is_error
      expr: http.res.status >= 400

stacks:
  errors_only:
    plugins:
      - type: access_logs
        config:
          mode: summary     # Start with summary for all traffic
          format: console
          rules:
            - name: "Capture errors at full detail"
              expr: is_error()
              mode: full    # Upgrade to full mode for errors

tap:
  direction: all
  ignore_loopback: true
  audit_include_dns: false
  http:
    stack: errors_only
```

**How it works**: All traffic is captured at `summary` level (minimal overhead), but requests with status codes ≥ 400 are automatically upgraded to `full` mode with complete headers and bodies.

**Common response code filters**:

* Authentication issues: `http.res.status == 401 || http.res.status == 403`
* Rate limiting: `http.res.status == 429`
* Client errors: `http.res.status >= 400 && http.res.status < 500`
* Server errors: `http.res.status >= 500`

**When to use response code filtering**:

* High-volume production debugging where you only care about failures
* Authentication troubleshooting (capture 401/403 fully, ignore successful 200s)
* Rate limit investigation (capture 429 responses with all headers)
* Error diagnosis without capturing thousands of successful requests

***

## Understanding Captured HTTPS Data

Qtap's eBPF hooks capture traffic at the kernel level **before TLS encryption**, giving you complete visibility into HTTPS traffic without proxies or certificate management.

### What You Can See

**Request details**:

* HTTP method (GET, POST, PUT, DELETE, etc.)
* Full URL (scheme, host, path, query parameters)
* All request headers (Authorization, Content-Type, custom headers)
* Request body (when mode is `full`)

**Response details**:

* Status code (200, 401, 500, etc.)
* All response headers (Content-Type, rate-limit headers, cookies)
* Response body (when mode is `full`)
* Response timing (duration in milliseconds)

**Process attribution** (unique to eBPF-based capture):

* Executable path (`/usr/bin/curl`, `/usr/bin/python3`, etc.)
* Process ID (PID)
* Container name (Docker/Kubernetes)
* Container/pod labels

**TLS information**:

* TLS detected (`is_tls: true`)
* TLS library used (`tlsProbeTypesDetected: ["openssl"]`)
* Protocol version (`http1`, `http2`)

### Capture Levels Explained

| Level     | What's Captured                                | Use Case                                |
| --------- | ---------------------------------------------- | --------------------------------------- |
| `summary` | Method, URL, status, timing, process info      | High-level monitoring, minimal overhead |
| `details` | Summary + all headers (no bodies)              | Debug headers, auth tokens, rate limits |
| `full`    | Everything (headers + request/response bodies) | Complete debugging, error diagnosis     |

### Example Captured Request (Full Mode)

This is what you'll see with `mode: full` and `format: console`:

```
=================================================================================
■ /usr/local/bin/python3 → POST https://api.stripe.com/v1/charges 402 Payment Required
=================================================================================

------------------ META ------------------
PID: 54321
Exe: /usr/local/bin/python3
Container: payment-service
Direction: egress-external
Bytes Sent: 156
Bytes Received: 287
Duration: 187ms

------------------ REQUEST ------------------
POST api.stripe.com http2
:method: POST
:path: /v1/charges
:scheme: https
:authority: api.stripe.com
Authorization: Bearer sk_test_...
Content-Type: application/x-www-form-urlencoded
Stripe-Version: 2023-10-16

------------------ REQUEST BODY ------------------
amount=2000&currency=usd&source=tok_visa

------------------ RESPONSE ------------------
402 Payment Required
:status: 402
Content-Type: application/json
Request-Id: req_abc123
Stripe-Version: 2023-10-16

------------------ RESPONSE BODY ------------------
{
  "error": {
    "type": "card_error",
    "code": "card_declined",
    "message": "Your card was declined."
  }
}
```

***

## Production Safety

### Performance

* Start with `mode: summary` for minimal overhead
* Use process or domain filters to reduce volume
* Typical overhead: 1-3% CPU

### Sensitive Data

Captured data may contain API keys and PII. For quick debugging:

* Data stays on your server in `/tmp`
* Remove logs when done: `rm /tmp/qtap-*.log`

### Cleanup

```bash
# Stop qtap (Ctrl+C in qtap session)
rm /tmp/qtap-*.yaml /tmp/qtap-*.log
```

**Need help?** For additional resources:

* [Traffic Processing with Plugins](https://docs.qpoint.io/getting-started/qtap/configuration/traffic-processing-with-plugins) - Complete plugin and rules syntax
* [Linux Kernel Lockdown for eBPF Applications](https://docs.qpoint.io/troubleshooting/linux-kernel-lockdown-for-ebpf-applications) - Permission issues
* Contact <support@qpoint.io>