Kubernetes Manifest
This guide covers how to deploy Qtap, an eBPF-based traffic monitoring agent, in Kubernetes environments.
Prerequisites
Kubernetes cluster on Linux hosts with supported kernel (5.10+)
kubectl
Helm
Generating Base Kubernetes Manifest
If you'd like to use the Helm chart as a base for building a Kubernetes manifest, you can do so with the following instructions:
helm repo add qpoint https://helm.qpoint.io/
helm template qtap qpoint/qtap > qtap-base.yaml
This command generates a base manifest file named qtap-base.yaml
.
Customizing Qtap's Configuration
If you wish to supply your own Qtap configuration file, first write it to a file named qtap-config.yaml
. Refer to the Configuration page for further information on the configuration file structure.
Then, instruct helm to use it like so:
helm template qtap qpoint/qtap \
--set-file config=qtap-config.yaml \
> qtap-base.yaml
To see additional helm chart configuration options:
helm show values qpoint/qtap
Deploying Qtap
Create a namespace and apply the modified manifest to your cluster:
kubectl create ns qpoint
kubectl apply -f qtap-base.yaml -n qpoint
Verifying the Deployment
To verify that Qtap is running:
kubectl get pods -n qpoint
You should see pods named qtap-xxxx
in the Running state.
To check the deployment logs:
kubectl logs -n qpoint pod qtap-xxxx
Uninstalling Qtap
To uninstall Qtap:
kubectl delete -f qtap-base.yaml -n qpoint
Understanding the Base Manifest
The base manifest creates several Kubernetes resources:
ServiceAccount: Provides an identity for the Qtap pods
ConfigMap: Stores the Qtap configuration
DaemonSet: Ensures Qtap runs on every node in the cluster
Key Components in the DaemonSet
The DaemonSet specification includes several important settings:
Host Access: Uses
hostPID: true
andhostNetwork: true
to access the host's process namespace and networkSecurity Context: Requires privileged access and specific capabilities (
CAP_BPF
,CAP_SYS_ADMIN
) for eBPF operationsVolume Mounts:
/sys
: Access to the host's system directories (required for eBPF)/run/containerd/containerd.sock
: Access to the container runtime socket (for rich container attribution). Optional.Configuration file
Probes: Health checks to ensure the Qtap pod is running correctly
Resource Limits: Controls how much CPU and memory Qtap can use
Example
Here's a complete deployment example with an external object store:
apiVersion: v1
kind: ServiceAccount
metadata:
name: qtap
labels:
app.kubernetes.io/name: qtap
app.kubernetes.io/instance: qtap
app.kubernetes.io/version: "v0.9.3"
automountServiceAccountToken: true
---
apiVersion: v1
kind: ConfigMap
metadata:
name: qtap-config
labels:
app.kubernetes.io/name: qtap
app.kubernetes.io/instance: qtap
app.kubernetes.io/version: "v0.9.3"
data:
tap-config.yaml: |
version: 2
services:
event_stores:
- id: console_stdout
type: stdout
object_stores:
- id: minio
type: s3
endpoint: minio.storage.svc.cluster.local:9000
bucket: qpoint
region: us-east-1
access_url: http://minio.storage.svc.cluster.local:9000/{{BUCKET}}/{{DIGEST}}
insecure: true
access_key:
type: env
value: S3_ACCESS_KEY
secret_key:
type: env
value: S3_SECRET_KEY
stacks:
default_stack:
plugins:
- type: debug
config:
mode: summary
- type: detect_errors
config:
rules:
- name: "All Errors"
trigger_status_codes:
- '4xx'
- '5xx'
only_categories:
- app
report_as_issue: true
record_req_headers: true
record_req_body: true
record_res_headers: true
record_res_body: true
tap:
direction: egress
ignore_loopback: true
audit_include_dns: true
http:
stack: default_stack
filters:
groups:
- eks
- kubernetes
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: qtap
labels:
app.kubernetes.io/name: qtap
app.kubernetes.io/instance: qtap
app.kubernetes.io/version: "v0.9.3"
spec:
selector:
matchLabels:
app.kubernetes.io/name: qtap
app.kubernetes.io/instance: qtap
template:
metadata:
labels:
app.kubernetes.io/name: qtap
app.kubernetes.io/instance: qtap
app.kubernetes.io/version: "v0.9.3"
spec:
hostPID: true
hostNetwork: true
serviceAccountName: qtap
securityContext:
null
containers:
- name: qtap
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- CAP_BPF
- CAP_SYS_ADMIN
privileged: true
readOnlyRootFilesystem: false
runAsGroup: 0
runAsNonRoot: false
runAsUser: 0
image: "us-docker.pkg.dev/qpoint-edge/public/qtap:v0"
imagePullPolicy: IfNotPresent
args: []
env:
- name: QPOINT_CONFIG
value: "/app/tap-config.yaml"
- name: S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: minio-credentials
key: access-key
- name: S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: minio-credentials
key: secret-key
- name: STATUS_LISTEN
value: "0.0.0.0:10001"
- name: LOG_LEVEL
value: "info"
- name: LOG_ENCODING
value: "json"
- name: TINI_SUBREAPER
value: "1"
ports:
- name: status
containerPort: 10001
protocol: TCP
startupProbe:
httpGet:
path: /readyz
port: status
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 20
readinessProbe:
httpGet:
path: /readyz
port: status
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 1
livenessProbe:
httpGet:
path: /healthz
port: status
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 3
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: config-volume
mountPath: /app/tap-config.yaml
subPath: tap-config.yaml
- mountPath: /sys
name: sys
readOnly: true
- name: containerd-socket
mountPath: /run/containerd/containerd.sock
volumes:
- name: config-volume
configMap:
name: qtap-config
items:
- key: tap-config.yaml
path: tap-config.yaml
- hostPath:
path: /sys
type: Directory
name: sys
- name: containerd-socket
hostPath:
path: /run/containerd/containerd.sock
type: Socket
Important Notes
Privileged Access: The Qtap pod requires privileged access for eBPF operations. Ensure your cluster's security policies allow this.
Token Security: For cloud-connected mode, keep your registration token secure and do not share it in public repositories.
Configuration Format: For local mode, ensure your configuration is correctly formatted and contains all necessary settings.
Host Access: The deployment mounts the host's
/sys
directory and container runtime socket. Ensure this is allowed in your cluster.Resource Management: You may need to adjust resource requests and limits based on your cluster's capacity and Qtap's workload.
DaemonSet Deployment: This deployment uses a DaemonSet to ensure Qtap runs on every node in your cluster. Adjust if this is not your intended behavior.
Common Customizations
The generated manifest provides a starting point. You may need to customize various aspects such as:
Resource limits and requests: Adjust based on your workload and node capacity
Node selectors or tolerations: Target specific nodes or allow scheduling despite taints
Environment variables: Add additional configuration or credentials
Volume mounts: Access additional host resources if needed
Security contexts: Adjust permissions based on your security requirements
Always review and test your modifications in a non-production environment before deploying to production.
Troubleshooting
If you encounter issues with your Qtap deployment, check the following:
Pod Status: Check if the pods are running
kubectl get pods -n qpoint
Pod Logs: Examine the logs for error messages
kubectl logs -n qpoint ds qtap # or, for a specific pod kubectl logs -n qpoint pod qtap-xxxx
Configuration: Verify your configuration is correctly formatted
kubectl describe configmap -n qpoint qtap-config
Permissions: Ensure the pod has the necessary permissions
kubectl describe pod -n qpoint qtap-xxxx
Kernel Support: Verify your nodes are running a supported kernel version (5.10+)
kubectl debug node/node-name -it --image=ubuntu uname -r
Last updated