CA Bundle Injection in Kubernetes

Introduction

This guide offers a systematic approach to troubleshoot and resolve issues with QPoint Operator Certificate Authority (CA) bundle injection in Docker containers running in Kubernetes. This is crucial when automatic CA bundle injection is not natively supported by QPoint for specific container images.

For an illustrative example, see our GPT4 demo (Python) or Artillery demo (Node.js), demonstrating a scenario where a CA environment variable needs to be defined in a Docker Container.

Natively Supported CA Bundle Paths

• /etc/ssl/certs/ca-certificates.crt Commonly Associated With: Debian-based distributions like Ubuntu. A standard path for CA certificates bundle.

• /etc/pki/tls/certs/ca-bundle.crt Commonly Associated With: Red Hat-based distributions including CentOS and Fedora. Part of the system's default setup for storing trusted CA certificates.

• /etc/ssl/ca-bundle.pem Commonly Associated With: Less common but found in certain Linux distributions or customized environments. Used in manually configured CA bundles.

• /etc/pki/tls/cacert.pem Commonly Associated With: Also found in Red Hat-based distributions, similar in use to ca-bundle.crt.

• /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem Commonly Associated With: Newer versions of Red Hat-based distributions. Part of the dynamic CA configuration system.

• /etc/ssl/cert.pem Commonly Associated With: Some Unix-like operating systems like FreeBSD and various Linux distributions. A more generic CA certificate path.

Symptoms of CA Bundle Injection Issues

• SSL/TLS Communication Errors: Including certificate verification failures or "unable to get local issuer certificate" messages.

• Application-Specific Errors: Failure to connect to external services using SSL/TLS.

• Log Messages: Indicating SSL/TLS certificate issues in containers or Kubernetes pods.

Identifying CA Bundle Injection Issues

1. Review Application Logs: Check for SSL/TLS-related error messages using kubectl logs <pod-name>.

2. Inspect Container Configuration: Examine the container's configuration and Dockerfile, looking for CA bundle path specifications.

3. Verify CA Bundle Path in the Container: Access the container's shell with kubectl exec -it <pod-name> -- /bin/bash and check the CA bundle paths.

4. Test SSL/TLS Connections from Within the Container: Use tools like curl or openssl to test external SSL/TLS connections.

5. Check QPoint Configuration and Logs: Look for indications of CA bundle handling issues in QPoint logs.

Fixing CA Bundle Injection Issues

1. Identify the Container Base Image: Understand the default paths and mechanisms for CA bundle storage in your base image.

2. Research Base Image Documentation: Look for information on SSL/TLS configurations and default CA bundle paths.

3. Inspect the Container's Dockerfile: Note any ENV instructions indicating the use of environment variables for CA certificates.

4. Examine the Application's Documentation: Some applications might allow specifying the CA bundle path through an environment variable.

Configuring Kubernetes Deployment

Modify the container specification in your Kubernetes deployment YAML using env to set the appropriate environment variable for your application's language.

Python Applications:

env:
  - name: REQUESTS_CA_BUNDLE
    value: "/etc/ssl/certs/ca-certificates.crt"

Node.js Applications:

env:
  - name: NODE_EXTRA_CA_CERTS
    value: "/etc/ssl/certs/ca-certificates.crt"

Ruby Applications:

env:
  - name: SSL_CERT_FILE
    value: "/etc/ssl/certs/ca-certificates.crt"

Go Applications:

env:
  - name: SSL_CERT_DIR
    value: "/etc/ssl/certs"
  - name: SSL_CERT_FILE
    value: "/etc/ssl/certs/ca-certificates.crt"

.NET Core Applications:

env:
  - name: SSL_CERT_FILE
    value: "/etc/ssl/certs/ca-certificates.crt"

Validating the Configuration

After deploying the updated configuration, check the container logs for SSL/TLS related errors and perform connectivity tests to services requiring SSL/TLS.