Introduction
In Kubernetes, securing communications between different components is essential for maintaining the integrity and confidentiality of your applications. Certificates play a pivotal role in this security ecosystem, enabling encrypted communication, and authentication, and ensuring that data remains tamper-proof. However, as Kubernetes environments grow in complexity, tracking which certificates are used by specific deployments can become challenging. This guide provides a comprehensive, secrets-focused approach to identifying and verifying the certificates used by a specific Kubernetes deployment, using the AppDynamics Cluster Agent as a practical example.
Understanding Kubernetes Certificates
What Are Certificates in Kubernetes?
Certificates in Kubernetes are digital documents used to secure communications between different components, such as API servers, nodes, and deployed applications. These certificates are typically X.509 certificates and are used for encryption, authentication, and integrity checks. In Kubernetes, certificates can be issued by a Certificate Authority (CA) and are often managed automatically, although there are scenarios where manual intervention is required.
What Are Secrets in Kubernetes?
Kubernetes Secrets are objects used to store sensitive information, such as passwords, OAuth tokens, and, importantly, TLS certificates. Storing certificates as secrets allows Kubernetes to manage and distribute them securely to the pods that require them.
Common Certificate Use Cases
Some common uses of certificates in Kubernetes include:
- Securing API Server Communications: Ensuring that communications between the Kubernetes API server and nodes are encrypted.
- Ingress Controllers: Handling SSL/TLS termination for external traffic entering the cluster.
- Mutual TLS (mTLS) Between Services: Encrypting communication between microservices within the cluster.
- Application Routes: Securing external access to applications through routes or services.
Understanding Kubernetes Secrets
Types of Secrets:
- kubernetes.io/tls: Used specifically for storing TLS certificates.
- Opaque: A generic type of secret that can store any key-value pair, including certificates.
- kubernetes.io/dockercfg: Used for storing Docker configuration, particularly for private image registries.
- kubernetes.io/service-account-token: Used to manage service account tokens.
Prerequisites
Before you begin, ensure you have the following:
- kubectl: The command-line tool for interacting with Kubernetes clusters.
- OpenSSL: A tool for working with SSL/TLS certificates.
- Access: Sufficient permissions to access and manage secrets in the Kubernetes cluster.
- AppDynamics Cluster Agent installed: Install the Cluster Agent
Step-by-Step Guide to Identifying Certificates via Secrets
Step 1: List All Secrets in the Namespace
Certificates are stored within secrets. Start by listing all secrets in the relevant namespace to identify potential candidates.
kubectl get secrets -n <namespace>
In our case I have deployed Cluster agent in appdynamics namespace, So I will reference appdynamics in subsequent steps.
Step 2: Filter Secrets Related to Certificates
Kubernetes secrets that store TLS certificates usually have the type kubernetes.io/tls. Filter secrets by this type to narrow down your search.
kubectl get secrets -n appdynamics -o json | jq '.items[] | select(.type=="kubernetes.io/tls") | .metadata.name'
In AppDynamics, We don’t have a TLS secret by default so the above should return nothing.
Step 3: Check Other Secrets
If no kubernetes.io/tls secrets are found, inspect other secrets in the namespace to see if they contain certificates.
This command will list all the secrets in the namespace, including those that might contain certificate data.
kubectl get secrets -n appdynamics
NAME TYPE DATA AGE
appdynamics-cluster-agent-dockercfg-zvglk kubernetes.io/dockercfg 1 30m
appdynamics-cluster-agent-token-5dt8s kubernetes.io/service-account-token 4 30m
appdynamics-infraviz-dockercfg-f9ncg kubernetes.io/dockercfg 1 30m
appdynamics-infraviz-token-5qpfs kubernetes.io/service-account-token 4 30m
appdynamics-operator-dockercfg-v9k7k kubernetes.io/dockercfg 1 30m
appdynamics-operator-token-478lb kubernetes.io/service-account-token 4 30m
builder-dockercfg-wkdnm kubernetes.io/dockercfg 1 30m
builder-token-nvjmq kubernetes.io/service-account-token 4 30m
cluster-agent-secret Opaque 1 30m
default-dockercfg-6dn5t kubernetes.io/dockercfg 1 30m
default-token-m7hkp kubernetes.io/service-account-token 4 30m
deployer-dockercfg-25ws4 kubernetes.io/dockercfg 1 30m
deployer-token-dd9s9 kubernetes.io/service-account-token 4 30m
sh.helm.release.v1.abhi-cluster-agent.v1 helm.sh/release.v1 1 30m
Step 4: Inspect the Content of a Secret
To examine the contents of a specific secret, such as appdynamics-cluster-agent-dockercfg-zvglk, use the following command:
kubectl get secret appdynamics-cluster-agent-dockercfg-zvglk -n appdynamics -o yaml
The output will look something like below
kubectl get secret appdynamics-cluster-agent-token-5dt8s -n appdynamics -o yaml
apiVersion: v1
data:
ca.crt: xxxxxxxx
namespace: YXBwZHluYW1pY3M=
service-ca.crt: xxxxxx==
token: xxxxxxx
kind: Secret
metadata:
annotations:
kubernetes.io/created-by: openshift.io/create-dockercfg-secrets
kubernetes.io/service-account.name: appdynamics-cluster-agent
kubernetes.io/service-account.uid: 582c10bc-b3fb-4435-9adb-7c6dfb25c2ff
creationTimestamp: "2024-09-03T17:42:20Z"
name: appdynamics-cluster-agent-token-5dt8s
namespace: appdynamics
resourceVersion: "84672"
uid: a7cc1cba-27e4-4b46-a33d-212614af9cad
type: kubernetes.io/service-account-token
Step 5: Decode and Inspect the Certificate To check the details of the certificate (e.g., ca.crt), decode it, and then use openssl to view its content:
echo "Content of ca.crt" | base64 - decode | openssl x509 -noout -text
If the certificate is found and properly decoded, the output will resemble:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: ...
Signature Algorithm: sha256WithRSAEncryption
Issuer: OU=openshift, CN=kube-apiserver-lb-signer
...
Conclusion
This guide has walked you through identifying the certificates used by a specific Kubernetes deployment, focusing on secrets. By following these steps, you can effectively manage and verify the certificates that secure your Kubernetes environments, ensuring the integrity and confidentiality of your applications.
