Cluster Agent Pod Stuck in CrashLoopBackOff

[sajo.sam]

I'm trying to deploy a cluster agent in my Kubernetes cluster to monitor the infrastructure using the kubectl CLI. I've followed the steps and executed these commands:

kubectl create -f cluster-agent-operator.yaml

kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>

kubectl create -f cluster-agent.yaml

However, the cluster agent pod is stuck in the "CrashLoopBackOff" state. The logs indicate an issue with the account access key:

[ERROR]: 2024-04-03 18:29:45 - main.go:183 - Account accessKey is not specified
[ERROR]: 2024-04-03 18:29:45 - main.go:184 - Please provide account accessKey before starting cluster-agent. Exiting...

• I've verified that the cluster-agent-secret contains the controller-key with the correct access key value.

What could be causing this issue despite providing the access key in the secret? Are there any additional configuration steps I might be missing?

Reference :

https://docs.appdynamics.com/appd/22.x/latest/en/infrastructure-visibility/monitor-kubernetes-with-t...

[Ryan.Paredez]

Hi @sajo.sam,

I found this TKB article. Please check it out and see if it helps.
https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-debug-common-Linux-Private-Synthetic-Ag...

[Ryan.Paredez]

Hi @sajo.sam,

Did you get a chance to check out the TKB article or have you found a solution you can share?

[sajo.sam]

Hi @Ryan.Paredez 

I tried but I'm stuck with another issue. The logs given below show it faces some errors with "Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration ": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[ERROR]: 2024-04-09 11:20:38 - secretconfig.go:68 - Problem With Getting /opt/appdynamics/cluster-agent/secret-volume/api-user Secret: open /opt/appdynamics/cluster-agent/secret-volume/api-user: no such file or directory

[INFO]: 2024-04-09 11:20:38 - main.go:78 - Kubernetes version: v1.29.0
[INFO]: 2024-04-09 11:20:38 - main.go:236 - Registering cluster agent with
 controller host : accountname.saas.appdynamics.com
 controller port : 8080
 account name : accountname
[WARNING]: 2024-04-09 11:20:38 - agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:356 - Established connection to Kubernetes API
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:68 - Cluster name: fromKube
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-04-09 11:21:38 - agentregistrationmodule.go:119 - Initial Agent registration

^ Post edited by @Ryan.Paredez to remove mentions and links to Account name. For security and privacy reasons, please redact the name of your Account in Community posts.

[Marcie.Sirbaugh]

I see this in your log, have you figured out why this is happening?
agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

[sajo.sam]

Hy @Marcie.Sirbaugh 

I just looked into it but didn't get any information from this because the namespace default is already there but I don't know where i'm missing

[Ryan.Paredez]

Hi @Marcie.Sirbaugh,

I see you have an open ticket with the same error you asked Sajo about 

agentregistrationmodule.go:352

Perhaps you can continue to share any outcomes from that interaction with your ticket here with Sajo.

[Rajesh.Ganapavarapu]

Hello @sajo.sam,

The error message you've encountered "agentregistrationmodule.go:132 - clusterId: -1" indicates that the cluster agent received incorrect controller information or that the controller is rejecting the registration.

To verify if you've configured the correct controller access key, you can execute the following command:

kubectl get secret cluster-agent-secret -n appdynamics -o jsonpath='{.data.controller-key}' | base64 --decode

Please note that DONOT share any controller sensitive information in this public platform.

Regarding the error mentioned in your comments:

[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

This is likely due to a network connection issue. You can check the network connection from one of the running pods using the following command:

curl -v -k -u singularity-agent@<accountname>  https://<controllerhost>.saas.appdynamics.com:<port>/sim/v2/agent/clusterRegistration

Please check the above if that helpful.

Best Regards,
Rajesh Ganapavarapu