Troubleshooting AppDynamics Integration with Kubernetes Cluster

[sajo.sam]

AIM: Integrate AppDynamics with a Kubernetes cluster using the provided documentation.

Issue: I've set up a Kubernetes cluster and aimed to integrate it with AppDynamics for monitoring. Following the provided documentation, I successfully created the cluster agent. However, encountered errors during logging and found that the cluster data isn't showing up in the AppDynamics interface.

Reference :Install the Cluster Agent with the Kubernetes CLI 

Logs and Findings: 

PS C:\Users\SajoSam> kubectl logs k8s-cluster-agent-5f8977b869-bpf5v
CA_PROPERTIES= -appdynamics.agent.accountName=myaccount -appdynamics.controller.hostName=mycontroller.saas.appdynamics.com -appdynamics.controller.port=8080 -appdynamics.controller.ssl.enabled=false -appdynamics.agent.monitoredNamespaces=default -appdynamics.agent.event.upload.interval=10 -appdynamics.docker.container.registration.interval=120 -appdynamics.agent.httpClient.timeout.interval=30
APPDYNAMICS_AGENT_CLUSTER_NAME=onepane-cluster
[ERROR]: 2024-03-26 09:55:04 - secretconfig.go:68 - Problem With Getting /opt/appdynamics/cluster-agent/secret-volume/api-user Secret: open /opt/appdynamics/cluster-agent/secret-volume/api-user: no such file or directory
[INFO]: 2024-03-26 09:55:04 - main.go:57 - check env variables and enable profiling if needed
[INFO]: 2024-03-26 09:55:04 - agentprofiler.go:22 - Cluster Agent Profiling not enabled!
[INFO]: 2024-03-26 09:55:04 - main.go:60 - Starting APPDYNAMICS CLUSTER AGENT version 24.2.0-317
[INFO]: 2024-03-26 09:55:04 - main.go:61 - Go lang version: go1.22.0
W0326 09:55:04.910967 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
[INFO]: 2024-03-26 09:55:04 - main.go:78 - Kubernetes version: v1.29.0
[INFO]: 2024-03-26 09:55:04 - main.go:233 - Registering cluster agent with
controller host : mycontroller.saas.appdynamics.com
controller port : 8080
account name : xxxxx
[INFO]: 2024-03-26 09:55:04 - agentregistrationmodule.go:356 - Established connection to Kubernetes API
[INFO]: 2024-03-26 09:55:04 - agentregistrationmodule.go:68 - Cluster name: onepane-cluster
[INFO]: 2024-03-26 09:55:04 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-03-26 09:55:34 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "http://mycontroller.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-03-26 09:55:34 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-03-26 09:55:34 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-03-26 09:56:04 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-03-26 09:56:34 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "http://mycontroller.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-03-26 09:56:34 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-03-26 09:56:34 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-03-26 09:57:04 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-03-26 09:57:34 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "http://mycontroller.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": dial tcp 35.84.229.250:8080: i/o timeout (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-03-26 09:57:34 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-03-26 09:57:34 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-03-26 09:58:04 - agentregistrationmodule.go:119 - Initial Agent registration

Question :

1. What could be the root cause of the failure to access the secret file /opt/appdynamics/cluster-agent/secret-volume/api-user?

2. What could be causing the timeout error during the registration request to the AppDynamics controller?

Could you help me with this?

Thank you

^ Post edited by @Ryan.Paredez to redact account name and controller name. For privacy and security reasons, please do not share your Account name or Controller URL. 

[Ryan.Paredez]

Hi @sajo.sam,

I'm going to see what I can find for you, in the meantime, have you seen/read this AppD Docs page: https://docs.appdynamics.com/appd/23.x/latest/en/infrastructure-visibility/monitor-kubernetes-with-t...

[sajo.sam]

Thanks, @Ryan.Paredez  for trying the suggested solution. It seems we're still encountering the same error referencing additional documentation. Regarding the issue with accessing the secret file '/opt/appdynamics/cluster-agent/secret-volume/api-user', it appears the file might be missing or inaccessible. Additionally, I attempted the examples provided in the link you shared (https://docs.appdynamics.com/appd/23.x/latest/en/infrastructure-visibility/monitor-kubernetes-with-t...)

[Ryan.Paredez]

Hi @sajo.sam,

Can you check out the replies to this post and see if this helps: https://community.appdynamics.com/t5/Infrastructure-Server-Network/Kubernetes-cluster-agent-can-not-...

[sajo.sam]

Hi @Ryan.Paredez 

Thank you for sharing the discussion link. I've carefully read the responses and tried the suggested solutions, but unfortunately, they didn't fix the problems I'm having. If you have any more suggestions or information, please share it with me.

[Rajesh.Ganapavarapu]

Hello @sajo.sam ,

This error message we see when there is an incorrect controller information passed. 

You may check the access key by running this below command.

kubectl get secret cluster-agent-secret -n appdynamics -o jsonpath='{.data.controller-key}' | base64 --decode

Also please check the network connection

Run the following command to check the connection
curl -v -k -u singularity-agent@dtvnprod:<access_key> https://dtvnprod.saas.appdynamics.com:443/sim/v2/agent/clusterRegistration

Hope this helps.

Best Regards,
Rajesh Ganapavarapu

[Ryan.Paredez]

Hi @sajo.sam,

Did you see the reply from @Rajesh.Ganapavarapu? Can you confirm if it helped? If it did, click the "Accept as Solution" button, if not, continue the conversation