For Java-based agents, quite often, we run into issues with connectivity when either the agent is migrating from an on-prem controller to a SaaS controller or redirecting from one controller to another or when the Network team introduces a network tool such as Zscaler or a simple Java upgrade. This is much more pronounced for Java agents than for Machine agents and Database agents, where the JRE is usually bundled with the product.
The JRE is the one which is used to start the JVM and as the name suggests this is a Virtual Machine for Java. So whatever exists on the system or node Java ignores that and only trusts what the virtual machine spawned has access to. For Java, the key here to establish a TLS connection with any server is that the public cert for the server needs to be trusted. For ease of consumption of this article, I will divert to 2 sections as the JRE is usually in control of MA and DB agents.
For Java agents:
The key parameter that the JVM uses to establish trust with the server is the truststore passed to the JVM. This ideally should have not just the certs for the controller but any secured service the JVM will connect to. Quite often developers will develop apps and pass a custom truststore to the same. This is the root of the issue in most scenarios as by passing a truststore to the JVM we are putting blinders on that JVM and the JVM will not trust the JRE default certs (which is usually at JRE/lib/security/cacerts) and only trust certs in the passed truststore. A quick check of the JVM args should be able to assist if a custom truststore is being passed or not. If not then the default JRE cacerts file is the truststore. If you have a java agent not connecting to the controller then add the public cert of the controller to the truststore.
If you don't have a custom truststore being passed to the JVM then check the cacerts file for the JRE and use any tool to open and check if the public cert offered by the controller is present within the file.
Network Connectivity Issue
Sometimes the certificate is not the issue but network connectivity is. For that we we use a neutral tool such as keytool which exists within each JRE/bin folder to test connectivity as well as show the entire TLS handshake on the terminal. Simply run the following:
keytool -printcert -sslserver <host>:<port> -J-Djavax.net.debug=ssl
If proxy is involved then you can add proxy details by adding extra JVM args such as -J-DproxyHost and -J-DproxyPort.
For eg.,
keytool -printcert -sslserver <insertyourtenant>.saas.appdynamics.com:443 -J-Djavax.net.debug=ssl -J-DproxyHost=10.8.9.234 -J-DproxyPort=443
The above command if it fails then the network team can be involved to check network connectivity between the server and controller.
For Machine and Database agent:
Since the JRE is usually bundled the chances of global root CAs not being present with the JRE is minimized, however, if you have an on-prem secured controller then the TLS handshake will fail till the public cert is available with the JVM. For that easiest option is to create a jks file holding the public certs and name the same cacerts.jks and place in the conf folder. The agents have a logic in them to add whatever certs you specify in the cacerts.jks file. Please note the file name must be cacerts.jks and if read the logs will mention the same including the actual cert details when debug logging is enabled
TLS handshake capture
In case you wish to capture in the logs the actual TLS handshake the JVM establishes with the servers then you need to enable TLS debugging and ensure the stdout and stderr files are actually being logged in a file. Please note for JVMs run as a service the stdout/stderr may be redirected to /dev/null and thus we need to restart the JVM and ensure the redirection is in place.
The JVM args are: -Djavax.net.ssl=debug -Djavax.net.debug=all
For eg.
nohup /app/appdynamics/machineagent/bin/machine-agent -Dappdynamics.http.proxyHost=10.8.9.234 -Dappdynamics.http.proxyPort=443 -Djavax.net.ssl=debug -Djavax.net.debug=all > /app/appdynamics/machineagent/bin/nohup.out 2>&1 &
