How do I resolve PKIX “path building failed” errors for my agent when connecting to my Controller over SSL?
When establishing a secure SSL-based connection between your agent and Controller, the agent connectivity may fail with a PKIX error — even if you followed the Enable SSL for the Java Agent documentation instructions. This may be due to configuration in your environment or in the agent.
In this article...
What PKIX error may occur if the agent-to-Controller connectivity fails?
Even if you followed the instructions in the Enable SSL for the Java Agent documentation, given your environment OR agent configuration, the agent connectivity to your Controller over SSL may fail with the following PKIK error:
[system-thread-0] 02 Jun 2021 07:07:47,848 WARN SystemAgent - Certificate chain validation failed sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target attempting validation.
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Error response from controller: Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]
What does the PKIX path building error mean?
PKIX stands for Public Key Infrastructure X509. Whenever Java attempts to connect to another application over SSL, the connection will only succeed if it can trust the application.
In Java, trust is handled with a keystore, also known as the truststore (typically <agent_home>/<version_number>/conf/cacerts.jks or <agent_home>/conf/cacerts.jks in the AppDynamics Agent context). This contains a list of all known certificate authority (CA) certificates, and Java will only trust certificates that are signed by one of the CAs or public certificates that exist within that keystore.
See Explaining the Chain of Trust - Learn What is it & How it Works and PKIX path building failed error message – CloudBees Support to learn more.
What causes this PKIX error?
The PKIX ‘path building failed’ error can arise due to:
- If the agent is not able to locate the truststore (
cacerts.jks), it won’t be able to validate the certificate produced by the Controller. Eventually, the PKIX ‘path building failed’ error will result.
- Some enterprises may have an SSL interpreting proxy sitting between the agents and the Controller. In such situations, the root CA’s (which issued the certificate to your proxy) SSL certificate needs to be imported into the agent’s truststore.
- In order to ensure a valid trust chain between the agent and the Controller, the intermediate certificate may also need to be imported, or the 'PKIX path building' will not be complete.
- It is possible that you would want to only use your application truststore. In that case, the relevant certificates need to be imported into your application truststore.
What is the most convenient resolution to the PKIX error?
The most convenient solution is to replace the Agent’s current truststore (cacerts.jks) with a working agent’s truststore (i.e., an agent sitting in the same environment as the non-working agent node in question, but that is communicating fine with the Controller).
To accomplish this:
- Take a backup of the
cacerts.jks in the non-working agent node
- Copy over the
cacerts.jks from an agent that is working fine and move it to the cacerts.jks in the non-working agent node’s directory, below :
<agent_home>/<version_number>/conf/cacerts.jks
or
<agent_home>/conf/cacerts.jks
- Restart the agent and wait for 1-2 minutes. Verify whether the agent begins to report to the Controller.
If importing the truststore from an agent that is working is not an option, then follow the steps below, under Troubleshooting by manually importing the certificates.
Troubleshooting by manually importing the certificates
On-Premise
The following steps are a comprehensive approach to the problem.
-
Review the output of the command below:
-
If it is a direct connection to the controller:
<jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> > cert.out
-
If it is a connection to the controller through a forward proxy:
<jre-home>/bin/keytool -J-Dhttps.proxyHost=<proxy_host> -J-Dhttps.proxyPort=<proxy_port> -printcert -sslserver <controller-host>:<controller-port> > cert.out
-
If it is a connection to the controller through a reverse proxy:
<jre-home>/bin/keytool -printcert -sslserver <LB_or_reverse_proxy_host>:<LB_or_reverse_proxy_port> > cert.out
These commands will print the content of the certificate being presented by the endpoint that your agent speaks to (given your environment).
As a result, you may see two or more certificates in the certificate chain and metadata for those certificates which get captured in cert.out in the above command.
- Review the certificate chain:
- If it is a direct connection to the controller:
<jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> -rfc > cert_rfc.out
- If it is a connection to the controller through a forward proxy:
<jre-home>/bin/keytool -J-Dhttps.proxyHost=<proxy_host> -J-Dhttps.proxyPort=<proxy_port> -printcert -sslserver <controller-host>:<controller-port> -rfc > cert_rfc.out
- If it is a connection to the controller through a reverse proxy:
<jre-home>/bin/keytool -printcert -sslserver <LB_or_reverse_proxy_host>:<LB_or_reverse_proxy_port> -rfc > cert_rfc.out
The above command helps us to print the raw certificate chain. Once we get this chain, it can be examined in a third-party tool called KeyStore Explorer. This helps you to visualize the certificate chain right from the top of the trust chain (i.e., the CA) down to the Controller.
- You can directly import this
cert_rfc.out file to third party tool KeyStore Explorer and export the root and the intermediate entities' certificates one by one by following the steps as seen under the heading Export a trusted certificate.
Alternatively, you may follow the steps below on the KeyStore Explorer:
- Select the top certificate by clicking on the top root certificate under the 'Certificate Hierarchy'.
- Click on Export in the bottom-most row of the KeyStore Explorer window.
- As soon as you click on Export in the above step, you'll see another window that reads as
Export Certificate from entry <your-certificate-name-selected-in-step1>. With Export Format as X.509 and PEM options selected, click on Export.
- Similarly, extract the root and the other intermediate certificates.
- Import the certificates into the agent's truststore by executing the below commands:
keytool -import -file root_ca.pem -alias root_ca -keystore cacerts.jks -storepass changeit
keytool -import -file intermediate1.pem -alias intermediate1 -keystore cacerts.jks -storepass changeit
keytool -import -file intermediate2.pem -alias intermediate2 -keystore cacerts.jks -storepass changeit
In the above code snippet, most of the time it will be enough to just import the root certificate. In rare cases, you might want to import the intermediate certificates as well.
If you wish to export the certificate from the browser, then please refer to Exporting SSL certificate from your browser.
- Restart the agent and after 1-2 minutes verify directly on the Controller whether the agent begins to report on the controller.
SaaS
- If you do not have a TLS intercepting proxy, follow the steps below, or follow the steps described above under the section, On-premise controllers even though you use a SaaS controller.
- Download all of the below certificates:
- Import each of the above-downloaded certificates using the following command:
keytool -import -alias rootCA -file <root_certificate_file_name> -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
For example:
keytool -import -alias digicertglobalrootCA -file DigiCertGlobalRootCA.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
keytool -import -alias digicertglobalrootG2 -file DigiCertGlobalRootG2.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
keytool -import -alias digicertglobalrootG3 -file DigiCertGlobalRootG3.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
keytool -import -alias identrust -file <file_name_for_IdenTrust_Commercial_Root_CA_1> -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
- Restart the agent and after 1-2 minutes verify directly on the Controller whether the agent begins to report on the controller.
How do I troubleshoot further if I continue to see agent-related PKIX errors?
If you continue to see the PKIX errors in the agent log even after importing the correct certificates, please contact AppDynamics Support.
Resource Summary
