How do I resolve PKIX “path building failed” errors for my agent when connecting to my Controller over SSL?
When establishing a secure SSL-based connection between your agent and Controller, the agent connectivity may fail with a PKIX error — even if you followed the Enable SSL for the Java Agent documentation instructions. This may be due to configuration in your environment or in the agent.
In this article...
What PKIX error may occur if the agent-to-Controller connectivity fails?
Even if you followed the instructions in the Enable SSL for the Java Agent documentation, given your environment OR agent configuration, the agent connectivity to your Controller over SSL may fail with the following PKIK error:
[system-thread-0] 02 Jun 2021 07:07:47,848 WARN SystemAgent - Certificate chain validation failed sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target attempting validation.
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Error response from controller: Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]
What does the PKIX path building error mean?
PKIX stands for Public Key Infrastructure X509. Whenever Java attempts to connect to another application over SSL, the connection will only succeed if it can trust the application.
In Java, trust is handled with a keystore, also known as the truststore (typically <agent_home>/<version_number>/conf/cacerts.jks or <agent_home>/conf/cacerts.jks in the AppDynamics Agent context). This contains a list of all known certificate authority (CA) certificates, and Java will only trust certificates that are signed by one of the CAs or public certificates that exist within that keystore.
See Explaining the Chain of Trust - Learn What is it & How it Works and PKIX path building failed error message – CloudBees Support to learn more.
What causes this PKIX error?
The PKIX ‘path building failed’ error can arise due to:
- If the agent is not able to locate the truststore (
cacerts.jks), it won’t be able to validate the certificate produced by the Controller. Eventually, the PKIX ‘path building failed’ error will result.
- Some enterprises may have an SSL interpreting proxy sitting between the agents and the Controller. In such situations, the root CA’s (which issued the certificate to your proxy) SSL certificate needs to be imported into the agent’s truststore.
- In order to ensure a valid trust chain between the agent and the Controller, the intermediate certificate may also need to be imported, or the 'PKIX path building' will not be complete.
- It is possible that you would want to only use your application truststore. In that case, the relevant certificates need to be imported into your application truststore.
What is the most convenient resolution to the PKIX error?
The most convenient solution is to replace the Agent’s current truststore (cacerts.jks) with a working agent’s truststore (i.e., an agent sitting in the same environment as the non-working agent node in question, but that is communicating fine with the Controller).
To accomplish this:
- Take a backup of the
cacerts.jks in the non-working agent node
- Copy over the
cacerts.jks from an agent that is working fine and move it to the cacerts.jks in the non-working agent node’s directory, below :
<agent_home>/<version_number>/conf/cacerts.jks
or
<agent_home>/conf/cacerts.jks
- Restart the agent and wait for 1-2 minutes. Verify whether the agent begins to report to the Controller.
If importing the truststore from an agent that is working is not an option, then follow the steps below, under Troubleshooting by manually importing the certificates.
Troubleshooting by manually importing the certificates
The following steps are a comprehensive approach to the problem.
-
Review the output of the command below:
<jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> > cert.out
This command will print the content of the certificate being presented by the endpoint that your agent speaks to (given your environment).
For example, it could be directly talking to the SaaS Controller without an SSL interpreting proxy server OR there could be an SSL interpreting proxy server being managed by your organization intercepting all outgoing communication from all agents to the Controller. As a result, you may see two or more certificates in the certificate chain and metadata for those certificates which get captured in cert.out in the above command.
- Review the certificate chain:
<jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> -rfc > cert_rfc.out
The above command helps us to print the raw certificate chain. Once we get this chain, it can be examined in a third-party tool called KeyStore Explorer. This helps you to visualize the certificate chain right from the top of the trust chain (i.e., the CA) down to the Controller.
- You can directly import this
cert_rfc.out file to third party tool KeyStore Explorer and export the root and the intermediate entities' certificates one by one by following the steps as seen under the heading Export a trusted certificate.
Alternatively, you may follow the steps below on the KeyStore Explorer:
- Select the top certificate by clicking on the top root certificate under the 'Certificate Hierarchy'.
- Click on Export in the bottom-most row of the KeyStore Explorer window.
- As soon as you click on Export in the above step, you'll see another window that reads as
Export Certificate from entry <your-certificate-name-selected-in-step1>. With Export Format as X.509 and PEM options selected, click on Export.
- Similarly, extract the root and the other intermediate certificates.
- Import the certificates into the agent's truststore by executing the below commands:
keytool -import -file root_ca.pem -alias root_ca -keystore cacerts.jks -storepass changeit
keytool -import -file intermediate1.pem -alias intermediate1 -keystore cacerts.jks -storepass changeit
keytool -import -file intermediate2.pem -alias intermediate2 -keystore cacerts.jks -storepass changeit
In the above code snippet, most of the time it will be enough to just import the root certificate. In rare cases, you might want to import the intermediate certificates as well.
If you wish to export the certificate from the browser, then please refer to Exporting SSL certificate from your browser.
- Restart the agent and after 1-2 minutes verify directly on the Controller whether the agent begins to report on the controller.
How do I troubleshoot further if I continue to see agent-related PKIX errors?
If you continue to see the PKIX errors in the agent log even after importing the correct certificates, please contact AppDynamics Support.
Resource Summary
