cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Madhup.Srivastawa
AppDynamics Team (Retired)

How do I resolve PKIX “path building failed” errors for my agent when connecting to my Controller over SSL?

When establishing a secure SSL-based connection between your agent and Controller, the agent connectivity may fail with a PKIX error — even if you followed the Enable SSL for the Java Agent documentation instructions. This may be due to configuration in your environment or in the agent.

 

In this article...


 

What PKIX error may occur if the agent-to-Controller connectivity fails?

Even if you followed the instructions in the Enable SSL for the Java Agent documentation, given your environment OR agent configuration, the agent connectivity to your Controller over SSL may fail with the following PKIK error:

 

 

[system-thread-0] 02 Jun 2021 07:07:47,848  WARN SystemAgent - Certificate chain validation failed sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target attempting validation.
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[system-thread-0] 02 Jun 2021 07:07:47,850 ERROR ControllerTimeSkewHandler - Error response from controller: Fatal transport error while connecting to URL [/controller/instance/1265056/current-time]

 

 

 

What does the PKIX path building error mean?

PKIX stands for Public Key Infrastructure X509Whenever Java attempts to connect to another application over SSL, the connection will only succeed if it can trust the application. 

In Java, trust is handled with a keystore, also known as the truststore (typically <agent_home>/<version_number>/conf/cacerts.jks or <agent_home>/conf/cacerts.jks in the AppDynamics Agent context). This contains a list of all known certificate authority (CA) certificates, and Java will only trust certificates that are signed by one of the CAs or public certificates that exist within that keystore. 

See Explaining the Chain of Trust - Learn What is it & How it Works and PKIX path building failed error message – CloudBees Support to learn more.

 

What causes this PKIX error? 

The PKIX  ‘path building failed’ error can arise due to:

  • If the agent is not able to locate the truststore (cacerts.jks), it won’t be able to validate the certificate produced by the Controller. Eventually, the PKIX ‘path building failed’ error will result.

  • Some enterprises may have an SSL interpreting proxy sitting between the agents and the Controller. In such situations, the root CA’s (which issued the certificate to your proxy) SSL certificate needs to be imported into the agent’s truststore.

  • In order to ensure a valid trust chain between the agent and the Controller, the intermediate certificate may also need to be imported, or the 'PKIX path building' will not be complete.

  • It is possible that you would want to only use your application truststore. In that case, the relevant certificates need to be imported into your application truststore.

 

What is the most convenient resolution to the PKIX error?

The most convenient solution is to replace the Agent’s current truststore (cacerts.jks) with a working agent’s truststore (i.e., an agent sitting in the same environment as the non-working agent node in question, but that is communicating fine with the Controller). 

 

To accomplish this:

  1. Take a backup of the cacerts.jks in the non-working agent node

  2. Copy over the cacerts.jks from an agent that is working fine and move it to the cacerts.jks in the non-working agent node’s directory, below :
    <agent_home>/<version_number>/conf/cacerts.jks

    or

    <agent_home>/conf/cacerts.jks

     

  3. Restart the agent and wait for 1-2 minutes. Verify whether the agent begins to report to the Controller.

 

If importing the truststore from an agent that is working is not an option, then follow the steps below, under Troubleshooting by manually importing the certificates.

 

Troubleshooting by manually importing the certificates

On-Premise

The following steps are a comprehensive approach to the problem.

  1. Review the output of the command below:

    1. If it is a direct connection to the controller:

      <jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> > cert.out
    2. If it is a connection to the controller through a forward proxy:

      <jre-home>/bin/keytool -J-Dhttps.proxyHost=<proxy_host> -J-Dhttps.proxyPort=<proxy_port> -printcert -sslserver <controller-host>:<controller-port> > cert.out
    3. If it is a connection to the controller through a reverse proxy:

      <jre-home>/bin/keytool -printcert -sslserver <LB_or_reverse_proxy_host>:<LB_or_reverse_proxy_port> > cert.out

      These commands will print the content of the certificate being presented by the endpoint that your agent speaks to (given your environment).

      As a result, you may see two or more certificates in the certificate chain and metadata for those certificates which get captured in cert.out in the above command.


  2. Review the certificate chain:
    1. If it is a direct connection to the controller:
      <jre-home>/bin/keytool -printcert -sslserver <controller-host>:<controller-port> -rfc > cert_rfc.out​
    2. If it is a connection to the controller through a forward proxy:
      <jre-home>/bin/keytool -J-Dhttps.proxyHost=<proxy_host> -J-Dhttps.proxyPort=<proxy_port> -printcert -sslserver <controller-host>:<controller-port> -rfc > cert_rfc.out
    3. If it is a connection to the controller through a reverse proxy:
      <jre-home>/bin/keytool -printcert -sslserver <LB_or_reverse_proxy_host>:<LB_or_reverse_proxy_port> -rfc > cert_rfc.out

      The above command helps us to print the raw certificate chain. Once we get this chain, it can be examined in a third-party tool called KeyStore Explorer. This helps you to visualize the certificate chain right from the top of the trust chain (i.e., the CA) down to the Controller.

  3. You can directly import this cert_rfc.out file to third party tool KeyStore Explorer and export the root and the intermediate entities' certificates one by one by following the steps as seen under the heading Export a trusted certificate.

    Alternatively, you may follow the steps below on the KeyStore Explorer:
    1. Select the top certificate by clicking on the top root certificate under the 'Certificate Hierarchy'.
    2. Click on Export in the bottom-most row of the KeyStore Explorer window.
    3. As soon as you click on Export in the above step, you'll see another window that reads as Export Certificate from entry <your-certificate-name-selected-in-step1>. With Export Format as X.509 and PEM options selected, click on Export.
    4. Similarly, extract the root and the other intermediate certificates.
    5. Import the certificates into the agent's truststore by executing the below commands:
      keytool -import -file root_ca.pem -alias root_ca -keystore cacerts.jks -storepass changeit
      keytool -import -file intermediate1.pem -alias intermediate1 -keystore cacerts.jks -storepass changeit
      keytool -import -file intermediate2.pem -alias intermediate2 -keystore cacerts.jks -storepass changeit


      In the above code snippet, most of the time it will be enough to just import the root certificate. In rare cases, you might want to import the intermediate certificates as well.


      If you wish to export the certificate from the browser, then please refer to
      Exporting SSL certificate from your browser.

 

  1. Restart the agent and after 1-2 minutes verify directly on the Controller whether the agent begins to report on the controller.

SaaS

  1. If you do not have a TLS intercepting proxy, follow the steps below, or follow the steps described above under the section, On-premise controllers even though you use a SaaS controller.
  2. Download all of the below certificates:
  3. Import each of the above-downloaded certificates using the following command:
    keytool -import -alias rootCA -file <root_certificate_file_name> -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit​
    For example: 
    keytool -import -alias digicertglobalrootCA -file DigiCertGlobalRootCA.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
    keytool -import -alias digicertglobalrootG2 -file DigiCertGlobalRootG2.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
    keytool -import -alias digicertglobalrootG3 -file DigiCertGlobalRootG3.crt.pem -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit
    keytool -import -alias identrust -file <file_name_for_IdenTrust_Commercial_Root_CA_1> -keystore <agent_home>/<version_number>/conf/cacerts.jks -storepass changeit

     

  4. Restart the agent and after 1-2 minutes verify directly on the Controller whether the agent begins to report on the controller.

How do I troubleshoot further if I continue to see agent-related PKIX errors?

If you continue to see the PKIX errors in the agent log even after importing the correct certificates, please contact AppDynamics Support.

 

Resource Summary

Comments
Vijay.Venkatarajulu
AppDynamics Team

@Madhup.Srivastawa 

Can you edit your original comment and add config path for machine agent as well for certs (<MachineAgentHomeDir>/conf)

 

Thanks,

Vijay

Madhup.Srivastawa
AppDynamics Team (Retired)

@Vijay.Venkatarajulu 

 

Sure Vijay, working on it. Thanks for your suggestions.

Claudia.Landivar
AppDynamics Team (Retired)

Hello, @Vijay.Venkatarajulu and @Madhup.Srivastawa - The article has been updated.

Claudia Landivar

Community Manager & Editor

Vijay.Venkatarajulu
AppDynamics Team
Claudia.Landivar
AppDynamics Team (Retired)

Sure thing, @Vijay.Venkatarajulu — thanks for pointing out the improvement.

Version history
Last update:
‎05-22-2024 09:11 AM
Updated by:
June 26 Webinar
Discover new Splunk integrations and AI innovations for Cisco AppDynamics.


Register Now!

Observe and Explore
Dive into our Community Blog for the Latest Insights and Updates!


Read the blog here