Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

Enable SSL Termination and Encryption Across Nodes for the Events Service

Enabling SSL termination keeps the incoming communication channel secure all the way to the Events Service. Enabling encryption across nodes keeps all communication between the Elastic Search nodes and the data center secure. You have to have both in order to keep communication encrypted coming into and between the Events Service.

 

Refer to https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html for more information on how to work with (importing) certificates and SSL.

 

Table of Contents

Enable SSL Termination

Events Service Configuration - Server Certificate keystore

Trust store

Events Service Configuration - Server Certificate trust store

EUM Server Certificate trust store

Controller Configuration - Server Certificate trust store

Analytics Agent Configuration - Server Certificate trust store

Using a Load Balancer

Enable Encryption Across Nodes

Releases 4.4.3 and Later

Releases 4.4.2 and Earlier

 

Enable SSL Termination

This topic provides information on how to secure an on-premises installation of the Events Service using SSL. Enabling SSL for the Events Service requires configuration changes for the Events Service and may involve changes for other components, such as the Controller, the Analytics Agent, or the load balancers, if any.

Terminology:

  • events-svc.jks: keystore containing the key/certificate pair for the Events Service https server.
  • events-svc: keystore alias referencing the key/pair certificate in the Events Service key store.
  • events-svc.csr: certificate signing request for the Events Service https server.
  • events-svc.crt: signed certificate for the Events Service https server.
  • root_ca.crt: root certificate of a private singing authority.
  • events-svc.chain: concatenated ordered set of intermediate certificate(s) for the server certificate.
  • events-svc.p7b: signed certificate with chain for the Events Service https server.
  • ${JAVA_HOME}: directory containing the JRE that is used to launch the Events Service.
  • cacerts: default JRE trusted root certificate store (${JAVA_HOME}/lib/security/cacerts).

Events Service Configuration - Server Certificate keystore

To establish trust between the Events Service and other components of the AppDynamics Platform, you must create a keystore for the Events Service that contains its own signed certificate. 

  1. Run the keytool command to create a new key pair for the Events Service and store it in es-keystore.jks. Follow the onscreen instructions to configure the certificate. This generates a self-signed certificate in the keystore. For example: This command creates a key pair with a validity of 730 days (2 years). The standard key size is 2048 bits with a 2 year validity. Replace 730 with the validity period appropriate for your environment, if desired. 

    ${JAVA_HOME}/bin/keytool -genkeypair -keyalg RSA -keysize 2048 -validity 730 -alias events-svc -keystore events-svc.jks

    Follow the onscreen instructions to configure the certificate. Note that:

    • The domain name used in the appdynamics.on.premise.event.service.url property and the http.event.endpoint property must match the Common Name (CN) of the certificate used by the Events Service.
    • Enter a password for the key that follows your corporate password policy.

    This generates a self-signed certificate in the keystore. We'll generate a signing request for the certificate next.

  2. Generate a certificate signing request (CSR) for the certificate you created, using key events-svc stored in events-svc.jks, which was generated in step 1.

    ${JAVA_HOME}/bin/keytool -certreq -alias event-svc -keystore events-svc.jks -file events-svc.csr
  3. Request the certificate from your PKI service provider in the preferred format PKCS#7. The PKI service provider will provide you with the certificate and intermediate certificate(s).

    When the PKI service provider returns the individual certificates, create two files and proceed to step 3A.
    Store the server certificate in the events-svc.crt file and the intermediate certificate(s) in the events-svc.chain file. 
    The intermediate certificates in the server.chain file must be ordered, with each certificate immediately following the certificate it signed. 
     
    3A. Convert the certificate in PKCS#7 format. It is important that the certificate is imported as one entry with it's intermediate certificates
    Create the PKCS#7 file.
    From the OpenSSL certificate, when using intermediate certificate(s):
    openssl crl2pkcs7 -nocrl -certfile events-svc.crt -certfile events-svc.chain -out events-svc.p7b
    From OpenSSL certificate:
    openssl crl2pkcs7 -nocrl -certfile events-svc.crt -out events-svc.p7b
  4. Import the signed certificate in the key store. For example: 

    ${JAVA_HOME}/bin/keytool -importcert -alias events-svc -keystore events-svc.jks -file events-svc.crt
  5. Stop the Events Service.
  6. In the Events Service API properties file, edit the properties after the line port: ${ad.dw.http.port} to appear as follows:
    ad.dw.https.enabled=true
    ad.dw.https.keyStorePath=<full path to es-keystore.jks>
    ad.dw.https.keyStorePassword=<es-keystore.jks password>
    ad.dw.https.trustStorePath: ${JAVA_HOME}/jre/lib/security/cacerts
    ad.dw.https.trustStorePassword=<password>
    Note: ad.dw.https.trustStorePassword can be any value, depending on what the password was set to when you created the trust store. It is "changeme" by default. ad.dw.https.certAlias=<alias>
  7. Start the Events Service.
  8. Once the node comes up successfully, repeat this procedure for all the nodes in the Events Service deployment. 

Trust store

The acquired server certificate must be trusted by the client. For self signed certificates and certificates with a private root CA, additional actions are required to create the trust.
  1. When using a self signed certificate export the certificate from the keystore
    ${JAVA_HOME}/bin/keytool -exportcert -alias events-svc -keystore events-svc.jks -file events-svc

Events Service Configuration - Server Certificate trust store

The web server embedded in the Events Service validates the server certificate and requires the complete certificate chain to be available. The chain is used to verify the trust.
  1. Run the following command:
    ${JAVA_HOME}/bin/keytool -importcert -alias events-crt -keystore cacerts -file events-svc.crt
    or
    ${JAVA_HOME}/bin/keytool -importcert -alias events-crt -keystore cacerts -file root_ca.crt 
  2. Restart the Events Service node.
  3. Reconfigure the Events Service node.

EUM Server Certificate trust store

Complete the following steps to establish trust between the EUM Server and the Events Service.

  1. Run the following commands:
    cd <eum_install_dir>
    jre/bin/keytool -importcert -alias events-crt -keystore jre/lib/security/cacerts -file events-svc.crt
    or
    cd <eum_install_dir>
    jre/bin/keytool -importcert -alias events-crt -keystore jre/lib/security/cacerts -file root_ca.crt
  2. Restart the EUM processor.
  3. Reconfigure the EUM processor.

Controller Configuration - Server Certificate trust store

Complete the following steps to establish trust between the Controller and the Events Service.

  1. Run the following commands:
    cd <controller_install_dir>
    jre8/bin/keytool -importcert -alias events-crt -keystore /appserver/glassfish/domains/domain1/config/cacerts -file events-svc.crt
    or
    cd <controller_install_dir>
    jre8/bin/keytool -importcert -alias events-crt -keystore /appserver/glassfish/domains/domain1/config/cacerts -file root_ca.crt
  2.  Restart the Controller.
  3. Reconfigure the Controller.

Analytics Agent Configuration - Server Certificate trust store

There are two ways to establish trust between the Analytics Agent and the Events Service.

  1. Import the certificate by either:
    -Using the JVM certs store, of the machine agent or standalone analytics agent, or,
    -Using the cacerts.jks file of the machine agent. This might require additional validation as the Analytics Agents runs as an independent processor.
  2. Reconfigure the analytics agent.
  3. Restart the analytics agent.

Using a Load Balancer

These steps are specific to NGINX.

Traffic from the Controller and Analytics Agent to NGINX (from controller/agent) is over http, but traffic from NGINX to Events Service is over https

  1. Add following settings to nginx.conf file under http section:

    upstream eventsvc {
      server <api-store-1>[:<port>];
      server <api-store-2>[:<port>];
    ...
    }
    
    server {
      location / {
        proxy_pass https://eventsvc;
    }
  2. Restart NGINX.

Enable Encryption Across Nodes

Elasticsearch does not have built in support for inter-node communication over SSL, but this can be achieved using the SearchGuard plugin.

SearchGuard uses relative paths for keystore and truststore, so copies of them need to be placed in the specified directory paths for SearchGuard to find the keystore and truststore.

Releases 4.4.3 and Later

  1. Navigate to the events-service base directory (heretoafter <install-dir>)
  2. Edit the existing properties file: <install-dir>/conf/events-service-api-store.properties - all SearchGuard properties are exposed in this property file. Note, in the properties below, the keyStoreAlias and trustStoreAlias properties need to have some value. If there is no alias in use, simply pass an empty string: "". 
    # If set to true then TLS will be enabled 
    ad.es.searchguard.ssl.enable=true 
    # Relative path of keyStore to ${APPLICATION_HOME}/elasticsearch/config
    ad.es.searchguard.ssl.transport.keyStoreFilePath=eventsservice_keystore.jks
    ad.es.searchguard.ssl.transport.keyStorePassword=changeit
    ad.es.searchguard.ssl.transport.keyStoreAlias=eventsservice 
    # Relative path of trustStore to ${APPLICATION_HOME}/elasticsearch/config
    ad.es.searchguard.ssl.transport.trustStoreFilePath=eventsservice_truststore.jks
    ad.es.searchguard.ssl.transport.trustStorePassword=changeit
    ad.es.searchguard.ssl.transport.trustStoreAlias=""
    ad.es.searchguard.ssl.transport.enforceHostnameVerification=false
    ad.es.searchguard.ssl.transport.resolveHostname=false
  3. Copy the entire SearchGuard plugin directory from the archive attached to this FEZ, found in the search-guard-ssl directory, to BOTH <install-dir>/plugins and <install-dir>/elasticsearch/plugins

    Example:

    cp -r /tmp/events-service-search-guard-plugin/search-guard-ssl <install-dir>/events-service/processor/plugins/
    cp -r /tmp/events-service-search-guard-plugin/search-guard-ssl <install-dir>/events-service/processor/elasticsearch/plugins/
  4. Create the directory <install-dir>/events-service/data/config if it does not already exist.
  5. Place the certificate keystore and truststore (eventsservice_keystore.jks and eventsservice_truststore.jks in the above configuration) to BOTH <install-dir>/events-service/data/config and <install-dir>/elasticsearch/config. If your certificate do not have SAN (Subject Alternative Name) for all of the nodes that make up the cluster, than ensure that each of the nodes have its own certificate in its keystore, and truststore has the certificate chain for the certificate of other nodes.
  6. The above steps must be followed on all nodes in the events-service cluster, using a different node certificate for each node.
  7. Restart all nodes to load the plugin: 
    ./bin/events-service.sh stop; ./bin/events-service.sh start -p conf/events-service-api-store.properties &
  8. Make any necessary modifications on the AppDynamics Controller(s), EUM Server(s), Analytics Agent(s) to update connectivity to be over HTTPS and also the truststore updates.

Releases 4.4.2 and Earlier

  1. Copy the entire SearchGuard plugin directory from the archive attached to this FEZ, found in the search-guard-ssl directory, to <install-dir>/plugins

    Example:

    cp -r /tmp/events-service-search-guard-plugin/search-guard-ssl <install-dir>/events-service/processor/plugins/
    
  2. Backup the <install-dir>/conf/events-service-api-store.yml file and copy and adjust below section in the <install-dir>/conf/events-service-api-store.yml file:

    Find the section: - className:com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticSearchSingleNodeModule and then find the nodeSettings: properties section nested in this ElasticSearchSingleNodeModule section. Add the following at the end of this section, right before the systemSettings: properties section begins. Change the highlighted values for your environment: 
                  ########## BEGIN SearchGuard Cluster SSL Configuration ##########
                  #SearchGuard Configuration
                  # Enable or disable node-to-node ssl encryption (default: true)
                  searchguard.ssl.transport.enabled: true
                  # JKS or PKCS12 (default: JKS)
                  searchguard.ssl.transport.keystore_type: JKS
                  # Relative path to the keystore file (mandatory, this stores the server certificates), must be placed under the config/ dir
                  searchguard.ssl.transport.keystore_filepath: eventsservice_keystore.jks
                  # Keystore password (default: changeit) in the order of above
                  searchguard.ssl.transport.keystore_password: changeit
                  # Alias name (default: first alias which could be found)
                  searchguard.ssl.transport.keystore_alias: eventsservice
                  # JKS or PKCS12 (default: JKS)
                  searchguard.ssl.transport.truststore_type: JKS
                  # Relative path to the truststore file (mandatory, this stores the client/root certificates), must be placed under the config/ dir
                 searchguard.ssl.transport.truststore_filepath: eventsservice_truststore.jks
                  # Alias name (default: trust all aliases)
                  #searchguard.ssl.transport.truststore_alias: my_alias
                  # Truststore password (default: changeit)
                  searchguard.ssl.transport.truststore_password: changeit
    
                  ########## END SearchGuard Cluster SSL Configuration ##########
  3. Create directory <install-dir>/events-service/data/config if it does not already exist.
  4. Place the certificate keystore and truststore (eventsservice_keystore.jks and eventsservice_truststore.jks in the above configuration) in the above directory. If your certificate do not have SAN (Subject Alternative Name) for all of the nodes that make up the cluster, than ensure that each of the nodes have its own certificate in its keystore, and truststore has the certificate chain for the certificate of other nodes.
  5. Make same changes on all of the cluster nodes and restart. Validate afterwards that SSL is enabled without issues by searching for SearchGuard in the logs files. Also validate the Platform Admin can still manage the nodes and see the cluster healthy.
  6. Make any necessary modifications on the AppDynamics Controller(s), EUM Server(s), Analytics Agent(s) to update connectivity to be over HTTPS and also the truststore updates.

If one or more nodes do not start properly, it's almost certainly an issue with the TLS certificates. We can help troubleshoot these issues.

Version history
Revision #:
25 of 31
Last update:
‎10-02-2018 01:44 PM
Updated by:
 
Labels (1)


Found this article helpful? Click the Thumbs Up button.
Have an additional comment? Post it below.
Comments

Step 3 seems unneccessary as we do not need to club the chains separately by converting it into a different format and then again try importing it back to events service keystore. This is too indirect step and has more chances of error. Rather we can just get the CSR response from the Authority and import the response back to the events service keystore from where we generated the CSR. For Root CA certificate and Intermediate CA certificate we import them separetly and thus the certificate chains get completed. Hence the steps can be as below post generating CSR:

 

a. Get the CSR signed response from the signing authority along with the root and intermediate certificate.

b. Import the root and intermediate certifcate directoly in the keystore first.

c. Import the CSR response to the keystore.

 

Note: This is the same way followd for EUM keystore

 

 

We need few more things to make it work correctly:

1. The hostname of Events Service should match the CN or the SubjectAlternativeNames from the certificate.

2. We need to change:

ad.dw.http.host

to the hostname matching the CN or the SubjectAlternativeNames