Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

How do I secure the Events Service and related Platform components (v4.4.3+)?

Securing your Events Service, especially when hosting your Events Service nodes in a shared environment, adds a extra layer of encryption around traffic sent to, from, and within your Events Service cluster. This traffic can include IP addresses, infrastructure-related details, event data, and credit card information. Follow the steps below to encrypt your Events Service and related platform components.

 

Table of Contents:

 

Platform Components and Segments to Secure

Below are the 4 Platform components, each connected by what we refer to as a “segment":

  1. ControllerPLATFORM-ALL-CONTROLLER.mydomain.me – 192.168.254.133
  2. Events Service Load BalancerPLATFORM-ALL-ES.mydomain.me – 192.168.254.131
  3. Events Service Nodes (3-Node Cluster)
    1. PLATFORM-ALL-ES-01.mydomain.me – 192.168.254.128
    2. PLATFORM-ALL-ES-02.mydomain.me – 192.168.254.129
    3. PLATFORM-ALL-ES-03.mydomain.me – 192.168.254.130
  4. EUM ServerPLATFORM-ALL-EUM.mydomain.me – 192.168.254.132

full-platform.png

  

Below are the 4 segments connecting the Platform components that we recommend securing:

  1. Segment 1: Events Service Load Balancer + Events Service Cluster Nodes (ex: 3-node)
  2. Segment 2: Events Service Node <- -> Events Service Node (Internode)
  3. Segment 3: Controller -> Events Service Load Balancer
  4. Segment 4: EUM Server -> Events Service Load Balancer

 

General Notes

  1. If all Events Service nodes in the Events Service cluster are not started simultaneously, communication attempts made between the nodes will timeout and prevent the Events Service from starting on any of the nodes.
  2. When upgrading your Platform components (Controller, Events Service nodes, and EUM Server), refer to our documentation to ensure it’s completed in the correct order.
  3. The certificate used to secure the platform in this example is a wildcard certificate issued by a third-party CA (CN = *.mydomain.me)​.
  4. All traffic is allowed between the different Platform components assuming you have applied the recommendations in Network and Port settings and how to prepare the Events Service host for Linux-based hosts.

 

Load Balance Events Service Traffic (Secured using Nginx LB)

To distribute load among the members of an Events Service cluster, you need to set up a load balancer. It is important to complete this step before securing your Events Service.

 

Events Service Cluster.png 

  

Note: The following example is just one method that can be used to load balance the Events Service cluster’s traffic. These instructions are for a Linux-based host. Additional examples and information on load balancer configurations are available here.

 

1. Edit the  /etc/hosts file from the Load Balancer running Nginx, as well as from each of the nodes.

127.0.0.1         localhost.localdomain localhost
192.168.254.128   PLATFORM-ALL-ES-01.mydomain.me PLATFORM-ALL-ES-01
192.168.254.129   PLATFORM-ALL-ES-02.mydomain.me PLATFORM-ALL-ES-02
192.168.254.130   PLATFORM-ALL-ES-03.mydomain.me PLATFORM-ALL-ES-03
192.168.254.131   PLATFORM-ALL-ES.mydomain.me PLATFORM-ALL-ES
192.168.254.132   PLATFORM-ALL-EUM.mydomain.me PLATFORM-ALL-EUM
192.168.254.133   PLATFORM-ALL-CONTROLLER.mydomain.me PLATFORM-ALL-CONTROLLER

 

2. Assuming the Nginx package and associated dependencies have been successfully installed using the instructions in step 1 here, see the sample configuration file below used for load balancing the cluster’s traffic.

 

PLATFORM-ALL-ES.mydomain.me.conf

upstream PLATFORM-ALL-ES.mydomain.me {
   server 192.168.254.128:9443;
   server 192.168.254.129:9443;
   server 192.168.254.130:9443;
   keepalive 15;
}
server {
   listen 9443 ssl;
   server_name platform-all-es.mydomain.me;
   ssl on;
   ssl_certificate /etc/ssl/private/star_mydomain_me_combined.crt;
   ssl_certificate_key /etc/ssl/private/star_mydomain_me.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_prefer_server_ciphers on;
proxy_ssl_server_name on;

   ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
   location / {
      proxy_pass_header Authorization;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_bind 192.168.254.131;
proxy_pass https://PLATFORM-ALL-ES.mydomain.me;
     proxy_http_version 1.1;
     proxy_set_header Connection "Keep-Alive";
     proxy_set_header Proxy-Connection "Keep-Alive";
   }
}

 

Securing Segment 1: Events Service Load Balancer + Events Service Cluster Nodes (ex: 3-node)

ES -1.png 

Notes:

  • The Enterprise Console was used to deploy a 3-node cluster using the default REST API value TCP port 9080 for traffic from the Events Service load balancer to each of the Events Service nodes.
  • Secured communication for the Events Service load balancer and to each of the Events Service nodes will take place over TCP port 9443.
  • You cannot update the REST API port after deploying an Events Service cluster from within the Enterprise Console GUI. The only solution to update the port value for an existing Events Service deployment is to manually update the REST API port in the Enterprise Console database.
  • You cannot enable SSL for the Events Service cluster using the Enterprise Console without using the Enterprise Console CLI.

 

Steps:

1. Update the REST API port from TCP port 9080 to TCP port 9443. (Note: Please back up your original database before completing these steps.)

  • Select the latest configured eventsServiceRestApiPort and verify the id and port value.
    USE platform_admin;
    SELECT id, property_value FROM `platform_admin`.`configuration_store` WHERE `property_key` = "eventsServiceRestApiPort" ORDER BY last_modified_at DESC LIMIT 1;
  •  For the purposes of this example, let’s say the query returned id = 2230 and the associated property_value = 9080:
    id = 2230  |   property_value = 9080

     

    You would update this value in the platform_admin database with the following command: 
    UPDATE `platform_admin`.`configuration_store` SET `property_value`='[\"9443\"]' WHERE `id`='2230';
  • Verify the update succeeded. In this example, the query should return: id = 2230 and the associated property_value = 9443
    SELECT id, property_value FROM `platform_admin`.`configuration_store` WHERE `property_key` = "eventsServiceRestApiPort" ORDER BY last_modified_at DESC LIMIT 1;

 

2. Use the Enterprise Console CLI to enable SSL for the cluster

  • First determine the job's parameters with the command below:
     bin/platform-admin.sh list-job-parameters --job enable-ssl --service events-service  
     Parameters for job "enable-ssl":
          keystorePath (STRING, required)
          keystorePassword (STRING, required)
          keystoreAlias (STRING, required)
  • Using the parameters allowed, specify the path to your keystore, the password for the keystore, and the alias configured for this keystore. Also verify that the keystore exists at the specified path before executing the command.
  • Run the following command: 
    bin/platform-admin.sh submit-job --service events-service --job enable_ssl --args keystorePath=/opt/AppDynamics/mykeystore.jks keystorePassword=changeit keystoreAlias=events-service
  • After successfully running the above command, Segment 1 of your Platform should be secured using the keystore specified above.

 

Troubleshooting Common Issues:

The Events Service health check may return the following unhealthy status if the property ad.dw.http.host within the events-service-api-store.properties file does not match the value specified in the certificates CN. For example, the error below was generated because the events-service-api-store.properties file for each of the nodes was configured to ad.dw.http.host=0.0.0.0 

events-service-api-store / Connection to ElasticSearch: clusterName=[appdynamics-events-service-cluster]: (healthy) clusterState=[HEALTHY], stateCounts=[{HEALTHY=100}]


events-service-api-store / Connection to [https://0.0.0.0:9443] with [DefaultAccountServiceClient]: (unhealthy) Host name '0.0.0.0' does not match the certificate subject provided by the peer (CN=*.mydomain.me, OU=Org Unit, O=Org, C=US)


events-service-api-store / Connection to [https://0.0.0.0:9443] with [DefaultEventServiceClient]: (unhealthy) Host name '0.0.0.0' does not match the certificate subject provided by the peer (CN=*.mydomain.me, OU=Org Unit, O=Org, C=US)
...

 

NODE #1:       ad.dw.http.host=0.0.0.0
NODE #2:       ad.dw.http.host=0.0.0.0
NODE #3:       ad.dw.http.host=0.0.0.0

 

In order to resolve the unhealthy report mentioned above, I needed to update the property on each of the nodes to their respective long hostname and restart each of the nodes simultaneously.

 

NODE #1:       ad.dw.http.host=PLATFORM-ALL-ES-01.mydomain.me
NODE #2:       ad.dw.http.host=PLATFORM-ALL-ES-02.mydomain.me
NODE #3:       ad.dw.http.host=PLATFORM-ALL-ES-03.mydomain.me

 

After performing the update and the restarting the cluster, you should no longer see unhealthy reports.

…..
  "events-service-api-store / Connection to ElasticSearch: clusterName=[appdynamics-events-service-cluster]" : {
    "healthy" : true,
    "message" : "clusterState=[HEALTHY], stateCounts=[{DEGRADED=3, HEALTHY=82}]"
  },
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [DefaultAccountServiceClient]" : {

    "healthy" : true
  },
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [DefaultEventServiceClient]" : {
    "healthy" : true
  },
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [DefaultQueryEventsClient]" : {
    "healthy" : true
  },
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [RestEventTypeClient]" : {
    "healthy" : true
  },
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [RestJobFrameworkClient]" : {
    "healthy" : true
  }
  "events-service-api-store / Connection to [https://PLATFORM-ALL-ES-02.mydomain.me:9443] with [RestSlmPerfConfigsClient]" : {
    "healthy" : true
  },
 .....
  "events-service-api-store / jobframework-module" : {
    "healthy" : true,
    "message" : "Job Framework instanceId: [PLATFORM-ALL-ES-02.mydomain.me], number of jobs executed: [0], running since [2018-05-25T08:59:52.132Z], currently executing jobs []"
  },
  "events-service-api-store / queues" : {
    "healthy" : true,
    "message" : "[1] queues [[biz-outcome-incoming-events] ratio: [0.00], size: [0], capacity: [1000]]"
  }

 

Securing Segment 2: Events Service Node <- -> Events Service Node (Internode)


ES -2.png

For instructions on how to secure the segment between nodes, review the steps outlined for version v4.4.3+ here: Enable Encryption Across Nodes

 

Securing Segment 3: Controller --> Events Service Load Balancer

 

             CONTROLLER.png   arrow-r.png .   Events Service Cluster.png

  

For instructions on how to secure the segment between the Controller and Events Service Load Balancer, see: Controller Configuration - Server Certificate trust store.

 

You will need to make the following changes to the Controller's admin.jsp controller settings:

appdynamics.on.premise.event.service.url --> https://platform-all-es.mydomain.me:9443
eum.es.host                              -->  https://platform-all-es.mydomain.me:9443

 

Securing Segment 4: EUM Server --> Events Service Load Balancer

Events Service Cluster.png        arrow-left.png   EUM Server.png

 

Note: If Analytics is enabled in the EUM Server’s properties file (ex: analytics.enabled=true) and the EUM Server cannot reach the Events Service, the EUM Server will fail to start.

 

1. The acquired server certificate must be trusted by the client. For self signed certificates and certificates with a private root CA, additional actions are required to create the trust. When using a self signed certificate, export the certificate from the keystore.

${JAVA_HOME}/bin/keytool -exportcert -alias events-svc -keystore events-svc.jks -file events-svc

 

2. The web server embedded in the Events Service validates the server certificate and requires the complete certificate chain to be available. The chain is used to verify the trust. Follow the instructions in Events Service Configuration - Server Certificate Trust Store to complete this step.

 

3. Establish trust between the EUM Server and the Events Service following the steps here: EUM Server Certificate trust store.

 

4. Connect the EUM Server to the Events Service following the steps outlined in End User Monitoring Connection Settings. When set correctly, these properties tell the EUM Server’s appserver what setting to use when connecting to the Events Service. These are the relevant properties from eum.properties file:

.....
# Web server properties
processorServer.httpPort=7001
processorServer.httpsPort=7002
processorServer.httpsProduction=true
processorServer.keyStorePassword=PASSWORD
processorServer.keyStoreFileName=eum.jks

# Analytics server properties
analytics.enabled=true
analytics.serverScheme=https
analytics.serverHost=PLATFORM-ALL-ES.mydomain.me
analytics.port=9443
analytics.accountAccessKey=8c59ea79-b297-4996-ade6-861c2378eb7b
....