Events service cluster status unhealthy and connection to elasticsearch fails

[Srishati.Yadav]

Hi Guys,

In our on-prem setup, after upgrade to version 4.4.3, we observed issues with Events service not able to connect to elastic search and events service cluster status is reported unhealthy. As a result EUM is also failing to start.

"events-service-api-store / Connection to ElasticSearch: clusterName=[appdynamics-events-service-cluster]" : {
"healthy" : false,
"message" : "clusterState=[UNHEALTHY], stateCounts=[{UNHEALTHY=100}]"

[Chitra.Lal]

Hi Srishati,

How have you upgraded the Events Service, manually or using PA? Post upgrade, did you start the elastic search process too? Can you verify manually if both the process are running fine?

Also please verify if JAVA_HOME is set correctly as an environment variable before you start the events service. Also, can you share the events-service-api-store.log and events-service-api-store.properties file as well as the ES logs to check.

How are you starting the events service? Is it manually or using windows service or using the eneterprise console? If its using the EC, then please share the platform-admin logs as well. 

Thank you,

Chitra

[Srishati.Yadav]

Hi Chitra, 

Events service cluster status is now reporting healthy. How do you restart the elasticsearch process ?

Now, only EUM is failing to start up.

12 Sep 2018 06:13:27.666 -0400 main AD.ALL ERROR
| Error Starting EUM Processor Server
| java.lang.RuntimeException: failed to update Analytics events. Please see log for details of errors!
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.syncEumAccountsToAnalyticsAccounts(EUMProcessorServerApplication.java:322)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:239)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:173)
| at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:40)
| at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:77)
| at io.dropwizard.cli.Cli.run(Cli.java:70)
| at io.dropwizard.Application.run(Application.java:80)
| at com.appdynamics.eumcloud.EUMProcessorServer.main(EUMProcessorServer.java:40)

[Srishati.Yadav]

We are not using EC to mamage the ES.

[Chitra.Lal]

Hi Srishati,

Could you please send me the EUM server logs files? Also if you disable the below property in your eum.properties file:

analytics.enabled=true 

Make this as  "analytics.enabled=false"

Is it able to start up fine?

Thank You,

Chitra

[Srishati.Yadav]

it works fine .

[Chitra.Lal]

Hi Shrishti,

Glad to hear that. Please let me know if we can consider this as resolved and close the issue here. 

Thanks,

Chitra

[Srishati.Yadav]

Sure