Events Service failed to start. Command failed due to an error: Invalid cluster state.

[kin]

Hello,

I'm trying to setup an on-premise AppDynamics platform. I followed documentation and installed events service on a dedicated machine without errors. So right now I have a controller server and events service server. But when I try to launch the events service via platform-admin I'm getting following error:

apd@apd-controller:~$ ./platform-admin.sh start-events-service
Starting Events Service.
Events Service failed to start.Command failed due to an error: Invalid cluster state.
API code 400

 Any ideas what is wrong with my cluster?

Regards

[Mohammed.Rayan]

Kim,

Can you please share the logs from your events-service host (events-service-api-store.log) & the platform-admin-server.log from the platform admin host

For instance: cd <events_service_install_dir>/logs/events-service-api-store.log

Regards,

Mohammed Rayan

[kin]

Excerpt from events-service-api-store.log:

[2017-08-01T10:00:29,499+02:00]  [INFO ]  [Thread-13]  [c.a.common.framework.util.SimpleApp]  Stopping [events-service-api-store]
[2017-08-01T10:00:29,510+02:00]  [INFO ]  [Thread-13]  [o.e.jetty.server.ServerConnector]  Stopped application@71739267{HTTP/1.1}{0.0.0.0:9080}
[2017-08-01T10:00:29,517+02:00]  [INFO ]  [Thread-13]  [o.e.jetty.server.ServerConnector]  Stopped admin@6d810e94{HTTP/1.1}{0.0.0.0:9081}
[2017-08-01T10:00:29,522+02:00]  [INFO ]  [Thread-13]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@5926f291{/,null,UNAVAILABLE}
[2017-08-01T10:00:29,557+02:00]  [INFO ]  [Thread-13]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@19b6b7e7{/,null,UNAVAILABLE}
[2017-08-01T10:00:29,560+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipelines]  Attempting to stop pipelines
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipeline]  Stopped pipeline [biz-outcome]
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipelines]  Pipelines have stopped
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.e.a.c.AsyncChunkerModule]  Stopping AsyncChunkingManager
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [pool-10-thread-1]  [c.a.a.p.e.a.c.AsyncChunkingManager]  Chunking State Logger stopped.
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.c.m.s.DefaultMetricCorrelationService]  Stopped metric correlation service.
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.c.m.s.FileBasedCorrelationPersistor]  Stopped metric correlation file cleanup thread.
[2017-08-01T10:00:30,571+02:00]  [INFO ]  [Thread-13]  [org.reflections.Reflections]  This message was throttled [1] times: Reflections took 2 ms to scan 1 urls, producing 2 keys and 2 values

[2017-08-01T10:00:30,571+02:00]  [WARN ]  [Thread-13]  [c.a.c.u.health.HealthReporterModule]  Task will be forcibly stopped now if it has not already stopped
[2017-08-01T10:00:30,572+02:00]  [INFO ]  [Thread-13]  [c.a.c.u.health.HealthReporterModule]  Stopped
[2017-08-01T10:00:30,572+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] stopping ...
[2017-08-01T10:00:30,627+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] stopped
[2017-08-01T10:00:30,628+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] closing ...
[2017-08-01T10:00:30,636+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] closed
[2017-08-01T10:00:30,640+02:00]  [INFO ]  [Thread-13]  [c.a.common.framework.util.SimpleApp]  Stopped [events-service-api-store]

And corresponding platform-admin-server.log:

ERROR [2017-08-01 10:21:47,330] com.appdynamics.platformadmin.resources.filters.ESClusterStateValidationFilter: Invalid cluster state. : running

Regards

[Mohammed.Rayan]

Kin,

Thanks for the snippet but can you share the complete log file to review the preceding entries to understand why it was stopped along with the events-service-api-store.properties file.

You can also use the below commands to retreive the logs

./platform-admin.sh retrieve-events-service-logs

Further, does this command list the events-service node fine for you

./platform-admin.sh list-events-service-nodes
./platform-admin.sh show-events-service-health

Regards,

Mohammed Rayan

[kin]

Hi,

The events service stopped because I restarted the server.

apd@apd-controller:~$ ./platform-admin.sh list-events-service-nodes
Listing Events Service nodes.
Cluster Member Id: 2, Cluster Member Host Address: apd-events, Cluster Member Running State: running, Cluster Member Type: api-store

Completed listing all the nodes.

apd@apd-controller:~$ ./platform-admin.sh show-events-service-health
Events Service cluster is not healthy.
Failure occurred at stage: es_cluster_health_stage
Error message:
Error reaching cluster

Regards

[Mohammed.Rayan]

Kin,

Can you please share the complete logs for us to review to understand the reason behind why it went to stopped state

Regards,

Mohammed Rayan

[kin]

Logs attached.

[Mohammed.Rayan · Accepted solution]

Kin,

I dont see anything thats stands obvious in the logs to know the reason why its not starting up,so can you try to start it manually and let us know if it starts up fine or not

1. cd <events_service_install_dir>

2. Execute the below command and make sure to set the JAVA_HOME variable before executing this command

export JAVA_HOME=<Path to your JRE>
nohup bin/events-service.sh start -p conf/events-service-api-store.properties &

If the issue persists, Please share the latest logs to review it further

Regards,

Mohammed Rayan