cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Events Service failed to start. Command failed due to an error: Invalid cluster state.

kin
New Poster

Hello,

I'm trying to setup an on-premise AppDynamics platform. I followed documentation and installed events service on a dedicated machine without errors. So right now I have a controller server and events service server. But when I try to launch the events service via platform-admin I'm getting following error:

apd@apd-controller:~$ ./platform-admin.sh start-events-service
Starting Events Service.
Events Service failed to start.Command failed due to an error: Invalid cluster state.
API code 400

 Any ideas what is wrong with my cluster?

 

Regards

7 REPLIES 7

Mohammed.Rayan
AppDynamics Team

Kim,

Can you please share the logs from your events-service host (events-service-api-store.log) & the platform-admin-server.log from the platform admin host

 

 

For instance: cd <events_service_install_dir>/logs/events-service-api-store.log

 

Regards,

Mohammed Rayan



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Excerpt from events-service-api-store.log:

[2017-08-01T10:00:29,499+02:00]  [INFO ]  [Thread-13]  [c.a.common.framework.util.SimpleApp]  Stopping [events-service-api-store]
[2017-08-01T10:00:29,510+02:00]  [INFO ]  [Thread-13]  [o.e.jetty.server.ServerConnector]  Stopped application@71739267{HTTP/1.1}{0.0.0.0:9080}
[2017-08-01T10:00:29,517+02:00]  [INFO ]  [Thread-13]  [o.e.jetty.server.ServerConnector]  Stopped admin@6d810e94{HTTP/1.1}{0.0.0.0:9081}
[2017-08-01T10:00:29,522+02:00]  [INFO ]  [Thread-13]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@5926f291{/,null,UNAVAILABLE}
[2017-08-01T10:00:29,557+02:00]  [INFO ]  [Thread-13]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@19b6b7e7{/,null,UNAVAILABLE}
[2017-08-01T10:00:29,560+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipelines]  Attempting to stop pipelines
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipeline]  Stopped pipeline [biz-outcome]
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.pipeline.framework.Pipelines]  Pipelines have stopped
[2017-08-01T10:00:29,567+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.e.a.c.AsyncChunkerModule]  Stopping AsyncChunkingManager
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [pool-10-thread-1]  [c.a.a.p.e.a.c.AsyncChunkingManager]  Chunking State Logger stopped.
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.c.m.s.DefaultMetricCorrelationService]  Stopped metric correlation service.
[2017-08-01T10:00:29,568+02:00]  [INFO ]  [Thread-13]  [c.a.a.p.c.m.s.FileBasedCorrelationPersistor]  Stopped metric correlation file cleanup thread.
[2017-08-01T10:00:30,571+02:00]  [INFO ]  [Thread-13]  [org.reflections.Reflections]  This message was throttled [1] times: Reflections took 2 ms to scan 1 urls, producing 2 keys and 2 values

[2017-08-01T10:00:30,571+02:00]  [WARN ]  [Thread-13]  [c.a.c.u.health.HealthReporterModule]  Task will be forcibly stopped now if it has not already stopped
[2017-08-01T10:00:30,572+02:00]  [INFO ]  [Thread-13]  [c.a.c.u.health.HealthReporterModule]  Stopped
[2017-08-01T10:00:30,572+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] stopping ...
[2017-08-01T10:00:30,627+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] stopped
[2017-08-01T10:00:30,628+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] closing ...
[2017-08-01T10:00:30,636+02:00]  [INFO ]  [Thread-13]  [org.elasticsearch.node]  [node-05db522f-d124-4da0-992d-85dc3d6d0714] closed
[2017-08-01T10:00:30,640+02:00]  [INFO ]  [Thread-13]  [c.a.common.framework.util.SimpleApp]  Stopped [events-service-api-store]

And corresponding platform-admin-server.log:

ERROR [2017-08-01 10:21:47,330] com.appdynamics.platformadmin.resources.filters.ESClusterStateValidationFilter: Invalid cluster state. : running

Regards

Kin,

Thanks for the snippet but can you share the complete log file to review the preceding entries to understand why it was stopped along with the events-service-api-store.properties file.

 

 

You can also use the below commands to retreive the logs

./platform-admin.sh retrieve-events-service-logs

 

Further, does this command list the events-service node fine for you

 

 

./platform-admin.sh list-events-service-nodes
./platform-admin.sh show-events-service-health

 

Regards,

Mohammed Rayan

 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Hi,

The events service stopped because I restarted the server.

apd@apd-controller:~$ ./platform-admin.sh list-events-service-nodes
Listing Events Service nodes.
Cluster Member Id: 2, Cluster Member Host Address: apd-events, Cluster Member Running State: running, Cluster Member Type: api-store

Completed listing all the nodes.
apd@apd-controller:~$ ./platform-admin.sh show-events-service-health
Events Service cluster is not healthy.
Failure occurred at stage: es_cluster_health_stage
Error message:
Error reaching cluster

Regards

Kin,

Can you please share the complete logs for us to review to understand the reason behind why it went to stopped state

 

Regards,

Mohammed Rayan



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Logs attached.

Kin,

I dont see anything thats stands obvious in the logs to know the reason why its not starting up,so can you try to start it manually and let us know if it starts up fine or not

 

1. cd <events_service_install_dir>

2. Execute the below command and make sure to set the JAVA_HOME variable before executing this command

export JAVA_HOME=<Path to your JRE>
nohup bin/events-service.sh start -p conf/events-service-api-store.properties &

If the issue persists, Please share the latest logs to review it further

 

Regards,

Mohammed Rayan



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.