cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Starting Events Service cluster

I'm trying to start the Events service nodes but it doesn't work

Spoiler

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --
job start

( 1/ 7) Load Events Service cluster configuration: SUCCESS
( 2/ 7) Load all ES cluster hosts into execution context.: SUCCESS

( 3/ 7) Start Events Service Cluster: Starting the Events Service api store node
( 3/ 7) Start Events Service Cluster: FAILED
Job failed to execute.
Failure occurred: Start Events Service Cluster
Error message:
Task failed: Starting the Events Service api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/_ping] failed due to [Failed to connect to /10.0.30.45:9080].

 

I also tried to start each node alone I get this error

Spoiler

pwd
/opt/appdynamics/platform/product/events-service/processor
[root@events-service-03 processor]# echo $JAVA_HOME
/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/
[root@events-service-03 processor]# ./bin/events-service.sh start -p ./conf/events-service-api-store.properties &
[1] 43742
[root@events-service-03 processor]# ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
[2021-06-01T10:52:52,627+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..]
[2021-06-01T10:52:52,631+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..]
[2021-06-01T10:52:52,636+02:00] Program arguments:

|-Xmx2990m|-Xms2990m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-Djute.maxbuffer=30000000|-Dzookeeper.snapshot.trust.empty=true|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField|-XX:CompileCommand=exclude,org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregator$1.collect|-XX:CompileCommand=exclude,org/apache/lucene/index/SortedNumericDocValuesWriter.flush|-XX:CompileCommand=exclude,org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm|-Xloggc:"/opt/appdynamics/platform/product/events-service/processor/bin/../logs/%p-gc.log"|-DAPPLICATION_HOME="/opt/appdynamics/platform/product/events-service/processor/bin/.."|-classpath|"/opt/appdynamics/platform/product/events-service/processor/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties"|-y|"/opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml"

CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField
CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField
CompilerOracle: exclude org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregatorstart.collect
CompilerOracle: exclude org/apache/lucene/index/SortedNumericDocValuesWriter.flush
CompilerOracle: exclude org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm
10:52:52.934 [main] INFO com.appdynamics.analytics.processor.AnalyticsService - Starting analytics processor with arguments [-p, /opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties, -y, /opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.

 

 

Solution Architect
11 REPLIES 11

Ryan.Paredez
Community Manager

Hi @Abdulrahman.Kazamel,

 

I couldn't find any relevant info on those errors, but I did find this AppD Knowledge base article that describes how to turn on debugging mode. Perhaps this will give you more insight into the issue.

 

https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-enable-the-Events-Service-debugging-por...

 

 


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hi @Ryan.Paredez 

I made some edit like 

ssh by hostname, not only with the IP address

maximize each node to 25 G ram 

 

reinstalled the events service cluster by 

bin/platform-admin.sh install-events-service --profile prod --hosts 10.0.30.45 10.0.30.46 10.0.30.47 --data-dir /opt/appdynamics/eventsservice --platform-name AppDPlatform

 

also in each machine 

 

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9080 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9081 -j ACCEPT
iptables -A INPUT -m state --state NEW -m multiport -p tcp --dports 9300:9400 -j ACCEPT

 

however, the firewall was off

and tried again to restarted the cluster by 

Spoiler
[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --job restart-cluster
( 1/ 6) Load Events Service cluster configuration: SUCCESS
( 2/ 6) Load all ES cluster hosts into execution context.: SUCCESS
( 3/ 6) Disable transient allocations on masters: Disable transient allocation in the api store node
( 3/ 6) Disable transient allocations on masters: SUCCESS
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: SUCCESS
( 5/ 6) Enable transient allocations on masters: Enable transient allocation in the api store node
( 5/ 6) Enable transient allocations on masters: FAILED
Job failed to execute.
Failure occurred: Enable transient allocations on masters
Error message:
Task failed: Enable transient allocation in the api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/v1/elasticsearch/_cluster/settings] failed due to [Failed to connect to /10.0.30.45:9080].

 

 

 

Solution Architect

Hi @Abdulrahman.Kazamel,

 

Thanks for the additional info. Let's see if the Community can jump in and help you out. In the meantime, if you happen to find a solution, please do come back and share the results here. Knowledge sharing is key to the peer-to-peer community. 

 

 


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hello @Ryan.Paredez 

 

I made a couple of configuration and at the end 

each time I do the installation the ports in each machine changes to 0.0.0.0 

 

For now, the issue was in DNS resolve as I had to make it manual in each machine at
/etc/hosts file
to give an example

eventsservice-01 10.0.30.xx
eventsservice-02 10.0.30.xx
eventsservice-03 10.0.30.xx


the installation done successfully after this configuration and yet still checking the health as it's not good.

 

*another update *


after the last configuration, the events service were not able to keep running so,

I changed every hostname to IP in the conf/events-service-api-store.properties in each machine.
I stopped/disabled the firewall and iptables,
and now it's up and running in green health since two hours!

 

kindly I would like to thank you for your support.

Thanks
Abdulrahman

Solution Architect

Hi @Abdulrahman.Kazamel,

I also encountered the similar issue as follows:

Screenshot 2023-06-07 141532.png

I want to verify, what you did to solve the issue are the following:

- add the event service machine to /etc/hosts file

- changed every hostname to IP in the conf/events-service-api-store.properties

Am I correct? Or is there any other changes need to be done.

Thank you.

Aribowo

Ryan.Paredez
Community Manager

Hi @Aribowo.Soegiarto,

Since this post is about 2 years old, you may want to reask the question on the forums, or you can reach out to your AppD Rep, or even contact AppD Support. 

 


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hi Aribowo,

Same issue here.

Thanks,

Luiz Polli

Ryan.Paredez
Community Manager

Hi @Luiz.Polli,

As I mentioned to @Aribowo.Soegiarto, given how old this post is, I think contacting AppD support is best. I was not able to find a ticket created by Aribowo. 



Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hi Ryan,
Creating a ticket now.

Thank you,
Luiz Polli