Starting Events Service cluster

[Abdulrahman.Kazamel]

I'm trying to start the Events service nodes but it doesn't work

Spoiler

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --
job start

( 1/ 7) Load Events Service cluster configuration: SUCCESS
( 2/ 7) Load all ES cluster hosts into execution context.: SUCCESS

( 3/ 7) Start Events Service Cluster: Starting the Events Service api store node
( 3/ 7) Start Events Service Cluster: FAILED
Job failed to execute.
Failure occurred: Start Events Service Cluster
Error message:
Task failed: Starting the Events Service api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/_ping] failed due to [Failed to connect to /10.0.30.45:9080].

I also tried to start each node alone I get this error

Spoiler

pwd
/opt/appdynamics/platform/product/events-service/processor
[root@events-service-03 processor]# echo $JAVA_HOME
/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/
[root@events-service-03 processor]# ./bin/events-service.sh start -p ./conf/events-service-api-store.properties &
[1] 43742
[root@events-service-03 processor]# ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
[2021-06-01T10:52:52,627+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..]
[2021-06-01T10:52:52,631+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..]
[2021-06-01T10:52:52,636+02:00] Program arguments:

|-Xmx2990m|-Xms2990m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-Djute.maxbuffer=30000000|-Dzookeeper.snapshot.trust.empty=true|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField|-XX:CompileCommand=exclude,org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregator$1.collect|-XX:CompileCommand=exclude,org/apache/lucene/index/SortedNumericDocValuesWriter.flush|-XX:CompileCommand=exclude,org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm|-Xloggc:"/opt/appdynamics/platform/product/events-service/processor/bin/../logs/%p-gc.log"|-DAPPLICATION_HOME="/opt/appdynamics/platform/product/events-service/processor/bin/.."|-classpath|"/opt/appdynamics/platform/product/events-service/processor/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties"|-y|"/opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml"

CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField
CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField
CompilerOracle: exclude org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregatorstart.collect
CompilerOracle: exclude org/apache/lucene/index/SortedNumericDocValuesWriter.flush
CompilerOracle: exclude org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm
10:52:52.934 [main] INFO com.appdynamics.analytics.processor.AnalyticsService - Starting analytics processor with arguments [-p, /opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties, -y, /opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.

[Ryan.Paredez]

Hi @Abdulrahman.Kazamel,

I couldn't find any relevant info on those errors, but I did find this AppD Knowledge base article that describes how to turn on debugging mode. Perhaps this will give you more insight into the issue.

https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-enable-the-Events-Service-debugging-por...

[Abdulrahman.Kazamel]

Hi @Ryan.Paredez 

I made some edit like 

ssh by hostname, not only with the IP address

maximize each node to 25 G ram 

reinstalled the events service cluster by 

bin/platform-admin.sh install-events-service --profile prod --hosts 10.0.30.45 10.0.30.46 10.0.30.47 --data-dir /opt/appdynamics/eventsservice --platform-name AppDPlatform

also in each machine 

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9080 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9081 -j ACCEPT
iptables -A INPUT -m state --state NEW -m multiport -p tcp --dports 9300:9400 -j ACCEPT

however, the firewall was off

and tried again to restarted the cluster by 

Spoiler

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --job restart-cluster
( 1/ 6) Load Events Service cluster configuration: SUCCESS
( 2/ 6) Load all ES cluster hosts into execution context.: SUCCESS
( 3/ 6) Disable transient allocations on masters: Disable transient allocation in the api store node
( 3/ 6) Disable transient allocations on masters: SUCCESS
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: SUCCESS
( 5/ 6) Enable transient allocations on masters: Enable transient allocation in the api store node
( 5/ 6) Enable transient allocations on masters: FAILED
Job failed to execute.
Failure occurred: Enable transient allocations on masters
Error message:
Task failed: Enable transient allocation in the api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/v1/elasticsearch/_cluster/settings] failed due to [Failed to connect to /10.0.30.45:9080].

[Ryan.Paredez]

Hi @Abdulrahman.Kazamel,

Thanks for the additional info. Let's see if the Community can jump in and help you out. In the meantime, if you happen to find a solution, please do come back and share the results here. Knowledge sharing is key to the peer-to-peer community. 

[Abdulrahman.Kazamel · Accepted solution]

Hello @Ryan.Paredez 

I made a couple of configuration and at the end 

each time I do the installation the ports in each machine changes to 0.0.0.0 

For now, the issue was in DNS resolve as I had to make it manual in each machine at
/etc/hosts file
to give an example

eventsservice-01 10.0.30.xx
eventsservice-02 10.0.30.xx
eventsservice-03 10.0.30.xx

the installation done successfully after this configuration and yet still checking the health as it's not good.

*another update *

after the last configuration, the events service were not able to keep running so,

I changed every hostname to IP in the conf/events-service-api-store.properties in each machine.
I stopped/disabled the firewall and iptables,
and now it's up and running in green health since two hours!

kindly I would like to thank you for your support.

Thanks
Abdulrahman