Error when trying to create event service - Connection Refused Error

[Luiz.Polli]

Hi experts,

I have created a new event service in a demo deployment, but it is failing to start up. I'm seeing some errors for failed to connect to in platform-admin-server.log.

INFO  [2023-05-11 18:24:50,425] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:24:50,426] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:24:51,742] com.appdynamics.platformadmin.resources.VersionResource: Found Enterprise Console version 23.4.0-10041, build

INFO  [2023-05-11 18:24:55,441] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:24:55,442] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:25:00,442] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:25:00,443] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

Could anyone help me on that?

[Ryan.Paredez]

Hi @Luiz.Polli,

I did some digging around and what I found was you need to also look into the events-service-api-store.log and the appdynamics-events-service-cluster.log file from the remove events service host to review this issue further.

For another member, this was the issue they had.

The issue is due to the presence of a stale id file and so can you get remove it and then restart it from the econsole

Remove this file

/usr/local/appdynamics/platform/product/events-service/processor/bin/../events-service-api-store.id

[Luiz.Polli]

Hello Ryan,
I tried to remove the events-service-api-store.id and did not work. Attached the log files. But something is wrong with connection refused to elasticsearch service.

The log below is for events-service-api-store.log

[2023-05-15T15:20:22,082-03:00]  [INFO ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Attempt [2]

[2023-05-15T15:20:22,091-03:00]  [WARN ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Error occurred while attempting to force cluster settings. Retry attempts remaining [1]
java.net.ConnectException: Connection refused
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        ... 1 common frames omitted

[2023-05-15T15:20:22,091-03:00]  [INFO ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Attempt [3]

[2023-05-15T15:20:22,097-03:00]  [ERROR]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Error occurred while attempting to force cluster settings
java.net.ConnectException: Connection refused
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        ... 1 common frames omitted
[root@appd-controller logs]#

The log below is for appdynamics-events-service-cluster.log 

[root@appd-controller logs]# tail -200 appdynamics-events-service-cluster.log
[2023-05-15T15:16:32,832][WARN ][o.e.c.l.LogConfigurator  ] [appd-controller] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /home/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-05-15T15:16:36,705][ERROR][o.e.b.Elasticsearch      ] [appd-controller] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
[root@appd-controller logs]#

Hope that information helps to find a solution.

Thank you,
Luiz Polli

[Ryan.Paredez]

Hi @Luiz.Polli,

Let's see if the Community can jump in and help out further. If you also discover something, please share that here too.

[Michael.Lee]

Hi Ryan, 

I'm also seeing this same error when installing Event Service  locally on the Enterprise Console host (centos 8 stream).   I installed the EC using  platform-setup-x64-linux-23.4.0.10041.sh   The  appdynamics-events-service-cluster.log shows the below:

[root@centos8stream logs]# cat appdynamics-events-service-cluster.log
[2023-07-11T03:26:32,998][WARN ][o.e.c.l.LogConfigurator ] [centos8stream.localdomain] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/opt/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-07-11T03:26:33,179][ERROR][o.e.b.Elasticsearch ] [centos8stream.localdomain] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]

Checking "netstat -an",  port 9080 is not showing - no surprises there.

Also note that " /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin" must be run as root user, otherwise it returns the error:

[appd@centos8stream appdynamics]$ /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin
Enterprise Console must be operated with user 'root'. Current user is appd

[Ryan.Paredez]

Hi @Michael.Lee,

Thanks for sharing. Since the community was not able to jump in and help out @Luiz.Polli I would recommend contacting Support or your AppD rep for more help with this.

Unless @Luiz.Polli was able to find a solution and can share it here. 

[Luiz.Polli · Accepted solution]

Hello team and sorry for the late response.
Sharing with you some steps that solved this issue.

1- First, you will need to create any user differently from the root. If you are trying to start from root user, it will fail.
2- Set the write, read and executable privileges for this new user to the appd directory.
3- Then, with the new user, change the property values in your events-services-api-store.properties file to IP address instead of hostnames.

egrep "dw.http.host|es.node.unicast.hosts|es.cluster.initial_master_nodes|es.cluster.http.hosts" ../conf/events-service-api-store.properties
ad.dw.http.host=X.X.X.X
ad.es.node.unicast.hosts=X.X.X.X:9300
ad.es.cluster.initial_master_nodes=X.X.X.X
ad.es.cluster.http.hosts:X.X.X.X:9200

4- Add the following environment variable to the new user.

export INSTALL_BOOTSTRAP_MASTER_ES8=true

 5- Restart the Event-Service(ES) manually. Proceed with the steps below:
(From events-service/processor directory) -> Need to run from this directory otherwise will fail

bin/events-service.sh stop -f && rm -r events-service-api-store.id && rm -r elasticsearch.id

If you have more than one cluster, run the following command on each node in the cluster. In my setup there is only one ES node.

nohup bin/events-service.sh start -p conf/events-service-api-store.properties &

After two or three minutes, check the opened ports.

netstat -ant|egrep "9080|9081|9200|9300"
-a -> all ports
-n -> numeric ports
-t -> tcp ports

Thanks,
Luiz Polli

[Luiz.Polli]

Follow the case link opened for this issue. But I'm not sure if you will have access to it.
https://help.appdynamics.com/hc/en-us/requests/367007

[Ryan.Paredez]

Hi @Luiz.Polli,

Other community members do not have access to view your ticket, so thank you for sharing a bit more detail in the post above. Is there are any other details you can share, please do so as a reply to the post so it can be seen.

Thanks!

[Luiz.Polli]

Hi Ryan,

That's all from my side. I hope it helps.

Thanks,
Luiz Polli