cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Error when trying to create event service - Connection Refused Error

Luiz.Polli
New Poster

Hi experts,

I have created a new event service in a demo deployment, but it is failing to start up. I'm seeing some errors for failed to connect to in platform-admin-server.log.

 

INFO  [2023-05-11 18:24:50,425] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:24:50,426] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:24:51,742] com.appdynamics.platformadmin.resources.VersionResource: Found Enterprise Console version 23.4.0-10041, build

INFO  [2023-05-11 18:24:55,441] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:24:55,442] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:25:00,442] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:25:00,443] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

INFO  [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping

WARN  [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].

 

Error_Event_Service_Not_Starting.png

Could anyone help me on that?

11 REPLIES 11

Ryan.Paredez
Community Manager

Hi @Luiz.Polli,

I did some digging around and what I found was you need to also look into the events-service-api-store.log and the appdynamics-events-service-cluster.log file from the remove events service host to review this issue further.

For another member, this was the issue they had.

The issue is due to the presence of a stale id file and so can you get remove it and then restart it from the econsole

Remove this file

/usr/local/appdynamics/platform/product/events-service/processor/bin/../events-service-api-store.id


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hello Ryan,
I tried to remove the events-service-api-store.id and did not work. Attached the log files. But something is wrong with connection refused to elasticsearch service.

The log below is for events-service-api-store.log

[2023-05-15T15:20:22,082-03:00]  [INFO ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Attempt [2]

[2023-05-15T15:20:22,091-03:00]  [WARN ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Error occurred while attempting to force cluster settings. Retry attempts remaining [1]
java.net.ConnectException: Connection refused
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        ... 1 common frames omitted

[2023-05-15T15:20:22,091-03:00]  [INFO ]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Attempt [3]

[2023-05-15T15:20:22,097-03:00]  [ERROR]  [elasticsearch-config-enforcer-0]  [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule]  Error occurred while attempting to force cluster settings
java.net.ConnectException: Connection refused
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
        at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
        at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        ... 1 common frames omitted
[root@appd-controller logs]#

The log below is for appdynamics-events-service-cluster.log 

[root@appd-controller logs]# tail -200 appdynamics-events-service-cluster.log
[2023-05-15T15:16:32,832][WARN ][o.e.c.l.LogConfigurator  ] [appd-controller] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /home/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-05-15T15:16:36,705][ERROR][o.e.b.Elasticsearch      ] [appd-controller] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
[root@appd-controller logs]#

Hope that information helps to find a solution.

Thank you,
Luiz Polli

Ryan.Paredez
Community Manager

Hi @Luiz.Polli,

Let's see if the Community can jump in and help out further. If you also discover something, please share that here too.


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Hi Ryan, 

I'm also seeing this same error when installing Event Service  locally on the Enterprise Console host (centos 8 stream).   I installed the EC using  platform-setup-x64-linux-23.4.0.10041.sh   The  appdynamics-events-service-cluster.log shows the below:

[root@centos8stream logs]# cat appdynamics-events-service-cluster.log
[2023-07-11T03:26:32,998][WARN ][o.e.c.l.LogConfigurator ] [centos8stream.localdomain] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/opt/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-07-11T03:26:33,179][ERROR][o.e.b.Elasticsearch ] [centos8stream.localdomain] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]

 

Checking "netstat -an",  port 9080 is not showing - no surprises there.

Also note that " /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin" must be run as root user, otherwise it returns the error:

[appd@centos8stream appdynamics]$ /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin
Enterprise Console must be operated with user 'root'. Current user is appd

Ryan.Paredez
Community Manager

Hi @Michael.Lee,

Thanks for sharing. Since the community was not able to jump in and help out @Luiz.Polli I would recommend contacting Support or your AppD rep for more help with this.

Unless @Luiz.Polli was able to find a solution and can share it here. 

 


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Luiz.Polli
Builder

Hello team and sorry for the late response.
Sharing with you some steps that solved this issue.

1- First, you will need to create any user differently from the root. If you are trying to start from root user, it will fail.
2- Set the write, read and executable privileges for this new user to the appd directory.
3- Then, with the new user, change the property values in your events-services-api-store.properties file to IP address instead of hostnames.

egrep "dw.http.host|es.node.unicast.hosts|es.cluster.initial_master_nodes|es.cluster.http.hosts" ../conf/events-service-api-store.properties
ad.dw.http.host=X.X.X.X
ad.es.node.unicast.hosts=X.X.X.X:9300
ad.es.cluster.initial_master_nodes=X.X.X.X
ad.es.cluster.http.hosts:X.X.X.X:9200

4- Add the following environment variable to the new user.

export INSTALL_BOOTSTRAP_MASTER_ES8=true

 5- Restart the Event-Service(ES) manually. Proceed with the steps below:
(From events-service/processor directory) -> Need to run from this directory otherwise will fail

bin/events-service.sh stop -f && rm -r events-service-api-store.id && rm -r elasticsearch.id

If you have more than one cluster, run the following command on each node in the cluster. In my setup there is only one ES node.

nohup bin/events-service.sh start -p conf/events-service-api-store.properties &

After two or three minutes, check the opened ports.

netstat -ant|egrep "9080|9081|9200|9300"
-a -> all ports
-n -> numeric ports
-t -> tcp ports


Thanks,
Luiz Polli






Luiz.Polli
Builder

Follow the case link opened for this issue. But I'm not sure if you will have access to it.
https://help.appdynamics.com/hc/en-us/requests/367007

Ryan.Paredez
Community Manager

Hi @Luiz.Polli,

Other community members do not have access to view your ticket, so thank you for sharing a bit more detail in the post above. Is there are any other details you can share, please do so as a reply to the post so it can be seen.

Thanks!


Thanks,

Ryan, Cisco AppDynamics Community Manager




Found something helpful? Click the Accept as Solution button to help others find answers faster.

Liked something? Click the Thumbs Up button.



Check out Observabiity in Action

new deep dive videos weekly in the Knowledge Base.

Luiz.Polli
Builder

Hi Ryan,

That's all from my side. I hope it helps.

Thanks,
Luiz Polli