Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
05-10-2023 04:49 AM - edited 05-11-2023 03:13 PM
Hi experts,
I have created a new event service in a demo deployment, but it is failing to start up. I'm seeing some errors for failed to connect to in platform-admin-server.log.
INFO [2023-05-11 18:24:50,425] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping
WARN [2023-05-11 18:24:50,426] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].
INFO [2023-05-11 18:24:51,742] com.appdynamics.platformadmin.resources.VersionResource: Found Enterprise Console version 23.4.0-10041, build
INFO [2023-05-11 18:24:55,441] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping
WARN [2023-05-11 18:24:55,442] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].
INFO [2023-05-11 18:25:00,442] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping
WARN [2023-05-11 18:25:00,443] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].
INFO [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Sending request to: http://appd-controller:9080/_ping
WARN [2023-05-11 18:25:05,468] com.appdynamics.orcha.modules.modules.UriExec: Connection to [http://appd-controller:9080/_ping] failed due to [Failed to connect to appd-controller/192.168.0.17:9080].
Could anyone help me on that?
Solved! Go to Solution.
05-14-2023 04:27 AM
Hi @Luiz.Polli,
I did some digging around and what I found was you need to also look into the events-service-api-store.log and the appdynamics-events-service-cluster.log file from the remove events service host to review this issue further.
For another member, this was the issue they had.
The issue is due to the presence of a stale id file and so can you get remove it and then restart it from the econsole
Remove this file
/usr/local/appdynamics/platform/product/events-service/processor/bin/../events-service-api-store.id
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
05-16-2023 04:51 AM
Hello Ryan,
I tried to remove the events-service-api-store.id and did not work. Attached the log files. But something is wrong with connection refused to elasticsearch service.
The log below is for events-service-api-store.log
[2023-05-15T15:20:22,082-03:00] [INFO ] [elasticsearch-config-enforcer-0] [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule] Attempt [2]
[2023-05-15T15:20:22,091-03:00] [WARN ] [elasticsearch-config-enforcer-0] [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule] Error occurred while attempting to force cluster settings. Retry attempts remaining [1]
java.net.ConnectException: Connection refused
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 common frames omitted
[2023-05-15T15:20:22,091-03:00] [INFO ] [elasticsearch-config-enforcer-0] [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule] Attempt [3]
[2023-05-15T15:20:22,097-03:00] [ERROR] [elasticsearch-config-enforcer-0] [c.a.a.p.e.i.m.OneToMultiNodeUpgradeEnforcerModule] Error occurred while attempting to force cluster settings
java.net.ConnectException: Connection refused
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1139)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1156)
at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule.getIndexSettings(OneToMultiNodeUpgradeEnforcerModule.java:79)
at com.appdynamics.analytics.processor.elasticsearch.index.maintenance.OneToMultiNodeUpgradeEnforcerModule$EnforcerTask.run(OneToMultiNodeUpgradeEnforcerModule.java:220)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 common frames omitted
[root@appd-controller logs]#
The log below is for appdynamics-events-service-cluster.log
[root@appd-controller logs]# tail -200 appdynamics-events-service-cluster.log
[2023-05-15T15:16:32,832][WARN ][o.e.c.l.LogConfigurator ] [appd-controller] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/home/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-05-15T15:16:36,705][ERROR][o.e.b.Elasticsearch ] [appd-controller] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
[root@appd-controller logs]#
Hope that information helps to find a solution.
Thank you,
Luiz Polli
05-19-2023 03:22 AM
Hi @Luiz.Polli,
Let's see if the Community can jump in and help out further. If you also discover something, please share that here too.
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
07-11-2023 12:45 AM
Hi Ryan,
I'm also seeing this same error when installing Event Service locally on the Enterprise Console host (centos 8 stream). I installed the EC using platform-setup-x64-linux-23.4.0.10041.sh The appdynamics-events-service-cluster.log shows the below:
[root@centos8stream logs]# cat appdynamics-events-service-cluster.log
[2023-07-11T03:26:32,998][WARN ][o.e.c.l.LogConfigurator ] [centos8stream.localdomain] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/opt/appdynamics/platform/product/events-service/processor/elasticsearch/config/log4j2.properties
[2023-07-11T03:26:33,179][ERROR][o.e.b.Elasticsearch ] [centos8stream.localdomain] fatal exception while booting Elasticsearch
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:260) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
Checking "netstat -an", port 9080 is not showing - no surprises there.
Also note that " /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin" must be run as root user, otherwise it returns the error:
[appd@centos8stream appdynamics]$ /opt/appdynamics/platform/platform-admin/bin/platform-admin.sh start-platform-admin
Enterprise Console must be operated with user 'root'. Current user is appd
07-11-2023 03:21 PM
Hi @Michael.Lee,
Thanks for sharing. Since the community was not able to jump in and help out @Luiz.Polli I would recommend contacting Support or your AppD rep for more help with this.
Unless @Luiz.Polli was able to find a solution and can share it here.
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
07-13-2023 12:09 PM
Hello team and sorry for the late response.
Sharing with you some steps that solved this issue.
1- First, you will need to create any user differently from the root. If you are trying to start from root user, it will fail.
2- Set the write, read and executable privileges for this new user to the appd directory.
3- Then, with the new user, change the property values in your events-services-api-store.properties file to IP address instead of hostnames.
egrep "dw.http.host|es.node.unicast.hosts|es.cluster.initial_master_nodes|es.cluster.http.hosts" ../conf/events-service-api-store.properties
ad.dw.http.host=X.X.X.X
ad.es.node.unicast.hosts=X.X.X.X:9300
ad.es.cluster.initial_master_nodes=X.X.X.X
ad.es.cluster.http.hosts:X.X.X.X:9200
4- Add the following environment variable to the new user.
export INSTALL_BOOTSTRAP_MASTER_ES8=true
5- Restart the Event-Service(ES) manually. Proceed with the steps below:
(From events-service/processor directory) -> Need to run from this directory otherwise will fail
bin/events-service.sh stop -f && rm -r events-service-api-store.id && rm -r elasticsearch.id
If you have more than one cluster, run the following command on each node in the cluster. In my setup there is only one ES node.
nohup bin/events-service.sh start -p conf/events-service-api-store.properties &
After two or three minutes, check the opened ports.
netstat -ant|egrep "9080|9081|9200|9300"
-a -> all ports
-n -> numeric ports
-t -> tcp ports
Thanks,
Luiz Polli
07-13-2023 12:13 PM
Follow the case link opened for this issue. But I'm not sure if you will have access to it.
https://help.appdynamics.com/hc/en-us/requests/367007
07-14-2023 10:00 AM
Hi @Luiz.Polli,
Other community members do not have access to view your ticket, so thank you for sharing a bit more detail in the post above. Is there are any other details you can share, please do so as a reply to the post so it can be seen.
Thanks!
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
07-14-2023 10:34 AM
Hi Ryan,
That's all from my side. I hope it helps.
Thanks,
Luiz Polli
User | Count |
---|---|
1 | |
1 | |
1 | |
1 | |
1 |
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form