- Community Hub
- Forum Q&A
- Business iQ (Analytics)
- Controller (SaaS, On Premise)
- Dashboards
- Dynamic Languages (Node.JS, Python, PHP, C/C++, Webserver Agent)
- End User Monitoring (EUM)
- Infrastructure (Server, Network, Database)
- Java (Java Agent, Installation, JVM, and Controller Installation)
- Licensing (including Trial)
- .NET (Agent, Installation)
- Smart Agent
- General Discussions
- Resources
- Groups
- Idea Exchange
Not a customer?
Start a free trial
Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Cisco AppDynamics Community
- Forums Q&A
- Controller
- Events service shows critical after fresh installa...
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2023 01:19 AM
I have installed events service in same EC, controller and EUM node for demo purpose but after event service installation the service shows as critical in and then stopped.
Please advise what could the reason due for this. I have installed the platform componets on windows server 2019 os.
[2023-12-03T12:11:14,393+03:00] [ERROR] [dw-58 - PUT /entitysearch/sync] [c.a.a.s.r.e.j.m.UnknownExceptionMapper] Unknown exception occurred while processing HTTP request. logCorrelationId=[946fed22-05db-4370-aa1f-3ce469cb4689] [{}]
java.lang.RuntimeException: java.net.ConnectException
at com.appdynamics.analytics.processor.entitysearch.store.EntitySearchElasticStore.sync(EntitySearchElasticStore.java:114)
at com.appdynamics.analytics.processor.entitysearch.resource.EntitySearchResourceImpl.sync(EntitySearchResourceImpl.java:112)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: null
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:319)
at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:336)
at com.appdynamics.analytics.processor.entitysearch.store.EntitySearchElasticStore.sync(EntitySearchElasticStore.java:107)
... 70 common frames omitted
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 common frames omitted
[2023-12-03T12:11:30,392+03:00] [ERROR] [dw-58 - GET /v1/elasticsearch/numberOfDataNodes] [c.a.a.s.r.e.j.m.UnknownExceptionMapper] Unknown exception occurred while processing HTTP request. logCorrelationId=[a3823670-1430-4185-b355-b5c3be28244e] [{}]
java.lang.RuntimeException: java.net.ConnectException
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:388)
at com.appdynamics.analytics.processor.elasticsearch.settings.ElasticSearchSettingsResource.getNumDataNodes(ElasticSearchSettingsResource.java:77)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: null
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.cluster.ElasticsearchClusterClient.health(ElasticsearchClusterClient.java:334)
at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getDetailedClusterHealth(ElasticSearchAdminFacade.java:122)
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:386)
... 70 common frames omitted
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 common frames omitted
[2023-12-03T12:11:31,443+03:00] [ERROR] [dw-59 - GET /v1/elasticsearch/numberOfDataNodes] [c.a.a.s.r.e.j.m.UnknownExceptionMapper] Unknown exception occurred while processing HTTP request. logCorrelationId=[54da9d56-e317-4276-8488-8358eb796a14] [{}]
java.lang.RuntimeException: java.net.ConnectException
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:388)
at com.appdynamics.analytics.processor.elasticsearch.settings.ElasticSearchSettingsResource.getNumDataNodes(ElasticSearchSettingsResource.java:77)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: null
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.cluster.ElasticsearchClusterClient.health(ElasticsearchClusterClient.java:334)
at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getDetailedClusterHealth(ElasticSearchAdminFacade.java:122)
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:386)
... 70 common frames omitted
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 common frames omitted
[2023-12-03T12:11:41,557+03:00] [WARN ] [health-report-thread-0] [c.a.c.u.health.HealthReporterModule] Report unhealthy [
deadlocks: (healthy)
events-service-api-store / BizOutcome [stage.extract]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]
events-service-api-store / BizOutcome [stage.filter]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]
events-service-api-store / BizOutcome [stage.parse]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]
events-service-api-store / BizOutcome [stage.upsert]: (healthy) Rates (Avg per second. Avg of last 5 min) batch size: [0.000000], success: [0.000000], failure: [0.000000]
events-service-api-store / Build information: (healthy) buildName=release/23.7.0.onprem.next-Analytics-release2370onpremnext-23.7.0-246, buildHash=aee38fe8666e1090e98c850a093335567ad2bad4, buildTimestamp=2023-08-03T12:19:23+0000, buildNumber=23.7.0-246, jobName=Analytics-release2370onpremnext, version=23.7.0-246
events-service-api-store / Configuration properties: (healthy) Dynamic properties: [query.default.pagination.size=1000, query.default.scroll.batch.size=1000, ad.accountmanager.accountConfigCacheExpireSeconds=900, query.funnel.batch.scroll.expiry.time.millis=60000, query.default.pagination.offset=0, ad.accountmanager.key.admin_service=REDACTED, ad.es.event.index.maxBulkUpdateSizeBytes=5000000, query.default.results.limit=100, query.nested.enable=false, query.max.bucket.nesting.level=5, ad.es.event.maxUpsertRequestBodySizeBytes=5000000, ad.accountmanager.key.account_service=REDACTED, ad.es.event.index.maxBulkUpdateNumDocs=200, query.exact.analyzed.wildcard.enable=true, query.validator.max.batch.count=20, query.max.forced.range.size=2000, ad.accountmanager.key.disabledKeys=REDACTED, ad.es.event.index.fieldNumberIncrement=500, ad.accountmanager.cacheSize=5000, query.default.aggregation.level.limit=10, schema.validator.max.custom.events.per.account=20, query.max.pagination.offset=10000, ad.metric.processor.enabled.retry.onMpFailure=true, query.max.pagination.size=10000, query.max.results.series.limit=30000, ad.es.response.filter.double.bucketFunctionsRequiredToConvertKeyToLong=REDACTED, query.max.concat.arguments=100, ad.accountmanager.eumAccountCacheExpireSeconds=3600, query.max.results.limit=10000, query.standard.search.call.timeout.millis=300000, query.max.aggregation.level.limit=100, query.funnel.batch.enable.processing=false, ad.env=dev, ad.feature.adql.454.functions.values={"all": ["now","toDate","toFloat","toInt","toString","ifNull","round"], "none": []}, ad.es.event.index.fieldIncrementThresholdPercentage=80, ad.metric.processor.enabled.clusters=[], query.max.export.threads=10, ad.es.cluster.name=[appdynamics-events-service-cluster], ad.metric.processor.enabled.accounts=[], ad.accountmanager.keyNamesCSV=REDACTED, query.max.export.results.limit=65000, ad.es.response.filter.percentile.values.nullToZero=false, query.max.aggregation.level.limit.customers=10000, ad.metric.processor.enabled.configCreateAll=false, ad.es.event.index.isDocumentReplaceEnabled=false, ad.metric.processor.enabled.customEvents=false, query.max.scroll.batch.size=10000, ad.es.rolling.maxShardsPerIndex=25, query.inner.hits.size.limit=100, query.aggressive.search.call.timeout.millis=30000, ad.es.request.filter.partialPathsMap={"browserrecord": {"domreadytime": "metrics.domreadytime","enduserresponsetime": "metrics.enduserresponsetime","firstbytetime": "metrics.firstbytetime"},"mobilesessionrecord": {"durationMS": "metrics.durationMS"},"sessionrecord": {"durationMS": "metrics.durationMS","pagename": "browserRecords.pagename","pagetype": "browserRecords.pagetype"}}, ad.es.event.maxPublishRequestBodySizeBytes=1000000, ad.accountmanager.key.service=REDACTED, query.funnel.batch.scroll.size=10000, ad.es.event.index.maxFieldsPerIndex=3000, query.scroll.mode.enable=true, query.funnel.batch.concurrency.limit=3, ad.accountmanager.key.deprecatedKeys=REDACTED, ad.accountmanager.key.controller=REDACTED, ad.es.healthCheck.reservoir.expDecayFactor=0.015, ad.es.healthCheck.updateInterval=1 minutes, ad.metric.processor.disabled.accounts=[], query.default.scroll.expiry.millis=60000, ad.accountmanager.key.jf=REDACTED, ad.es.response.filter.boolean.eventTypesToFilter=[mobilesnapshot, mobilecrashreport, sessionrecord, mobilesessionrecord, synthsessionrecord], query.funnel.internal.results.limit=50000, query.max.top.level.aggregation.level.limit=1000, ad.es.healthCheck.reservoir.size=100, query.reject.time.unbounded.queries=false, ad.accountmanager.key.ops=REDACTED, ad.feature.adql.454.functions.envs={"default": "none", "dev": "all"}, ad.accountmanager.key.slm=REDACTED, query.cardinality.precision.threshold=-1, publish.validator.max.fields.per.event=255, ad.accountmanager.key.mds=REDACTED, ad.accountmanager.key.eum=REDACTED]
events-service-api-store / Connection to ElasticSearch: (unhealthy) java.net.ConnectException
java.lang.RuntimeException: java.net.ConnectException
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getNodesStatsResponse(ElasticSearchClientFacade.java:451)
at com.appdynamics.analytics.processor.elasticsearch.clusters.ClusterHealthStateReporter.isPartOfCluster(ClusterHealthStateReporter.java:38)
at com.appdynamics.analytics.processor.elasticsearch.clusters.AbstractClusterHealthStateReporter.updateHistogram(AbstractClusterHealthStateReporter.java:106)
at com.appdynamics.analytics.processor.elasticsearch.clusters.AbstractClusterHealthStateReporter.updateHealthCheckResult(AbstractClusterHealthStateReporter.java:93)
at com.appdynamics.common.util.health.AsynchronousHealthCheckable.runUpdateHealthCheckResult(AsynchronousHealthCheckable.java:123)
at com.appdynamics.common.util.health.AsynchronousHealthCheckable$2.run(AsynchronousHealthCheckable.java:147)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getNodesStatsResponse(ElasticSearchAdminFacade.java:443)
at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getNodesStatsResponse(ElasticSearchClientFacade.java:449)
... 11 more
Caused by: java.net.ConnectException
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 more
events-service-api-store / Connection to [http://0.0.0.0:9080/biz_outcome] with [DefaultBizOutcomeDefinitionClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/events/query] with [DefaultAsyncQueryEventsClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/events/query] with [DefaultQueryEventsClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/account] with [DefaultAccountServiceClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/accounts/meters] with [RestMeterServiceClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin/accounts] with [RestAccountsServiceClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin] with [AsyncJobScannerClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin] with [DefaultEventServiceAdminClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/jf] with [RestJobFrameworkClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/querymanagement/cancel] with [RestQueryManagementClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v1/slm] with [RestSlmPerfConfigsClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v3/accounts] with [RestEventTypeClient]: (healthy)
events-service-api-store / Connection to [http://0.0.0.0:9080/v3/events] with [DefaultEventServiceClient]: (healthy)
events-service-api-store / Resource [_ping - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.122272], user error: [0.000000], timeout: [0.000000], error: [0.000000]
events-service-api-store / Resource [entitysearch/sync - PUT]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.074595]
events-service-api-store / Resource [v1/account - POST]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.059934]
events-service-api-store / Resource [v1/elasticsearch/numberOfDataNodes - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.053746]
events-service-api-store / Resource [v1/store/report - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.074411], user error: [0.000000], timeout: [0.000000], error: [0.000000]
events-service-api-store / SystemAccessKeyAuthHandler: (healthy) deprecated: [], disabled: []
events-service-api-store / jobframework-module: (unhealthy) Job Framework instanceId: [SYS2.ZOOM.COM], number of jobs executed: [0], running since [<null>], currently executing jobs []
events-service-api-store / queues: (healthy) [1] queues [[biz-outcome-incoming-events] ratio: [0.00], size: [0], capacity: [1000]]
]
[2023-12-03T12:11:47,688+03:00] [INFO ] [cache-refresh-scheduler-0] [c.a.a.p.event.meter.DefaultMeters] Quota refresh executor stats : [java.util.concurrent.ThreadPoolExecutor@29bd2b0f[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]]
[2023-12-03T12:11:47,688+03:00] [INFO ] [cache-refresh-scheduler-0] [c.a.a.p.event.meter.DefaultMeters] Submitted [0] tasks to refresh the quota remaining caches
[2023-12-03T12:11:48,353+03:00] [ERROR] [main] [c.a.a.p.e.n.s.ElasticsearchDependencyModule] Elasticsearch was never healthy.
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 40 attempts.
at com.github.rholder.retry.Retryer.call(Retryer.java:174)
at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.waitForHealthiness(ElasticsearchDependencyModule.java:96)
at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.waitForHealthinessAndHandle(ElasticsearchDependencyModule.java:65)
at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.lambda$registerDependencyMonitor$0(ElasticsearchDependencyModule.java:50)
at io.dropwizard.lifecycle.setup.LifecycleEnvironment$ServerListener.lifeCycleStarted(LifecycleEnvironment.java:117)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:194)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:74)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
at com.appdynamics.common.framework.AbstractApp.callRunServer(AbstractApp.java:274)
at com.appdynamics.common.framework.AbstractApp.runUsingFile(AbstractApp.java:268)
at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:255)
at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:175)
at com.appdynamics.analytics.processor.AnalyticsService.main(AnalyticsService.java:76)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.appdynamics.common.executor.command.windows.AppRunServiceInternalCommand.execute(AppRunServiceInternalCommand.java:91)
at com.appdynamics.common.executor.CommandExecutor.execute(CommandExecutor.java:38)
at com.appdynamics.analytics.processor.executor.AnalyticsServiceExecutor.main(AnalyticsServiceExecutor.java:99)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.appdynamics.common.executor.standalone.ProxyMain.callActualMain(ProxyMain.java:165)
at com.appdynamics.common.executor.standalone.ProxyMain.main(ProxyMain.java:106)
Caused by: java.net.ConnectException: null
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.exists(ElasticsearchIndicesClient.java:620)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.exists(ElasticsearchIndicesClient.java:636)
at com.appdynamics.analytics.processor.util.startup.StartupHelpers.startupIndexExists(StartupHelpers.java:107)
at com.appdynamics.analytics.processor.util.startup.StartupHelpers.upsertStartupIndex(StartupHelpers.java:97)
at com.appdynamics.analytics.processor.util.startup.StartupHelpers.lambda$isElasticsearchHealthy$2(StartupHelpers.java:87)
at com.appdynamics.common.util.locks.InterProcessClusterLock.acquireAndExecute(InterProcessClusterLock.java:35)
at com.appdynamics.analytics.processor.util.startup.StartupHelpers.isElasticsearchHealthy(StartupHelpers.java:85)
at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.lambda$waitForHealthiness$2(ElasticsearchDependencyModule.java:101)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
... 29 common frames omitted
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.base/java.lang.Thread.run(Unknown Source)
[2023-12-03T12:11:48,354+03:00] [ERROR] [main] [c.a.a.p.e.n.s.ElasticsearchDependencyModule] Elasticsearch never successfully started up, stopping server startup.
[2023-12-03T12:11:48,356+03:00] [INFO ] [main] [c.a.common.framework.util.SimpleApp] Stopping [events-service-api-store]
[2023-12-03T12:11:48,384+03:00] [INFO ] [main] [o.e.jetty.server.AbstractConnector] Stopped application@4cf1ec2{HTTP/1.1, (http/1.1)}{0.0.0.0:9080}
[2023-12-03T12:11:48,385+03:00] [INFO ] [main] [o.e.jetty.server.AbstractConnector] Stopped admin@16eb01d4{HTTP/1.1, (http/1.1)}{0.0.0.0:9081}
[2023-12-03T12:11:48,385+03:00] [INFO ] [main] [o.e.j.server.handler.ContextHandler] Stopped i.d.j.MutableServletContextHandler@40bac624{/,null,STOPPED}
[2023-12-03T12:11:48,385+03:00] [INFO ] [main] [o.e.j.server.handler.ContextHandler] Stopped i.d.j.MutableServletContextHandler@19740583{/,null,STOPPED}
[2023-12-03T12:11:48,398+03:00] [INFO ] [main] [c.a.a.p.c.m.s.DefaultMetricCorrelationService] Stopped metric correlation service.
[2023-12-03T12:11:48,398+03:00] [INFO ] [main] [c.a.a.p.c.m.s.FileBasedCorrelationPersistor] Stopped metric correlation file cleanup thread.
[2023-12-03T12:11:48,399+03:00] [INFO ] [main] [c.a.a.pipeline.framework.Pipelines] Pipelines have stopped
[2023-12-03T12:11:48,399+03:00] [INFO ] [main] [c.a.a.p.j.JobFrameworkModule] Stopping job framework scheduler
[2023-12-03T12:11:48,399+03:00] [INFO ] [main] [org.quartz.core.QuartzScheduler] Scheduler QuartzScheduler_$_SYS2.ZOOM.COM shutting down.
[2023-12-03T12:11:48,399+03:00] [INFO ] [main] [org.quartz.core.QuartzScheduler] Scheduler QuartzScheduler_$_SYS2.ZOOM.COM paused.
[2023-12-03T12:11:48,400+03:00] [INFO ] [main] [org.quartz.core.QuartzScheduler] Scheduler QuartzScheduler_$_SYS2.ZOOM.COM shutdown complete.
[2023-12-03T12:11:48,400+03:00] [INFO ] [main] [c.a.a.p.j.JobFrameworkModule] Stopped job framework scheduler
[2023-12-03T12:11:48,400+03:00] [INFO ] [main] [c.a.common.util.event.EventBuses] Stopped
[2023-12-03T12:11:49,415+03:00] [WARN ] [main] [c.a.c.u.health.HealthReporterModule] Task will be forcibly stopped now if it has not already stopped
[2023-12-03T12:11:49,415+03:00] [INFO ] [main] [c.a.c.u.health.HealthReporterModule] Stopped
[2023-12-03T12:11:49,423+03:00] [INFO ] [main] [c.a.common.framework.util.SimpleApp] Stopped [events-service-api-store]
[2023-12-03T12:11:49,423+03:00] [INFO ] [main] [c.a.a.p.e.n.s.ElasticsearchDependencyModule] Successfully stopped all server components
Solved! Go to Solution.
4 REPLIES 4
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-19-2023 09:13 AM
@Anees Ur.Rahman confirmed with me they used this existing post to find a solution.
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2024 06:31 AM
Good morning,
We had the same error and attempted to resolve this through the same method above, but the difference we have is the fact that we have a cluster of events services. In other words we have 3 master nodes. One node will work fine, but the other two will start and then shut down stating the elasticsearch never started. Is there a similar solution for that?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-08-2024 09:26 AM
I would suggest searching the community for existing content. If you don't find anything, you can always reach out to AppD Support.
Thanks,
Ryan, Cisco AppDynamics Community Manager
Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
Check out Observabiity in Action
new deep dive videos weekly in the Knowledge Base.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2025 08:50 AM
@Joseph.McNellage were you ever able to resolve this issue? We are encountering the exact same thing and it appears to be a problem with ElasticSearch. I'm wondering if upgrading to the newest version might resolve the issue.
Join us on Feb 26 to explore Splunk AppDynamics deployment strategies, SaaS models, agent rollout plans, and expert best practices.
Register Now
Dive into our Community Blog for the Latest Insights and Updates!
Read the blog here
Related Content
- Machine agent - RegistrationTask - Encountered error during registration. in Infrastructure (Server, Network, Database)
- Events Services can not start up -- immediately after creating a controller with Enterprise Console in Controller (SaaS, On Premises)
- Event Service with elasticsearch 8 Intallation in End User Monitoring (EUM)
- How to start the Elastic Search process? in Business iQ (Analytics)
Top Kudoed Authors
| User | Count |
|---|---|
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form
By clicking subscribe, I have read and understood the Privacy Policy and Terms of Use.