Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
04-25-2017 02:33 AM
Hello!
We are making POC of AppDynamics with our Customer and we've met the following issue. The dbagent shows Wait Time graph in AppDynamics GUI but shows no SQL queries and "Internal Error 500" when we try to open Queries tab. In agent.log we see the following error:
25 Apr 2017 14:40:05,756 WARN [Event-Uploader-Scheduler-2] DBMeasurementUploader:81 - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
at java.lang.Thread.run(Thread.java:744)
We installed Platform Admin and Event Service before the DB agent. Can anyone help us to relsolve this issue?
04-25-2017 10:03 PM
Hi,
Based on the exception it looks like internal events service which is much needed for showing wait state information on DB might not be running. So
could you please follow below steps and let us know if you see any improvement.
1) Stop the events service by running this command:
Navigate to <controller-install-dir>/bin
Stop Events Service using
./controller.sh stop-events-service
wait for 1-2 mins
and then start using
controller.sh start-events-service
Navigate to <controller-install-dir>/bin
Stop Events Service using
controller.bat stop-events-service
wait for 1-2 mins
and then start using
controller.bat start-events-service
If events service is installed as windows service, with elevated privileges(Run As Administrator)
From services.msc, stop and start the service corresponding to controller events service
Wait for 5 mins and then see if you still see any such messages.
Please find the docs link for the same
https://docs.appdynamics.com/d
Post that things should be normal and you should be able to see db agent stats reporting to controllerprovide you have given all permissions to the user whom you configured on DB Collectors.
PS : Any query which is taking less than 1 second will not be shown on Database Query tab.
Thanks
Rajesh
04-27-2017 01:06 AM
Hi Rajesh,
It didn't help. Made steps:
[dynamics@app-dyn ~]$ cd /home/dynamics/appdynamics/controller/bin/
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin
[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf
[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:27,298+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:27,450+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin
[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf
[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:33,931+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:34,072+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:08 YEKT 2017
[dynamics@app-dyn bin]$ ./controller.sh start-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin
[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf
[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
Starting the Events Service... This may take a few minutes. Check the /home/dynamics/appdynamics/controller/logs/events-service-startup.log for details.
[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log
12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:
|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:46 YEKT 2017
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:48 YEKT 2017
[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log
12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:
|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"
Errors from db-agent log:
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,687 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,699 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Write Query Stats - null
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
at com.singularity.ee.rest.controller.request.dbmon.WriteQueryStatsRequest.<init>(WriteQueryStatsRequest.java:30)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadrQueryStats(DBMeasurementUploader.java:119)
at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:137)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,424 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,425 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
We installed platform admin and events service on this server with separated MySQL db.
05-01-2017 04:53 AM
Hi,
Thanks for the update. Could you please send us details as listed here
1) Zipped logs folder located under /home/dynamics/appdynamics/controller
2) DB Agent logs folder
Thanks
Rajesh
05-02-2017 01:35 AM
Hi Rajesh,
The logs are attached to the post. Delete some server.log_... files (with repeated messages) as logs file is to big for a post.
05-02-2017 10:32 PM
Hi,
Can you share the output of the below URL's. You can run this on the same host where your events-service is up and running
curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v
Note: 9081 is the admin port and 9200 is cluster port of the events-service
Regards,
Mohammed Rayan
05-03-2017 03:05 AM
Hi Rajesh,
The results are:
[dynamics@app-dyn ~]$ curl http://localhost:9081/healthcheck?pretty=true
{
"deadlocks" : {
"healthy" : true
},
"events-service-api-store / BizOutcome [stage.extract]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / BizOutcome [stage.filter]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / Build information" : {
"healthy" : true,
"message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1"
},
"events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [TimeRangeChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [CatchAllChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [SeriesChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
05-03-2017 03:06 AM
"events-service-api-store / Chunking.LastChunkTimer [TimeRangeChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [CatchAllChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [SeriesChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [TimeRangeChunkingStrategy]" : {
"healthy" : true,
"message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Configuration properties" : {
"healthy" : true,
"message" : "Dynamic properties: [query.chunking.strategies.time.range.splits=[86400000, 604800000, 2592000000], schema.validator.max.custom.events.per.account=20, query.chunking.num.worker.threads=[10, 20, 30, 40], query.chunking.enabled=true, query.chunking.strategies.series.limits=[10, 20, 30, 40], query.chunking.query.timeout.ms=600000, query.chunking.strategies.catch.all.limits=[86400000, 604800000, 2592000000], query.chunking.more.logging=true, query.chunking.async.dump.interval.ms=0, query.chunking.min.time.ms=43200000, publish.validator.max.fields.per.event=255, query.validator.max.batch.count=20, query.chunking.strategies=[SeriesChunkingStrategy, TimeRangeChunkingStrategy], query.chunking.strategies.progress.heuristic.ms=604800000]"
},
"events-service-api-store / Index compaction management" : {
"healthy" : true,
"message" : "Healthcheck for [Index compaction management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Job Scan management" : {
"healthy" : true,
"message" : "Healthcheck for [Job Scan management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Resource [entitysearch/query]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [entitysearch/sync]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish - meter]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query - meter]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - create]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - delete]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - get]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - update]" : {
"healthy" : true,
"message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
05-03-2017 03:08 AM - edited 05-03-2017 03:21 AM
Hi Mohammed,
The result are attached.
05-03-2017 03:27 AM
Hi there,
Thanks for the output and it's very clear now about why we are seeing the below messages repeatedly about (all shards failed) in your server logs.
Log Snippet:
#|2017-05-02T12:18:05.795+0500|SEVERE|glassfish 4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=307;_ThreadName=http-listener-1(5);_TimeMillis=1493709485795;_LevelValue=1000;|Encountered runtime exception RestException(statusCode=503, code=Service.Unavailable, errorMessage=Service unavailable: [all shards failed], developerMessage=Exception Message Chain : Failed to execute phase [query_fetch], all shards failed Caused/related: all shards failed
From the healthcheck output,I infer your cluster state is in RED state due to unassigned shards and hence the above error which says "all shards failed" as a result of it.
"events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : false, "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [2], relocating shards: [0], initializing shards: [0], unassigned shards: [18], timed out: [false]" },
Below Shards output shows which indices are in UNASSIGNED state and all the dbmon event type indices are in UNASSIGNED
[dynamics@app-dyn ~]$ curl http://localhost:9200/_cat/shards?v index shard prirep state docs store ip node async_executor_status 0 p UNASSIGNED appdynamics_version 0 p UNASSIGNED biz_outcome_definitions 0 p UNASSIGNED event_type_extracted_fields 0 p UNASSIGNED appdynamics_meters_v2 0 p UNASSIGNED event_type_metadata_hidden_fields_v1 0 p UNASSIGNED appdynamics_entity_metadata 0 p STARTED 0 159b node-2c7781cf-58e5-4784-b11c-5fc6272af932 customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 1 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 2 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 0 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 1 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 2 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 0 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2 p UNASSIGNED customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0 p UNASSIGNED slm_performance_configs_v1 0 p UNASSIGNED appdynamics_accounts 0 p STARTED 2 17.6kb node-2c7781cf-58e5-4784-b11c-5fc6272af932 event_type_metadata 0 p UNASSIGNED appdynamics_api_keys_v1 0 p UNASSIGNED
This can happen for a number of reasons and hence to understand the reason behind it, Please execute the following command and share its output
curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
Regards,
Mohammed Rayan
User | Count |
---|---|
4 | |
3 | |
3 | |
1 | |
1 | |
1 |
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form