No query data in Databasa statistics

[number33]

Hello!

We are making POC of AppDynamics with our Customer and we've met the following issue. The dbagent shows Wait Time graph in AppDynamics GUI but shows no SQL queries and "Internal Error 500" when we try to open Queries tab. In agent.log we see the following error:

25 Apr 2017 14:40:05,756  WARN [Event-Uploader-Scheduler-2] DBMeasurementUploader:81 - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement  - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)

We installed Platform Admin and Event Service before the DB agent. Can anyone help us to relsolve this issue?

[Rajesh.Putta]

Hi,

Based on the exception it looks like internal events service which is much needed for showing wait state information on DB might not be running. So

could you please follow below steps and let us know if you see any improvement.

1) Stop the events service by running this command:

On Linux

Navigate to <controller-install-dir>/bin
Stop Events Service using
./controller.sh stop-events-service

wait for 1-2 mins

and then start using

controller.sh start-events-service

On Windows

Navigate to <controller-install-dir>/bin

Stop Events Service using
controller.bat stop-events-service

wait for 1-2 mins

and then start using

controller.bat start-events-service

If events service is installed as windows service, with elevated privileges(Run As Administrator)

From services.msc, stop and start the service corresponding to controller events service

Wait for 5 mins and then see if you still see any such messages.

Please find the docs link for the same
https://docs.appdynamics.com/display/PRO42/Start+or+Stop+the+Controller#StartorStoptheController-Sta...

Post that things should be normal and you should be able to see db agent stats reporting to controllerprovide you have given all permissions to the user whom you configured on DB Collectors.

PS : Any query which is taking less than 1 second will not be shown on Database Query tab.

Thanks

Rajesh

[number33]

Hi Rajesh,

It didn't help. Made steps:

[dynamics@app-dyn ~]$ cd /home/dynamics/appdynamics/controller/bin/
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:27,298+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:27,450+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:33,931+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:34,072+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:08 YEKT 2017
[dynamics@app-dyn bin]$ ./controller.sh start-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
Starting the Events Service... This may take a few minutes. Check the /home/dynamics/appdynamics/controller/logs/events-service-startup.log for details.
[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO  c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:46 YEKT 2017
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:48 YEKT 2017

[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO  c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"

Errors from db-agent log:

[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,687  WARN DBMeasurementUploader - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement  - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,699  WARN DBMeasurementUploader - Fail to upload Query Stats Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Write Query Stats  - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WriteQueryStatsRequest.<init>(WriteQueryStatsRequest.java:30)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadrQueryStats(DBMeasurementUploader.java:119)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:137)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361  WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,424 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,425  WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480  WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544  WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601  WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]

We installed platform admin and events service on this server with separated MySQL db.

[Rajesh.Putta]

Hi,

Thanks for the update. Could you please send us details as listed here

1) Zipped logs folder located under /home/dynamics/appdynamics/controller

2) DB Agent logs folder

Thanks

Rajesh

[number33]

Hi Rajesh,

The logs are attached to the post. Delete some server.log_... files (with repeated messages) as logs file is to big for a post.

[Mohammed.Rayan]

Hi,

Can you share the output of the below URL's. You can run this on the same host where your events-service is up and running

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

 Note: 9081 is the admin port and 9200 is cluster port of the events-service

Regards,

Mohammed Rayan

[number33]

Hi Rajesh,

The results are:

[dynamics@app-dyn ~]$ curl http://localhost:9081/healthcheck?pretty=true
{
  "deadlocks" : {
    "healthy" : true
  },
  "events-service-api-store / BizOutcome [stage.extract]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
  },
  "events-service-api-store / BizOutcome [stage.filter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
  },
  "events-service-api-store / Build information" : {
    "healthy" : true,
    "message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1"
  },
  "events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.FirstChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.LastChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.LastChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },

 

[number33]

  "events-service-api-store / Chunking.LastChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.TotalChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.TotalChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Chunking.TotalChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
  },
  "events-service-api-store / Configuration properties" : {
    "healthy" : true,
    "message" : "Dynamic properties: [query.chunking.strategies.time.range.splits=[86400000, 604800000, 2592000000], schema.validator.max.custom.events.per.account=20, query.chunking.num.worker.threads=[10, 20, 30, 40], query.chunking.enabled=true, query.chunking.strategies.series.limits=[10, 20, 30, 40], query.chunking.query.timeout.ms=600000, query.chunking.strategies.catch.all.limits=[86400000, 604800000, 2592000000], query.chunking.more.logging=true, query.chunking.async.dump.interval.ms=0, query.chunking.min.time.ms=43200000, publish.validator.max.fields.per.event=255, query.validator.max.batch.count=20, query.chunking.strategies=[SeriesChunkingStrategy, TimeRangeChunkingStrategy], query.chunking.strategies.progress.heuristic.ms=604800000]"
  },
  "events-service-api-store / Index compaction management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Index compaction management]: Is running flag is: [false], last execution successful flag is: [true]"
  },
  "events-service-api-store / Job Scan management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Job Scan management]: Is running flag is: [false], last execution successful flag is: [true]"
  },
  "events-service-api-store / Resource [entitysearch/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [entitysearch/sync]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/publish - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/publish]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/query - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/schema - create]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/schema - delete]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/schema - get]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },
  "events-service-api-store / Resource [events/schema - update]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
  },

[number33]

Hi Mohammed,

The result are attached.

[Mohammed.Rayan]

Hi there,

Thanks for the output and it's very clear now about why we are seeing the below messages repeatedly about (all shards failed)  in your server logs.

Log Snippet:

#|2017-05-02T12:18:05.795+0500|SEVERE|glassfish 4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=307;_ThreadName=http-listener-1(5);_TimeMillis=1493709485795;_LevelValue=1000;|Encountered runtime exception
RestException(statusCode=503, code=Service.Unavailable, errorMessage=Service unavailable: [all shards failed], developerMessage=Exception Message Chain : Failed to execute phase [query_fetch], all shards failed
Caused/related: all shards failed

From the healthcheck output,I infer your cluster state is in RED state due to unassigned shards and hence the above error which says "all shards failed" as a result of it.

 "events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [2], relocating shards: [0], initializing shards: [0], unassigned shards: [18], timed out: [false]"
  },
  

Below Shards output shows which indices are in UNASSIGNED state and all the dbmon event type indices are in UNASSIGNED

[dynamics@app-dyn ~]$ curl http://localhost:9200/_cat/shards?v
index                                                                                               shard prirep state      docs  store ip           node       
async_executor_status                                                                               0     p      UNASSIGNED                                     
appdynamics_version                                                                                 0     p      UNASSIGNED                                     
biz_outcome_definitions                                                                             0     p      UNASSIGNED                                     
event_type_extracted_fields                                                                         0     p      UNASSIGNED                                     
appdynamics_meters_v2                                                                               0     p      UNASSIGNED                                     
event_type_metadata_hidden_fields_v1                                                                0     p      UNASSIGNED                                     
appdynamics_entity_metadata                                                                         0     p      STARTED       0   159b  node-2c7781cf-58e5-4784-b11c-5fc6272af932
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0     p      UNASSIGNED                                     
slm_performance_configs_v1                                                                          0     p      UNASSIGNED                                     
appdynamics_accounts                                                                                0     p      STARTED       2 17.6kb  node-2c7781cf-58e5-4784-b11c-5fc6272af932
event_type_metadata                                                                                 0     p      UNASSIGNED                                     
appdynamics_api_keys_v1                                                                             0     p      UNASSIGNED  

This can happen for a number of reasons and hence to understand the reason behind it, Please execute the following command and share its output

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

Regards,

Mohammed Rayan