No query data in Databasa statistics

[number33]

Hello!

We are making POC of AppDynamics with our Customer and we've met the following issue. The dbagent shows Wait Time graph in AppDynamics GUI but shows no SQL queries and "Internal Error 500" when we try to open Queries tab. In agent.log we see the following error:

25 Apr 2017 14:40:05,756  WARN [Event-Uploader-Scheduler-2] DBMeasurementUploader:81 - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement  - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)

We installed Platform Admin and Event Service before the DB agent. Can anyone help us to relsolve this issue?

[number33]

Mohammed,

The result of the command:

[dynamics@app-dyn ~]$ curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
async_executor_status                                                                               0 p UNASSIGNED NODE_LEFT
appdynamics_version                                                                                 0 p UNASSIGNED NODE_LEFT
biz_outcome_definitions                                                                             0 p UNASSIGNED NODE_LEFT
event_type_extracted_fields                                                                         0 p UNASSIGNED NODE_LEFT
appdynamics_meters_v2                                                                               0 p UNASSIGNED NODE_LEFT
event_type_metadata_hidden_fields_v1                                                                0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0 p UNASSIGNED NODE_LEFT
slm_performance_configs_v1                                                                          0 p UNASSIGNED NODE_LEFT
event_type_metadata                                                                                 0 p UNASSIGNED NODE_LEFT
appdynamics_api_keys_v1                                                                             0 p UNASSIGNED NODE_LEFT
[dynamics@app-dyn ~]$

[Mohammed.Rayan]

Thanks for the update and the reason seems to be node left and this again can be due to many reasons and so can you try to run the below command which will manually allocate the unassigned shards

Run the below command on your events-service host

curl -XPUT 'localhost:9200/_cluster/settings' -d
'{ "transient":
  { "cluster.routing.allocation.enable" : "all" 
  }
}'

Then verify the status using the following commands again and share the output.

curl http://<events_service_machine>:9081/healthcheck?pretty=true

curl http://<events_service_machine>:9200/_cat/shards?v

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details | grep UNASSIGNED

Regards,

Mohammed Rayan

[number33]

Mohammed,

The results are attached

[Mohammed.Rayan]

Hi there,

I still see the state of your cluster in RED and shards are still Unassigned.

Have you done a restart after the allocation command of your events-service and if not, can you please re-run the allocation command and restart your events-service and then check the health status using the earlier provided commands

Regards,

Mohammed Rayan

[number33]

Hi Mohammed,

I cannot make the steps because our trial license is expired...