Database Monitoring : An error occurred while getting wait state information. How do I troubleshoot?

[Gaurav.Soni]

Symptoms

While accessing (Database monitoring) collector data (or other tabs, such as Queries or Sessions) you may encounter one of the following errors: 

Case 1

Error
We aren't able to load data from the event service. If this is a SaaS controller, please open a support ticket. If this is on-premise installation, please make sure that the event service is running. See the documentation for instructions on starting the event service. For more details on this error, please see the controller server log.

Error
Error occurred while getting wait state information

Case 2

Error
The server encountered an internal error () that prevented it from fulfilling this request

Error
Error occurred while getting wait state information

Diagnosis

1. Try to reproduce the issue
2. Look in controller/logs/server.log to see if the error thrown in the server.log is something similar to the following:
   
   

   Case 1

   [#|2016-11-09T13:03:57.996-0800|SEVERE|glassfish3.1.2|com.sun.jersey.spi.container.ContainerResponse|_ThreadID=185;_ThreadName=Thread-5;|The RuntimeException could not be mapped to a response, re-throwing to the HTTP container com.appdynamics.analytics.shared.rest.exceptions.ClientException: Could not execute request to http://<events-service-host>:9080/v2/events/dbmon-wait-time/search  Root cause: "request timeout"

   Case 2

   [#|2016-10-25T06:06:40.023-0500|SEVERE|glassfish3.1.2|com.singularity.ee.controller.beans.analytics.client.AccountCreatingAnalyticsClient|_ThreadID=221;_ThreadName=Thread-5;|Could not find account [customer1_6819814d-14f6-46e9-8b2c-53d0770655f9] via lookup. java.lang.reflect.InvocationTargetException Root cause:  RestException(statusCode=401, code=Auth.Unauthorized, message=The supplied auth information is incorrect or not authorized., developerMessage=)

Solutions

When diagnosis indicates Case 1

1. Immediately confirm whether the local Events Service locally on the Controller host is running or stopped.

NOTE | It may also be that the Events Service in use is configured on another dedicated host machine. Just make sure the Events Service is running and that it can be connected from the Controller. The easy test would be $> curl http://<events-service-host>:port/healthcheck?pretty=true

2. If it is stopped, start it as follows: 
   

   $> cd Controller/bin
   $> ./controller.sh start-events-service​

When diagnosis indicates Case 2

This issue usually arises because either of these Controller properties (through which database monitoring is configured) have the following issues:

• appdynamics.analytics.local.store.controller.key 
  (the Controller key mapped to the Events Service in use)
  The Controller settings for the property has a bad key
  
  
• appdynamics.analytics.local.store.url
  (the Events Service URL to which the Controller will connect to fetch DB monitoring events data)
  The Controller setting points the property to an incorrect Events Service
  

What do I do if I am using the Events Service that accompanies the Controller package?

1. appdynamics.analytics.local.store.url: set to http://localhost:9080 
2. appdynamics.analytics.local.store.controller.key: leave the property as the default key (pre-populated from the time Controller installed or upgraded)

What do I do if I am using the external Events Service?

1. appdynamics.analytics.local.store.url: set the same as appdynamics.analytics.server.store.url
2. appdynamics.analytics.local.store.controller.key: set the same as appdynamics.analytics.server.store.controller.key
3. ad.accountmanager.key.controller in events-servcie/conf/events-service-api-store.properties: set the same as appdynamics.analytics.server.store.controller.key