How to resolve Database Agent intermittently stops sending metrics to the Controller

[Anka.Thanneeru]

 

Problem:

Scenario 1:

The Agent stops collecting or sending metrics to the Controller intermittently, but the Agent is shown as "running." All, or nearly all collectors are temporarily disabled via the UI.

Agent logs include error messages similar to the following:

[<Athene Database - Prod>-Scheduler-1] 14 May 2017 09:08:28,910  INFO ADBCollector - DB Collector Athene Database - Prod is temporarily disabled.



[<35111_CDW_DB_PROD>-Monitor-Scheduler-3] 29 Jun 2017 00:59:27,047 ERROR ManagedMonitorDelegate - Fatal transport error while connecting to URL [/controller/instance/317/events]: java.net.SocketException: Broken pipe



Scenario 2:

The Database Agent has crashed abruptly, or has crashed with Out Of Memory
 errors.
 

Solution:

1. First, determine if the Database Agent is running with heap settings.

• Database Agent logs should contain a parameter similar to -Xmx1536m in the INFO DBAgent - JVM Args: section.
• If this parameter is not listed, the user needs to add the heap settings to the startup script for the Agent.
• Example:

• java -Xmx1536m -Djava.library.path=E:\dbagent_Prod\auth\x64 -Ddbagent.name=FCA -Database -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=taskkill /F /PID %p -jar db-agent.jar



2. Once the heap settings are provided, then check if the settings match the Database Agent systems requirements.  

• Documentation: Database Visibility Requirements and Supported Environments

• Recommendation: 1 GB of heap space and an additional 256 MB of heap space for each collector.
• When monitoring production databases, the heap usage per collector may reach 512 MB.
    


3. Increase the heap value based on the number of collectors.
  

• Example: Monitoring 5 databases instances requires ( 5 x 256 MB ) + 1024 MB = 2,304 MB


4. Database Agent restart is required for the above changes.