What is the backend discovery life cycle?

[pclark]

Confirming Backend Discovery and Instrumentation

This article describes the backend discovery life cycle. Understanding these details can help you debug instrumentation issues of backends that are not automatically discovered. The phases referred to in this article refer to the phases in the discovery life cycle. See AppDynamics Auto-Discovery Life Cycle.

Typical examples of backends are databases instance and remote services, such as web services, JMS clients and message queues, HTTP backends and so on.

Backends are born from exit points. An exit point call from an instrumented node to an uninstrumented node or other service results in backend discovery.

The activity of a supported remote service or database is identified by its type and related properties. Each type of backend has a list of properties associated with it. The properties are a list of key-value pairs. For example, for a JDBC database, the following screen capture shows the properties that can be used to uniquely identify the JDBC database.

Phase One: Was Backend Discovery Successful?

Determine if the exit point was successfully instrumented:

1. Was the backend limit reached? If you see backends, but some are missing, check that the backend limit was not reached. There is a limit of 300 per application. If you have hit the backend limit, you may need to revise the backend discovery configuration. See All Other Traffic Backends.

2. If the backend limit was not reached, is your backend one that is discovered by default? To determine this, check the documentation for your app agent type. Use the docs version that corresponds to your agent version: 

• Latest: Supported Environments and Versions
• 3.9 Supported Environments and Versions 

3. If your backend is not on the supported list, you can configure a custom exit point. See Configure Custom Exit Points.

4.  If you are using a supported backend and you are not seeing what you expect, find the BCT log (Byte Code Transformer log) and review the entries for exit point interceptors starting with the text exit.<your_framework>. For example, exit.jdbc for a JDBC database or exit.jms for a JMS service. See Agent Log Files for details on requesting the agent logs.

      a. If you find evidence in the BCT log that the exit point was instrumented, check for exit point exceptions in the agent logs. If you find exceptions for your exit point, contact support for additional assistance. If you do not find any exceptions, move on to phase 2, backend registration.

      b. If your exit point was not instrumented,  and you know the exit point that you want to instrument, configure a custom exit point. If you do not know the exit point to instrument, look at the Call Graph to find the method and configure the custom exit point.

Phase Two: Backend Registration

When a backend is successfully discovered, it is assigned a name based on the rules for the exit point type and any custom backend detection configuration (if applicable). Every backend is stored in the backend table in the controller database. The agent detects when the backend is hit and sends a registration request to the controller.

Registration is the process where each object is assigned an ID by the controller. After the ID is assigned, the communication between the agent and controller uses the ID to identify exactly which objects are being reported and stored in the controller database. You can review the backend registration log entries in the REST Log.

Confirm successful backend registration:

1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files.

2. Look for the backend registration request log entries. In the following request, you can see three backends were discovered and sent to the controller for registration.

 <unregistered-backends>
Size : 3
<UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=JMS, properties=[Name:DESTINATION_TYPE, Value:QUEUE, Name:DESTINATION_NAME, Value:OrderQueue, Name:VENDOR, Value:Active MQ]], metaInfo=[], applicationId=0, createdOn=0, displayName=Active MQ-OrderQueue, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]>
<UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=WEB_SERVICE, properties=[Name:SERVICE_NAME, Value:OrderService]], metaInfo=[], applicationId=0, createdOn=0, displayName=OrderService, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]>
<UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=JDBC, properties=[Name:HOST, Value:LOCALHOST, Name:PORT, Value:3306, Name:SCHEMA, Value:APPDY, Name:MAJOR_VERSION, Value:5.5.16-log, Name:URL, Value:jdbc:mysql://localhost:3306/appdy, Name:VENDOR, Value:MySQL DB]], metaInfo=[], applicationId=0, createdOn=0, displayName=APPDY-MySQL DB, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]>
<unregistered-backends/>

3. Look for the corresponding backend registration responses. 

In the following response you can see that the JMS exit point has an id=12 and is named  OrderQueue. ID 15 was assigned to the MySQL JDBC backend.

 <resolved-backend-calls>
<12>::<NodeResolutionInfo[exitPointType=JMS, properties=[Name:DESTINATION_NAME, Value:OrderQueue, Name:DESTINATION_TYPE, Value:QUEUE, Name:VENDOR, Value:Active MQ]]>
<14>::<NodeResolutionInfo[exitPointType=WEB_SERVICE, properties=[Name:SERVICE_NAME, Value:OrderService]]>
<15>::<NodeResolutionInfo[exitPointType=JDBC, properties=[Name:HOST, Value:LOCALHOST, Name:MAJOR_VERSION, Value:5.5.16-log, Name:PORT, Value:3306, Name:SCHEMA, Value:APPDY, Name:URL, Value:jdbc:mysql://localhost:3306/appdy, Name:VENDOR, Value:MySQL DB]]>
...
<resolved-backend-calls/>

4. If you do not see successful registration request and response, look in the Controller Logs for exceptions related to backend registration.

Note: If you exceed the backend limit, you see log messages similar to the following:

AD Thread Pool-Global465] 23 Dec 2014 10:36:16,587 INFO BTOverflowCounter - AD Thread Pool-Global465] 23 Dec 2014 10:36:16,587 INFO BTOverflowCounter - DroppedBackend{key='https://rxservice.company.com/RxService/Account MgmtService', count=35, identifyingProperties={SERVICE_NAME=AccountMgmtService}, exitType=WEB_SERVICE}

Phase Three: Metrics Registration

After successful backend registration, the agent keeps a backend map in memory with the ID, type, and name for each backend. The next time the backend is hit, the agent is ready to report metrics. A similar process of request and response between the agent and controller occurs and the controller assigns each metric an ID.

Was backend metric registration successful?

There are three types of backend metrics:

• Business transaction (BT) metrics - aggregate metrics for the specific BT for the specified backend.
• Tier metrics - aggregate metrics across the tier for the specified backend.
• Backend metrics - overall aggregate metrics for the backend

Use the following steps to confirm metric registration:

1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files.

2. Look for log entries showing metric registration.

For example to find metrics related to the MySQL database registered in the previous example (ID=15), a        search of the REST log for the string "[UNRESOLVED][15]" results in the following types of log entries:

<metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/>
<metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:81|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/>
<metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:82|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/>
<metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/>
<metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/>
<metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/>
<metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/>
<metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/>
<metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/>

• The metrics with names such as "BTM|BTs|BT:80|...", "BTM|BTs|BT:81|...," and "BTM|BTs|BT:82|..." are BT backend metrics. You may recognize the BT ID numbers, 80,81, and 82.
• The metrics with names such as "BTM|Application Summary..." are tier level metrics for this backend.
• The metrics with names such as "BTM|Backends|... " are aggregate backend metrics.

3.  Look for the corresponding metric registration response entries.
In the following response you can see the IDs assigned to the metrics.

For example, the first line in the example shows ID=1925 is assigned to the "Errors per Minute" metric for BT 80 on this MySQL backend (UNRESOLVED 15).

<metric id="1925" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Errors per Minute"/>
<metric id="1936" name="BTM|BTs|BT:81|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/>
<metric id="1959" name="BTM|BTs|BT:82|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/>
<metric id="1967" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/>
<metric id="1935" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/>
<metric id="2015" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/>
<metric id="1937" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Average Response Time (ms)"/>...
<metric id="1991" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/>
<metric id="2017" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Calls per Minute"/>

4. If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to backend metric registration.

Phase Four: Backend Metric Reporting

After successful metric registration, metrics are reported to the controller every minute. Using the metric id, you can search the REST log for the metric upload.

Was backend metric reporting successful?

Use the following steps to confirm backend metric upload:

1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files.

2. Look for log entries showing metric reporting uploads similar to the following:

<metric id='1936', value[sum=177, count=499, min=0, max=7, current=1]>
<metric id='1959', value[sum=160, count=1, min=160, max=160, current=160]>
<metric id='1967', value[sum=157, count=518, min=0, max=4, current=0]>
<metric id='1935', value[sum=611, count=688, min=0, max=79, current=1]>
<metric id='2015', value[sum=1353, count=1, min=1353, max=1353, current=1353]>
<metric id='1937', value[sum=285, count=1353, min=0, max=2, current=0]>
<metric id='1991', value[sum=136, count=1, min=136, max=136, current=136]>
<metric id='2017', value[sum=1144, count=1, min=1144, max=1144, current=1144]>

3. If you do not see successful backend metric data uploads, look in the Controller Logs for related exceptions.