Discussion Feed

To make sure you have all of the information you need when troubleshooting agent issues, we've indexed several articles below that will guide you through how to resolve common issues and answer frequently asked questions.   Contents Agents not reporting to the Controller Difficulty installing, starting, or upgrading an agent Agent crashes and overhead, timeout issues, and errors General functionality   Agents not reporting to the Controller If your agents aren’t reporting data to the Controller, we have multiple articles to help you resolve this issue for different types of agents: Why aren't Application Agents reporting to the Controller? Why isn’t the Machine Agent reporting? How do I resolve an agent connectivity issue related to the "SSLEnabled" property? Why is the Java Agent not reporting to the Controller? All Java Agents have been registered but are reporting status as down. What do I do? How to resolve Analytics Agent failing to connect to the Controller registration endpoint How to troubleshoot missing custom metrics or extensions’ metrics in the Controller Documentation:   Connect the Controller and Agents Documentation:   Dynamic Language Agent Proxy Documentation:   Standalone Machine Agent FAQs and Troubleshooting   Difficulty installing, starting, or upgrading an agent Users who are installing a new agent, upgrading an existing agent or trying to resolve an agent startup failure should follow the steps outlined in these resources: Troubleshooting Agent Issues - Installation and Start-up How to resolve errors while deploying the Java Agent AppDynamics Upgrade Checklist For Any Release   (Note: This is a good resource for users who are upgrading to the latest version of AppDynamics.) Documentation:   Troubleshooting Java Agent Issues Documentation:   Resolve .NET Agent Installation and Configuration Issues Documentation:   Troubleshooting Analytics Agents Issues   Agent crashes and overhead, timeout issues, and errors The resources below can help you troubleshoot issues pertaining to crashes and increased overhead, timeouts, and errors. Agent overhead or crash Database Agent Resource Overhead Timeout issues Errors pertaining to account keys   General functionality If you have questions related to agent functionality that don’t fall into the categories above, we may have addressed them in the resources below.   Troubleshooting Agent Issues - FAQs   (Note: This article addresses several functionality-related questions, including export capabilities and assigning agents to multiple applications) How do I delete an abandoned Machine Agent? What is the best way to set up alerts for App agent and Machine agent status? Documentation:   Troubleshooting Analytics Agents Issues   (Note: In addition to touching upon installation issues, this document describes specific scenarios you may encounter with workarounds.)   Last Updated: 3/28/19 ... View more

Updated 9/28/18   The following article is intended for users who wish to implement custom correlation between .NET applications/tiers using the AppDynamics' .NET App Agent version 3.8 and higher.   You can also implement custom correlation using the AppDynamics' .NET Core Microservices Agent version 4.5.2 and higher by following slightly different steps.   For information regarding the Java Agent, see here: Custom Correlation for Java Applications   Contents: What is Custom Correlation? When is Custom Correlation Needed? Use Cases Implementation Steps to Configure Custom Correlation Identify the Producer and Consumer Methods to Instrument Configure the Producer Configure the Consumer Configure Custom Correlation for the .NET Microservices Agent Restart the coordinator and the instrumented applications   What is Custom Correlation?    Custom correlation enables the user to configure AppDynamics to correlate transactions across tiers, or across parent-child threads in complex multithreaded applications when the default detection mechanisms are not capable of auto-correlating.    Correlation establishes the relationship between exit points on upstream tiers to entry points on downstream tiers. For transaction correlation to work, you must install the .NET Agent on both tiers.   To correlate a distributed invocation, the AppDynamics agent must propagate a unique correlation key by using the extension points of the distributed protocol or by decorating the payload.   The AppDynamics agent needs to know what methods to measure. When defining a custom correlation consider: How to propagate the correlation key. What exactly you want to measure.   When is Custom Correlation Needed?    Use custom correlation when monitoring applications that use unsupported frameworks and protocols.   Use custom correlation XML configuration for business transaction correlation across tiers when the following conditions exist:   Producer There is an easily-defined method call to configure as an exit point on the producer. There is a method call where one of the parameters or a return value exposes the payload object, and that method call occurs during execution of the exit point on the producer. Consumer There is an easily-defined method call to configure as a POCO entry point on the consumer. There is a method call where one of the parameters or a return value exposes the payload object.   In custom correlation, the  producer   refers to the originating tier that makes the exit call. The  consumer   refers to the downstream tier that is the destination for the producer's exit call. If the .NET Agent automatically discovers the exit point for the transaction on the producer, you don't have to manually configure a producer. If it discovers the entry point on the consumer, you don't have to manually configure a consumer.     Use Cases    For custom correlation on communication protocols between nodes such as HTTP and JMS: Add transaction contextual information for remote calls. For example, AppDynamics can add transaction context onto HTTP headers and JMS message properties. For custom correlation within a node: For ESB job objects and ExecutorService thread workers, save the transaction context against thread hand-offs so they can be picked up when async segments start.   Implementation   To implement custom correlation, use XML configuration to specify how to instrument the application. The primary factors governing the exact configuration are the data structures and the sequence of code execution. Use the attached  sample XML configuration  as a starting point.     NOTE:  For AppDynamics version 3.8, the correlation element does not inherit instrumentation element configuration from parent producer or consumer elements. Configure a different method in the correlation Instrumentation element for correlation to work.  In most cases, the producer and consumer reside on separate machines, so they would have separate configuration files. The attached sample file demonstrates two standalone applications on the same machine: Producer.exe and Consumer.exe.   Steps to Configure Custom Correlation:   Before you begin:  Install and configure the .NET Agent on both the producer and the consumer. If the producer is a Windows service or standalone application that  does not implement any of the frameworks we instrument by default, configure a  POCO entry point  for a class/method in your service for the agent to begin instrumentation.   Identify producer and/or consumer methods to instrument. Configure a custom exit point on the producer and/or POCO entry point on the consumer. Configure the producer to add the correlation metadata to the exit point. (Optional) Configure and identify validation methods on the producer. Configure the consumer to read the entry point. (Optional) Configure and identify validation methods on the consumer. Restart the coordinator and instrumented applications.   To identify producer and consumer methods to instrument:   Before you configure custom correlation, you must identify the methods to instrument on both the producer and the consumer.   On the producer identify the following: An exit point method that calls to the consumer. A payload method that executes within the exit call method execution and whose parameters or return value expose the payload object.   On the consumer: A POCO entry point method that handles the producer's exit call. A payload method whose parameters or return value expose the payload object.   Currently, for the .NET Agent, the instrumentation method (exit or entry point) and the payload methods must be different methods.   To configure the producer:   Configure the agent properties in the config.xml file in the agent c onfig  directory. See .NET Agent Directory Structure. All custom correlation configurations go inside the activities element, a child of the App Agents element. See .NET Agent Configuration Properties.   Configure the exit point for the producer in the producer Instrumentation   element. Configure the payload method in the correlation Instrumentation   element.   The following is a sample  a ctivities   element with a  p roducer  element:   <activities>      <producer>          <!-- Configure the exit point. -->          <instrumentation>              < class -name>Producer.IdentifyTransaction</ class -name>              <method-name>Test</method-name>              <match-type>MATCHES_CLASS</match-type>              <param-types>Producer.IdentifyTransaction</param-types>          </instrumentation>          <identifiers>              <identifier name= "IdentifyTransactionTest" >                                <data-gatherer-type>INVOKED</data-gatherer-type>                  <getter-chain> this </getter-chain>                  <user-defined-name>IdentifyTransactionTest</user-defined-name>                  <transformer-type>USER_DEFINED_NAME</transformer-type>              </identifier>          </identifiers>          <correlation>              <!-- Configure the payload method. -->              <instrumentation>                  < class -name>Producer.IdentifyTransaction</ class -name>                  <method-name>TestCorrelation</method-name>                  <match-type>MATCHES_CLASS</match-type>                  <param-types>Producer.IdentifyTransaction</param-types>              </instrumentation>               <payload-pointer>                  <data-gatherer-type>POSITION</data-gatherer-type>                  <position> 0 </position>                  <getter-chain> this </getter-chain>                  <transformer-type>GETTER_METHODS</transformer-type>              </payload-pointer>              <payload-operation>                  <access-type>method</access-type>                  <access-method>setCorrelationKey</access-method>                  <param-types>String</param-types>              </payload-operation>          </correlation>      </producer>        </activities>   To configure the consumer:   Configure the agent properties in the config.xml file in the agent c onfig  directory. See .NET Agent Directory Structure. All custom correlation configurations go inside the activites element, a child of the App Agents element. See .NET Agent Configuration Properties. Configure the entry point for the producer in the consumer Instrumentation   element. Configure the payload method in the correlation Instrumentation   element. The following is a sample activities element with a consumer element:   <activities>          <consumer identify-transaction= "true" >              <!-- Configure the entry point. -->              <instrumentation>                  < class -name>Consumer.IdentifyTransaction</ class -name>                  <method-name>Test</method-name>                  <match-type>MATCHES_CLASS</match-type>              </instrumentation>              <correlation>                  <!-- Configure the payload method. -->                  <instrumentation>                      < class -name>Consumer.IdentifyTransaction</ class -name>                      <method-name>TestCorrelation</method-name>                      <match-type>MATCHES_CLASS</match-type>                  </instrumentation>                  <payload-pointer>                      <data-gatherer-type>POSITION</data-gatherer-type>                      <position> 0 </position>                      <getter-chain> this </getter-chain>                      <transformer-type>GETTER_METHODS</transformer-type>                  </payload-pointer>                  <payload-operation>                      <access-type>method</access-type>                      <access-method>getCorrelationKey</access-method>                  </payload-operation>              </correlation>          </consumer>      </activities>   Configure Custom Correlation for the .NET Microservices Agent:   You can implement custom correlation configuration for the .NET Core Microservices Agent in much of the same way as instructed above. The difference is that the location of the XML configuration goes into the AppDynamicsConfig.json file and not the  config.xml file.   Configure the file that contains the custom correlation XML in the AppDynamicsConfig.json file. NOTE: The supplied file name can contain an absolute or relative path. If the path is relative (or only a file name is supplied), the location is resolved relative to the AppDynamicsConfig.json file. {     "controller" : {         ...     },     "application" : {         ...     },    "instrumentors" : {        "customCorrelationConfig" : "custom-correlation.xml" , "enable" : [ ... ], "disable" : [ ... ] },    .... } See AppDynamicsConfig.json File for a sample .json file. Populate the custom correlation configuration file:   <?xml version= "1.0" encoding= "UTF-8" standalone= "no" ?> <activities>     <consumer identify-transaction= " true " >         <instrumentation>             < class- name>Application.Program</ class- name>             <method-name>Method</method-name>             <match-type>MATCHES_CLASS</match-type>         </instrumentation>         <correlation>             ....         </correlation>     </consumer> </activities> The XML inside the <activities> element has the exact same schema as the existing custom correlation XML found under the <activities> element in config.xml (MSI agent). Restart the coordinator and the instrumented applications:   After you complete the configuration on the producer and consumer, you must restart the AppDynamics.Agent.Coordinator service and instrumented applications.         Published on 6/5/2015. Updated for 4.5.2 on 9/28/2018. ... View more

Table of Contents Use Cases Problem Solution Example API script   Many of our customers use the  Controller Metrics and Events API to retrieve metric data information and information on various types of activities in a monitored environment, including Controller events. This article outlines a customer's use case and includes a generalized example script.    Use Case   The customer is using the AppDynamics metrics API every minute to load the metric data from the Controller to an external system. The customer has multiple Controllers in the AppDynamics SaaS environment and hosts hundreds of applications. They load dozens of metrics from each application.   The customer is using the API for the following use cases: Use a third-party tool for troubleshooting in cases where they need 1-minute granularity for longer than 24 hours. For example, the customer often needs to perform root cause analysis but does not get to it until a few days after the incident occurred.  Use third-party tools to calculate percentiles, consolidate metrics across different controllers at the app or tier level as required and create alerts. Use  a third-party tool  to retain data for several years. Since their peak periods occur just once per year, customers might need data to be retained for 3 years to provide meaningful comparisons and plan their future deployments better.   Problem   The customer uses a set of python scripts to retrieve the metrics. They consistently run into timeout errors and worry that the APIs are not scaling to their requirements.     Our team engaged with the customer and offered to review their process and scripts. Upon review, we found that the python script opens 1000+ of concurrent connections to the Controller. There are only 10 threads dedicated to processing the REST requests on the server side.  As a result, the threads wait indefinitely and then timeout.     Solution   Use a thread pool to invoke the REST APIs. This way the customer will have control on the concurrency. This should help to sustain 10 concurrent requests on the server side, getting the best performance.   Decrease the frequency of the job. For example, run the job every 5 minutes rather than every one minute.  We recommend restricting the number of Metric data points returned per REST call to less than 50K.   Metric REST API Example A RESTful API is an application program interface (API) that uses HTTP requests to GET data. The following section gives some examples of wildcards usage on Metric Data REST API. (This section scrolls horizontally.) The same example is included in the attached Metric REST API Example document for easy copy and paste.     URL:  /controller/rest/applications/100/metric-data?metric-path=<metric-path>&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Method: GET URL Parameters:  metric-path: Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A time-range-type: BEFORE_NOW duration-in-mins: 1 rollup=true Success Response: Response headers: HTTP/1.1 200 OK Server: AppDynamics Content-Type: application/xml Error Response:  REST endpoints can fail is many ways. From unauthorized access to wrongful parameters etc. Listed few error types: HTTP/1.1 403 Forbidden Server: AppDynamics Content-Language: Content-Type: text/html Incorrect Parameters: Data type ("text"|"bin"|""): text Response code: Non HTTP response code: java.net.URISyntaxException Response message: Non HTTP response message: Malformed esAPPDe Sample Request Call: GET   https://hostname/controller/rest/applications/100/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Cookie Data: JSESSIONID=c45eeb9747d8aa24338bc2b973cf; X-CSRF-TOKEN=9a3dddf1c54ebca291c46ab10522faa7bae7c477 Request  Headers: Connection:keep-alive Accept-Language:en-US,en;q=0.5 Accept-Encoding:gzip, deflate User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0 Accept: application/json, text/plain, */* Authorization: Basic Q2FwaXRhbE9uZS1Qcm9kJTQwQ2FwaXRhbE9uZS1Qcm9kOnRlc3QxMjM= X-CSRF-TOKEN:9a3dddf1c54ebca291c46ab10522faa7bae7c477 Sample Response Data:  <metric-datas><metric-data>  <metricId>1578650</metricId>  <metricPath>Business Transaction  Performance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|Individual  Nodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices|Average Block  Time(ms)</metricPath>  <metricName>BTM|BTs|BT:23889|Component:617|Average  Block Time (ms)</metricName>  <frequency>ONE_MIN</frequency>  <metricValues>           <metric-value>           <startTimeInMillis><timeInMilliseconds></startTimeInMillis>           <value>0</value>           <min>0</min>           <max>0</max>           <current>0</current>           <sum>0</sum>           <count>0</count>          <standardDeviation>0.0</standardDeviation>           <occurrences>0</occurrences>           <useRange>true</useRange>           </metric-value>  </metricValues> </metric-data> </metric-datas> Metric Calls example of wildcards at various level:  1.Six wild card (|*|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|BusinessTransactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices| METRIC-NAME</metricPath> 2.Five wild card (|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes | METRIC-NAME</metricPath> 3.Four wild card (|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices |BVLInterfaceWS2.Process|METRIC-NAME</metricPath> 4.Three wild card (|*|*|*)  <metricPath>Business TransactionPerformance|Business Transactions| APPDInterface_PROD/APPDWebServices|METRIC-NAME</metricPat> 5.Two wild card (|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions |METRIC-NAME</metricPath> 6.One wild card (|*)  <metricPath>BusinessTransactionPerformance|METRIC-NAME</metricPath> Request  to keep metrics under 50K:  https://<HostName>/controller/rest/applications/<ApplicationID>/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C TOMCAT-APP %7C Transactions.SA %7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true If REST Call is fetching more than 50K metrics, you can replace wild characters with hard-coded values (e.g. TOMCAT-APP & Tranaction.SA), which will control metric count.       Find a test API script on Github here:  Appdynamics / metric-loader   ... View more

Contents: Installing a new Agent Upgrading an existing Agent Resolve start-up failure     Installation   Q. I get the following error message when installing a Machine Agent. How should I proceed?  Failed to find Premain-Class manifest attribute in /usr/share/appdynamics/machineagent.jar Error occurred during initialization of VM   A. The error message shows that the Machine Agent jar was used as a Java Agent. >java -javaagent:d:\AppDynamics\installs\3.5.6\MachineAgent\machineagent.jar -version Failed to find Premain-Class manifest attribute in d:\AppDynamics\installs\3.5.6\MachineAgent\machineagent.jar Error occurred during initialization of VM.  Agent library failed to init: instrument   Verify the command that is being used to start the Machine Agent. Follow the documentation to Install the Standalone Machine Agent. Open a separate command prompt and start the Machine Agent using  java -Dmetric.http.listener=true -jar machineagent.jar command.     Q.  Is there a way to install the appDynamics database agent without the wizard, such as simply configuring the installation XML files?  I want to monitor a client but I am having trouble when installing the agent. I cannot log in to the DB from outside the network and opening up the listener port on the DB for me right now is an issue. Is there a workaround?   A. You do not necessarily have to install the Database Agent on the database server. The Database Agent can reside on any server as long as that server can communicate to the database host. See Database Visibility System Requirements.      Q. I'm having trouble installing a PHP Agent. What steps should I take?   A. After PHP Agent installation there are a few cases when the agent does not report to the Controller. Use the following article for basic troubleshooting steps to verify the installation and place the agent configuration files in proper place. PHP Agent installation troubleshooting guide.     Q. Is it possible to deploy an agent using Puppet?   A. Every deployment is different,  so this task doesn't lend itself to one master Puppet script maintained by AppDynamics. However, here is a sample that can help you get started on your own Puppet module. Agent deployment using Puppet.     Upgrade   Q. What are the steps to upgrade a mobile agent?   A. For iOS and Android, follow these steps:    Upgrade iOS SDK and Upgrade the Android Mobile Agent     Q. Where can I find information on how to upgrade an agent?    A. The following documentation links will walk you through the requirements and steps for upgrading an agent. Upgrade the .NET Agent Upgrade the Node.js Agent Upgrade the Java Agent Upgrade the PHP Agent Upgrade the Python Agent Upgrade Analytics Agent Upgrade the Database Agent Upgrade the Standalone Machine Agent Upgrade the Apache Agent     Start-Up   Q. My Analytics Agent fails to start and throws the following error message. What steps should I take? java.lang.NoSuchMethodError: io.dropwizard.logging.LoggingFactory.bootstrap   A. This error message is normally caused by the presence of a duplicate jar file from a different version of the Analytic Agent.    It can occur when the user unzips a new version of their machine-agent.zip file into the same directory where the older Machine Agent resides, creating a duplicate presence of the same jar files.   The solution is to first take a backup of the old agent, then delete or rename the Machine Agent, and then unzip the new machine-agent.zip  .       Q. My Analytic Agent fails to start and shows the following error message in the logs. What do I do? Analytics agent failed to connect to the controller registration endpoint.   A. This issue occurs when using a self-signed certificate for the Controller. The Analytics Agent uses a truststore for the JRE to validate the certificate of the connecting server. The truststore does not have self-signed certificates added to the store by default.   Import the client certificate to the JRE truststore, which is located in the Java directory used by the Analytics Agent by default. Example:   JAVA_HOME/lib/security/cacerts      More Troubleshooting Guides: FAQs Why aren’t Application Agents reporting to the Controller?   Published 2/26/2018 ... View more

Contents: Delete an abandoned Machine Agent Set up alerts for Agent status Issue with account keys Agent overhead or crash Timeout issue Multiple apps on the same server List of Agents   Q. Is there a way to  delete an abandoned Machine Agent when it is not attached to any APM agents?    A. Yes, t here are two ways to delete the abandoned machine agent, either from controller UI by creating a dummy application or through a manual step to delete the agent from the database. The steps for both are located here:  Delete abandoned machine agent.     Q.  What is the best way to set up alerts for App Agent Status and Machine Agent Status?    A. There are two solutions for this, found here: Agent Status on .NET      Q. What does the error message "account key doesn't match with the current account for machine id" mean?   A. The message indicates that account key generated by AppDynamics controller during the registration may be changed at this point in time. This could happen if you migrate the Controller without shutting down the agents. Restart the agent to resolve this error.     Q. How to resolve Agent causing overhead or crash.   A. Follow these steps to troubleshoot: Check the agent logs for any communication errors between Agent and Controller. Make sure ALD has not been turned on for a long time. The Agent requires 30 MB of permgen space. Set this parameter ( -XX:MaxPermSize=<value> ) to a higher value (256m at least) and try again. Get the crash report and heap dump for analysis. Check the agent log level is set to INFO (default log level value if not changed to debug/trace) at agent_dir/conf/logging/log4j.xml . Make sure periodic sampling and occurrence request sampling is not set to a low value (default 100 requests and 10 min, update/disable as per requirement). Check the agent logs if any repetitive exceptions are logged. Disable the agent from the node dashboard and monitor the heap usage whether the overhead/crash is still happening to find whether agent/webapp instrumentation is raising the issue.     Q. When requesting agent logs in <Application> -> Servers -> App Servers -> <tier> -> <node> -> Agents, the log status times out. One possible cause is someone first requested the log and reset the agent before the log sends to the Controller.   A. Reset agent before requesting the agent log.       Q. Is it possible to assign a Machine Agent to multiple applications on the same server?   A. We need to run   one Machine Agent process for each application   on a machine. Add the VM parameter ( -Dappdynamics.agent.uniqueHostId=<<some unique value per application>> ) to both Machine Agent and the App Agent. Provide the same value to the machine - app agent pair. For example, if you have 2 nodes (n1 and n2 ) reporting to applications (a1 and a2). Let's say m1 and m2 are your machine agents for the corresponding nodes. You will need to pass   -Dappdynamics.agent.uniqueHostId=a1   for n1 and m1. and   -Dappdynamics.agent.uniqueHostId=a2   for n2 and m2. The uniqueHostId value can be any unique value but should be different for each application. After restarting the JVMs with the above options you should see machine agents associating with corresponding applications.    Q.  I want to get the list of AppD agents. I mean, we can see those list of agents in settings, but I want to get all of those agents to an Excel sheet. Is there any way that we can export the list of agents and details?    A. It's not out of the box,  but there is a workaround. Create small program which can read  /controller/rest/applications - List of all applications and get list of nodes from /controller/rest/applications/<application-name|application-id>/nodes Go to AppDynamics Agents page. Take screenshot. Trim the picture. Copy it to One Note. Right-click and Copy to Text.   More Troubleshooting Guides: Installation and Start-up Why aren’t Application Agents reporting to the Controller?   Published 2/26/2018 ... View more