Discussion Feed

To make sure you have all of the information you need when troubleshooting agent issues, we've indexed several articles below that will guide you through how to resolve common issues and answer frequently asked questions.   Contents Agents not reporting to the Controller Difficulty installing, starting, or upgrading an agent Agent crashes and overhead, timeout issues, and errors General functionality   Agents not reporting to the Controller If your agents aren’t reporting data to the Controller, we have multiple articles to help you resolve this issue for different types of agents: Troubleshooting Agent Issues - Reporting and Connectivity Why isn’t the Machine Agent reporting? How do I troubleshoot agents not reporting to the Controller? How do I resolve an agent connectivity issue related to the "SSLEnabled" property? Why is the Java agent not reporting to the Controller? All Java agents have been registered but are reporting status as down. What do I do? How to resolve Analytics Agent failing to connect to the Controller registration endpoint How to troubleshoot missing custom metrics or extensions’ metrics in the Controller Documentation:   Connect the Controller and Agents Documentation:   Dynamic Language Agent Proxy Documentation:   Standalone Machine Agent FAQs and Troubleshooting   Difficulty installing, starting, or upgrading an agent Users who are installing a new agent, upgrading an existing agent or trying to resolve an agent startup failure should follow the steps outlined in these resources: Troubleshooting Agent Issues - Installation and Start-up How to resolve errors while deploying the Java Agent AppDynamics Upgrade Checklist For Any Release   (Note: This is a good resource for users who are upgrading to the latest version of AppDynamics.) Documentation:   Troubleshooting Java Agent Issues Documentation:   Resolve .NET Agent Installation and Configuration Issues Documentation:   Troubleshooting Analytics Agents Issues   Agent crashes and overhead, timeout issues, and errors The resources below can help you troubleshoot issues pertaining to crashes and increased overhead, timeouts, and errors. Agent overhead or crash Database Agent Resource Overhead Timeout issues Errors pertaining to account keys   General functionality If you have questions related to agent functionality that don’t fall into the categories above, we may have addressed them in the resources below.   Troubleshooting Agent Issues - FAQs   (Note: This article addresses several functionality-related questions, including export capabilities and assigning agents to multiple applications) How do I delete an abandoned machine agent? What is the best way to set up alerts for App agent and Machine agent status? Documentation:   Troubleshooting Analytics Agents Issues   (Note: In addition to touching upon installation issues, this document describes specific scenarios you may encounter with workarounds.) ... View more

Updated 9/28/18   The following article is intended for users who wish to implement custom correlation between .NET applications/tiers using the AppDynamics' .NET App Agent version 3.8 and higher.   You can also implement custom correlation using the AppDynamics' .NET Core Microservices Agent version 4.5.2 and higher by following slightly different steps.   For information regarding the Java Agent, see here: Custom Correlation for Java Applications   Contents: What is Custom Correlation? When is Custom Correlation Needed? Use Cases Implementation Steps to Configure Custom Correlation Identify the Producer and Consumer Methods to Instrument Configure the Producer Configure the Consumer Configure Custom Correlation for the .NET Microservices Agent Restart the coordinator and the instrumented applications   What is Custom Correlation?    Custom correlation enables the user to configure AppDynamics to correlate transactions across tiers, or across parent-child threads in complex multithreaded applications when the default detection mechanisms are not capable of auto-correlating.    Correlation establishes the relationship between exit points on upstream tiers to entry points on downstream tiers. For transaction correlation to work, you must install the .NET Agent on both tiers.   To correlate a distributed invocation, the AppDynamics agent must propagate a unique correlation key by using the extension points of the distributed protocol or by decorating the payload.   The AppDynamics agent needs to know what methods to measure. When defining a custom correlation consider: How to propagate the correlation key. What exactly you want to measure.   When is Custom Correlation Needed?    Use custom correlation when monitoring applications that use unsupported frameworks and protocols.   Use custom correlation XML configuration for business transaction correlation across tiers when the following conditions exist:   Producer There is an easily-defined method call to configure as an exit point on the producer. There is a method call where one of the parameters or a return value exposes the payload object, and that method call occurs during execution of the exit point on the producer. Consumer There is an easily-defined method call to configure as a POCO entry point on the consumer. There is a method call where one of the parameters or a return value exposes the payload object.   In custom correlation, the  producer   refers to the originating tier that makes the exit call. The  consumer   refers to the downstream tier that is the destination for the producer's exit call. If the .NET Agent automatically discovers the exit point for the transaction on the producer, you don't have to manually configure a producer. If it discovers the entry point on the consumer, you don't have to manually configure a consumer.     Use Cases    For custom correlation on communication protocols between nodes such as HTTP and JMS: Add transaction contextual information for remote calls. For example, AppDynamics can add transaction context onto HTTP headers and JMS message properties. For custom correlation within a node: For ESB job objects and ExecutorService thread workers, save the transaction context against thread hand-offs so they can be picked up when async segments start.   Implementation   To implement custom correlation, use XML configuration to specify how to instrument the application. The primary factors governing the exact configuration are the data structures and the sequence of code execution. Use the attached  sample XML configuration  as a starting point.     NOTE:  For AppDynamics version 3.8, the correlation element does not inherit instrumentation element configuration from parent producer or consumer elements. Configure a different method in the correlation Instrumentation element for correlation to work.  In most cases, the producer and consumer reside on separate machines, so they would have separate configuration files. The attached sample file demonstrates two standalone applications on the same machine: Producer.exe and Consumer.exe.   Steps to Configure Custom Correlation:   Before you begin:  Install and configure the .NET Agent on both the producer and the consumer. If the producer is a Windows service or standalone application that  does not implement any of the frameworks we instrument by default, configure a  POCO entry point  for a class/method in your service for the agent to begin instrumentation.   Identify producer and/or consumer methods to instrument. Configure a custom exit point on the producer and/or POCO entry point on the consumer. Configure the producer to add the correlation metadata to the exit point. (Optional) Configure and identify validation methods on the producer. Configure the consumer to read the entry point. (Optional) Configure and identify validation methods on the consumer. Restart the coordinator and instrumented applications.   To identify producer and consumer methods to instrument:   Before you configure custom correlation, you must identify the methods to instrument on both the producer and the consumer.   On the producer identify the following: An exit point method that calls to the consumer. A payload method that executes within the exit call method execution and whose parameters or return value expose the payload object.   On the consumer: A POCO entry point method that handles the producer's exit call. A payload method whose parameters or return value expose the payload object.   Currently, for the .NET Agent, the instrumentation method (exit or entry point) and the payload methods must be different methods.   To configure the producer:   Configure the agent properties in the config.xml file in the agent c onfig  directory. See .NET Agent Directory Structure. All custom correlation configurations go inside the activities element, a child of the App Agents element. See .NET Agent Configuration Properties.   Configure the exit point for the producer in the producer Instrumentation   element. Configure the payload method in the correlation Instrumentation   element.   The following is a sample  a ctivities   element with a  p roducer  element:   <activities>      <producer>          <!-- Configure the exit point. -->          <instrumentation>              < class -name>Producer.IdentifyTransaction</ class -name>              <method-name>Test</method-name>              <match-type>MATCHES_CLASS</match-type>              <param-types>Producer.IdentifyTransaction</param-types>          </instrumentation>          <identifiers>              <identifier name= "IdentifyTransactionTest" >                                <data-gatherer-type>INVOKED</data-gatherer-type>                  <getter-chain> this </getter-chain>                  <user-defined-name>IdentifyTransactionTest</user-defined-name>                  <transformer-type>USER_DEFINED_NAME</transformer-type>              </identifier>          </identifiers>          <correlation>              <!-- Configure the payload method. -->              <instrumentation>                  < class -name>Producer.IdentifyTransaction</ class -name>                  <method-name>TestCorrelation</method-name>                  <match-type>MATCHES_CLASS</match-type>                  <param-types>Producer.IdentifyTransaction</param-types>              </instrumentation>               <payload-pointer>                  <data-gatherer-type>POSITION</data-gatherer-type>                  <position> 0 </position>                  <getter-chain> this </getter-chain>                  <transformer-type>GETTER_METHODS</transformer-type>              </payload-pointer>              <payload-operation>                  <access-type>method</access-type>                  <access-method>setCorrelationKey</access-method>                  <param-types>String</param-types>              </payload-operation>          </correlation>      </producer>        </activities>   To configure the consumer:   Configure the agent properties in the config.xml file in the agent c onfig  directory. See .NET Agent Directory Structure. All custom correlation configurations go inside the activites element, a child of the App Agents element. See .NET Agent Configuration Properties. Configure the entry point for the producer in the consumer Instrumentation   element. Configure the payload method in the correlation Instrumentation   element. The following is a sample activities element with a consumer element:   <activities>          <consumer identify-transaction= "true" >              <!-- Configure the entry point. -->              <instrumentation>                  < class -name>Consumer.IdentifyTransaction</ class -name>                  <method-name>Test</method-name>                  <match-type>MATCHES_CLASS</match-type>              </instrumentation>              <correlation>                  <!-- Configure the payload method. -->                  <instrumentation>                      < class -name>Consumer.IdentifyTransaction</ class -name>                      <method-name>TestCorrelation</method-name>                      <match-type>MATCHES_CLASS</match-type>                  </instrumentation>                  <payload-pointer>                      <data-gatherer-type>POSITION</data-gatherer-type>                      <position> 0 </position>                      <getter-chain> this </getter-chain>                      <transformer-type>GETTER_METHODS</transformer-type>                  </payload-pointer>                  <payload-operation>                      <access-type>method</access-type>                      <access-method>getCorrelationKey</access-method>                  </payload-operation>              </correlation>          </consumer>      </activities>   Configure Custom Correlation for the .NET Microservices Agent:   You can implement custom correlation configuration for the .NET Core Microservices Agent in much of the same way as instructed above. The difference is that the location of the XML configuration goes into the AppDynamicsConfig.json file and not the  config.xml file.   Configure the file that contains the custom correlation XML in the AppDynamicsConfig.json file. NOTE: The supplied file name can contain an absolute or relative path. If the path is relative (or only a file name is supplied), the location is resolved relative to the AppDynamicsConfig.json file. {     "controller" : {         ...     },     "application" : {         ...     },    "instrumentors" : {        "customCorrelationConfig" : "custom-correlation.xml" , "enable" : [ ... ], "disable" : [ ... ] },    .... } See AppDynamicsConfig.json File for a sample .json file. Populate the custom correlation configuration file:   <?xml version= "1.0" encoding= "UTF-8" standalone= "no" ?> <activities>     <consumer identify-transaction= " true " >         <instrumentation>             < class- name>Application.Program</ class- name>             <method-name>Method</method-name>             <match-type>MATCHES_CLASS</match-type>         </instrumentation>         <correlation>             ....         </correlation>     </consumer> </activities> The XML inside the <activities> element has the exact same schema as the existing custom correlation XML found under the <activities> element in config.xml (MSI agent). Restart the coordinator and the instrumented applications:   After you complete the configuration on the producer and consumer, you must restart the AppDynamics.Agent.Coordinator service and instrumented applications.         Published on 6/5/2015. Updated for 4.5.2 on 9/28/2018. ... View more

Table of Contents Use Cases Problem Solution Example API script   Many of our customers use the  Controller Metrics and Events API to retrieve metric data information and information on various types of activities in a monitored environment, including Controller events. This article outlines a customer's use case and includes a generalized example script.    Use Case   The customer is using the AppDynamics metrics API every minute to load the metric data from the Controller to an external system. The customer has multiple Controllers in the AppDynamics SaaS environment and hosts hundreds of applications. They load dozens of metrics from each application.   The customer is using the API for the following use cases: Use a third-party tool for troubleshooting in cases where they need 1-minute granularity for longer than 24 hours. For example, the customer often needs to perform root cause analysis but does not get to it until a few days after the incident occurred.  Use third-party tools to calculate percentiles, consolidate metrics across different controllers at the app or tier level as required and create alerts. Use  a third-party tool  to retain data for several years. Since their peak periods occur just once per year, customers might need data to be retained for 3 years to provide meaningful comparisons and plan their future deployments better.   Problem   The customer uses a set of python scripts to retrieve the metrics. They consistently run into timeout errors and worry that the APIs are not scaling to their requirements.     Our team engaged with the customer and offered to review their process and scripts. Upon review, we found that the python script opens 1000+ of concurrent connections to the Controller. There are only 10 threads dedicated to processing the REST requests on the server side.  As a result, the threads wait indefinitely and then timeout.     Solution   Use a thread pool to invoke the REST APIs. This way the customer will have control on the concurrency. This should help to sustain 10 concurrent requests on the server side, getting the best performance.   Decrease the frequency of the job. For example, run the job every 5 minutes rather than every one minute.  We recommend restricting the number of Metric data points returned per REST call to less than 50K.   Metric REST API Example A RESTful API is an application program interface (API) that uses HTTP requests to GET data. The following section gives some examples of wildcards usage on Metric Data REST API. (This section scrolls horizontally.) The same example is included in the attached Metric REST API Example document for easy copy and paste.     URL:  /controller/rest/applications/100/metric-data?metric-path=<metric-path>&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Method: GET URL Parameters:  metric-path: Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A time-range-type: BEFORE_NOW duration-in-mins: 1 rollup=true Success Response: Response headers: HTTP/1.1 200 OK Server: AppDynamics Content-Type: application/xml Error Response:  REST endpoints can fail is many ways. From unauthorized access to wrongful parameters etc. Listed few error types: HTTP/1.1 403 Forbidden Server: AppDynamics Content-Language: Content-Type: text/html Incorrect Parameters: Data type ("text"|"bin"|""): text Response code: Non HTTP response code: java.net.URISyntaxException Response message: Non HTTP response message: Malformed esAPPDe Sample Request Call: GET   https://hostname/controller/rest/applications/100/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Cookie Data: JSESSIONID=c45eeb9747d8aa24338bc2b973cf; X-CSRF-TOKEN=9a3dddf1c54ebca291c46ab10522faa7bae7c477 Request  Headers: Connection:keep-alive Accept-Language:en-US,en;q=0.5 Accept-Encoding:gzip, deflate User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0 Accept: application/json, text/plain, */* Authorization: Basic Q2FwaXRhbE9uZS1Qcm9kJTQwQ2FwaXRhbE9uZS1Qcm9kOnRlc3QxMjM= X-CSRF-TOKEN:9a3dddf1c54ebca291c46ab10522faa7bae7c477 Sample Response Data:  <metric-datas><metric-data>  <metricId>1578650</metricId>  <metricPath>Business Transaction  Performance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|Individual  Nodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices|Average Block  Time(ms)</metricPath>  <metricName>BTM|BTs|BT:23889|Component:617|Average  Block Time (ms)</metricName>  <frequency>ONE_MIN</frequency>  <metricValues>           <metric-value>           <startTimeInMillis><timeInMilliseconds></startTimeInMillis>           <value>0</value>           <min>0</min>           <max>0</max>           <current>0</current>           <sum>0</sum>           <count>0</count>          <standardDeviation>0.0</standardDeviation>           <occurrences>0</occurrences>           <useRange>true</useRange>           </metric-value>  </metricValues> </metric-data> </metric-datas> Metric Calls example of wildcards at various level:  1.Six wild card (|*|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|BusinessTransactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices| METRIC-NAME</metricPath> 2.Five wild card (|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes | METRIC-NAME</metricPath> 3.Four wild card (|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices |BVLInterfaceWS2.Process|METRIC-NAME</metricPath> 4.Three wild card (|*|*|*)  <metricPath>Business TransactionPerformance|Business Transactions| APPDInterface_PROD/APPDWebServices|METRIC-NAME</metricPat> 5.Two wild card (|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions |METRIC-NAME</metricPath> 6.One wild card (|*)  <metricPath>BusinessTransactionPerformance|METRIC-NAME</metricPath> Request  to keep metrics under 50K:  https://<HostName>/controller/rest/applications/<ApplicationID>/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C TOMCAT-APP %7C Transactions.SA %7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true If REST Call is fetching more than 50K metrics, you can replace wild characters with hard-coded values (e.g. TOMCAT-APP & Tranaction.SA), which will control metric count.       Find a test API script on Github here:  Appdynamics / metric-loader   ... View more

The AppDynamics Innovation Labs Team strives to develop and cultivate innovation tools, skills, and resources across the organization to drive self-disruption. I recently spoke with Jen Leung who filled me in on the 2018 Hackathon, first of many opportunities this year for AppDynamos to combine work and play.   Q. Jen, thank you so much for speaking with us. Can you talk a little about the 2018 Hackathon? What did the Innovation Labs Team try to achieve?    A. Hacky New Year 2018   was a cultural hackathon activity to  energize the San Francisco office  by  cultivating creativity and teamwork  in a  safe and inspiring environment.   Our goal was to host an event that would encourage every participant to  have fun, meet & work with new AppDynamos, and get creative!   93 participants formed 24 teams in the San Francisco office and came up with new and creative ideas. 83% of participants formed teams with people they do not work with, and at least 13% pulled in AppDynamics employees who usually work remotely.   The rules were simple:  Teams must be 2-6 people (no individuals) across 2 or more departments (which was left open to interpretation) and there was no hacking allowed before the day of the hackathon.         Q. How did this event differ from hackathons in previous years?   A.   This year we opened the hackathon to the entire company, whereas in years past the hackathons tended to be mostly engineering teams and R&D. We encouraged teams to pursue any ideas, not limited to code. We also did away with the Shark Tank judging format and instead went with more of a science fair that allowed participants to roam around and check out each other's creations.          Q. How did you determine the winners?   A.   We gave out multiple awards including  Crowd Fave, Team Spirit award, and the Power of the Brains award which highlighted outstanding creative and technical achievement using the collective brain power of all members.    The Team Spirit and Crowd Fave awards went to " Team Bletchley Park" for their innovative project preserving vote results integrity with a fraud and tamper-proof voting system on the Ethereum blockchain! You can read more about their work here: "Hackathoning   with IoT, Blockchain, and AppDynamics  by Swathi Bhat".   Team "It's EC2 Save Money" also won the Crowd Fave as well as the Power of the Brains award for unlocking infrastructure cost efficiencies via node-level cost transparency and cost-savings recommendations.    Team "It's EC2 Save Money"   Another Crowd Fave was team "Extensionators" for their smart recommendations for tailored AppD monitoring solutions via backend detection and a next-gen extensions platform.    Team "Electric Boogaloo" was awarded a Power of the Brains award for enforcing conference room etiquette by integrating room calendars with “smart” light bulbs.    Power of the Brains went to team "Say Cheese" for working on gauging employee sentiment around the office with emotion detection via live video.   Team "Cask-Lager-Ipa" won a Team Spirit award for making AppD configuration and daily performance status checks easier for the IT crowd...taking our product to where they are...command line interface for AppD.    And team "Mad Hatters" came up with an idea to have fun and do good by sewing and selling hats to raise donations for the Children’s Hospital.    There were many more teams working on fun, innovative projects who participated in the hackathon.   Mad Hatters     Q. What sort of feedback did you receive after the hackathon?   A. We heard from many participants that the event was fun, had a great turnout, and encouraged people to learn new skills. The format of the hackathon encouraged technical and non-technical participants to mix. The downside was timing, as some reported that they were not able to participate due to a heavy workload. In the future, we'd like to provide more feedback on the judging scores and also keep the originality of ideas in mind.       ... View more

Contents: Agent is not reporting to Controller Standalone Machine Agent is not reporting metrics Agent log file is empty Connectivity issues Permission issues Machine Agent reports all metrics as zero     Q. The agent is  not reporting to the Controller. What steps should I take to troubleshoot this issue?   A. Look for the following error messages in the agent logs and  check these items on the machine where the agent is installed: How to troubleshoot agent not reporting to the Controller       Q. How do I resolve the Standalone Machine Agent not reporting metrics as expected?   A. There are numerous reasons why a Machine Agent would not report to the Controller. Find solutions to the most common issues here: Machine Agent Not Reporting       Q. What do I do when the  agent log file is empty or nothing is being reported to the Controller?   A. The solution could be anywhere from directory installation to network connectivity. Find solutions to the most common issues here: App Agent Not Reporting       Q. What happens when the agent is not able to communicate with the Controller due to connectivity issues?   A. If you are seeing the following error message in the logs, check the SSLEnabled property in controller-info.xml file or any parameters configured in startup files. Ensure that SSLEnabled is set to true   for HTTPS, and false for HTTP.   [Thread-0] 22 Nov 2017 09:09:43,469 ERROR ConfigurationChannel - Exception: athenetest.saas.appdynamics.com:443 failed to respond org.apache.http.NoHttpResponseException: athenetest.saas.appdynamics.com:443 failed to respond     Q. What are some other common reasons why an agent would not report to the Controller?   A. If the agent logs are empty, it is possible that the user starting the application server does not have correct permissions on the agent folder. Check permissions in the agent folder and sub-folders.   Additionally, check for any connectivity issues, c heck the application name, tier name  and  node name, and  make sure the Controller host and port are correct. If multi-tenant Controller, check the account settings.        Q. What would cause the Machine Agent to report all metrics as zero?   A. Use the following steps to troubleshoot the issue: Verify in the machine agent log that the Machine Agent is only collecting zero-values. Change permissions on the machine agent installation folder for the user ID which the Machine Agent was started under  chmod -R 777   <machine-agent-install > . Restart the Machine Agent and verify that the zero values are no longer being reported. If restarting does not help, then d isable Sigar Hardware monitoring located in the  JavaHardwareMonitoring  directory. In the monitor.xml file (located at: <machine-agent-install>/monitors/JavaHardwareMonitor/* ) change the enabled property to false. Example: <monitor>  <name>SigarHardwareMonitor</name>  <type>managed</type>  <enabled>true</enabled> ---> change this to false –> <enabled>false</enabled> And enable OS-specific hardware monitoring found in the HardwareMonitoring directory.  In monitor.xml file (location: <machine-agent-install>/monitors/HardwareMonitor/* )  <monitor>  <name>HardwareMonitor</name>  <type>managed</type>  <enabled>false</enabled> -- -> change this to true – > <enabled>true</enabled> Restart the Machine Agent and verify that the zero values are no longer being reported.   More Troubleshooting Guides: FAQs Installation and Start-up     Published 2/26/2018 ... View more