Discussion Feed

Table of Contents Use Cases Problem Solution Example API script   Many of our customers use the  Controller Metrics and Events API to retrieve metric data information and information on various types of activities in a monitored environment, including Controller events. This article outlines a customer's use case and includes a generalized example script.    Use Case   The customer is using the AppDynamics metrics API every minute to load the metric data from the Controller to an external system. The customer has multiple Controllers in the AppDynamics SaaS environment and hosts hundreds of applications. They load dozens of metrics from each application.   The customer is using the API for the following use cases: Use a third-party tool for troubleshooting in cases where they need 1-minute granularity for longer than 24 hours. For example, the customer often needs to perform root cause analysis but does not get to it until a few days after the incident occurred.  Use third-party tools to calculate percentiles, consolidate metrics across different controllers at the app or tier level as required and create alerts. Use  a third-party tool  to retain data for several years. Since their peak periods occur just once per year, customers might need data to be retained for 3 years to provide meaningful comparisons and plan their future deployments better.   Problem   The customer uses a set of python scripts to retrieve the metrics. They consistently run into timeout errors and worry that the APIs are not scaling to their requirements.     Our team engaged with the customer and offered to review their process and scripts. Upon review, we found that the python script opens 1000+ of concurrent connections to the Controller. There are only 10 threads dedicated to processing the REST requests on the server side.  As a result, the threads wait indefinitely and then timeout.     Solution   Use a thread pool to invoke the REST APIs. This way the customer will have control on the concurrency. This should help to sustain 10 concurrent requests on the server side, getting the best performance.   Decrease the frequency of the job. For example, run the job every 5 minutes rather than every one minute.  We recommend restricting the number of Metric data points returned per REST call to less than 50K.   Metric REST API Example A RESTful API is an application program interface (API) that uses HTTP requests to GET data. The following section gives some examples of wildcards usage on Metric Data REST API. (This section scrolls horizontally.) The same example is included in the attached Metric REST API Example document for easy copy and paste.     URL:  /controller/rest/applications/100/metric-data?metric-path=<metric-path>&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Method: GET URL Parameters:  metric-path: Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A time-range-type: BEFORE_NOW duration-in-mins: 1 rollup=true Success Response: Response headers: HTTP/1.1 200 OK Server: AppDynamics Content-Type: application/xml Error Response:  REST endpoints can fail is many ways. From unauthorized access to wrongful parameters etc. Listed few error types: HTTP/1.1 403 Forbidden Server: AppDynamics Content-Language: Content-Type: text/html Incorrect Parameters: Data type ("text"|"bin"|""): text Response code: Non HTTP response code: java.net.URISyntaxException Response message: Non HTTP response message: Malformed esAPPDe Sample Request Call: GET   https://hostname/controller/rest/applications/100/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true Cookie Data: JSESSIONID=c45eeb9747d8aa24338bc2b973cf; X-CSRF-TOKEN=9a3dddf1c54ebca291c46ab10522faa7bae7c477 Request  Headers: Connection:keep-alive Accept-Language:en-US,en;q=0.5 Accept-Encoding:gzip, deflate User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0 Accept: application/json, text/plain, */* Authorization: Basic Q2FwaXRhbE9uZS1Qcm9kJTQwQ2FwaXRhbE9uZS1Qcm9kOnRlc3QxMjM= X-CSRF-TOKEN:9a3dddf1c54ebca291c46ab10522faa7bae7c477 Sample Response Data:  <metric-datas><metric-data>  <metricId>1578650</metricId>  <metricPath>Business Transaction  Performance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|Individual  Nodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices|Average Block  Time(ms)</metricPath>  <metricName>BTM|BTs|BT:23889|Component:617|Average  Block Time (ms)</metricName>  <frequency>ONE_MIN</frequency>  <metricValues>           <metric-value>           <startTimeInMillis><timeInMilliseconds></startTimeInMillis>           <value>0</value>           <min>0</min>           <max>0</max>           <current>0</current>           <sum>0</sum>           <count>0</count>          <standardDeviation>0.0</standardDeviation>           <occurrences>0</occurrences>           <useRange>true</useRange>           </metric-value>  </metricValues> </metric-data> </metric-datas> Metric Calls example of wildcards at various level:  1.Six wild card (|*|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|BusinessTransactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices| METRIC-NAME</metricPath> 2.Five wild card (|*|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes | METRIC-NAME</metricPath> 3.Four wild card (|*|*|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices |BVLInterfaceWS2.Process|METRIC-NAME</metricPath> 4.Three wild card (|*|*|*)  <metricPath>Business TransactionPerformance|Business Transactions| APPDInterface_PROD/APPDWebServices|METRIC-NAME</metricPat> 5.Two wild card (|*|*)  <metricPath>BusinessTransactionPerformance|Business Transactions |METRIC-NAME</metricPath> 6.One wild card (|*)  <metricPath>BusinessTransactionPerformance|METRIC-NAME</metricPath> Request  to keep metrics under 50K:  https://<HostName>/controller/rest/applications/<ApplicationID>/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C TOMCAT-APP %7C Transactions.SA %7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true If REST Call is fetching more than 50K metrics, you can replace wild characters with hard-coded values (e.g. TOMCAT-APP & Tranaction.SA), which will control metric count.       Find a test API script on Github here:  Appdynamics / metric-loader   ... View more

The AppDynamics Innovation Labs Team strives to develop and cultivate innovation tools, skills, and resources across the organization to drive self-disruption. I recently spoke with Jen Leung who filled me in on the 2018 Hackathon, first of many opportunities this year for AppDynamos to combine work and play.   Q. Jen, thank you so much for speaking with us. Can you talk a little about the 2018 Hackathon? What did the Innovation Labs Team try to achieve?    A. Hacky New Year 2018   was a cultural hackathon activity to  energize the San Francisco office  by  cultivating creativity and teamwork  in a  safe and inspiring environment.   Our goal was to host an event that would encourage every participant to  have fun, meet & work with new AppDynamos, and get creative!   93 participants formed 24 teams in the San Francisco office and came up with new and creative ideas. 83% of participants formed teams with people they do not work with, and at least 13% pulled in AppDynamics employees who usually work remotely.   The rules were simple:  Teams must be 2-6 people (no individuals) across 2 or more departments (which was left open to interpretation) and there was no hacking allowed before the day of the hackathon.         Q. How did this event differ from hackathons in previous years?   A.   This year we opened the hackathon to the entire company, whereas in years past the hackathons tended to be mostly engineering teams and R&D. We encouraged teams to pursue any ideas, not limited to code. We also did away with the Shark Tank judging format and instead went with more of a science fair that allowed participants to roam around and check out each other's creations.          Q. How did you determine the winners?   A.   We gave out multiple awards including  Crowd Fave, Team Spirit award, and the Power of the Brains award which highlighted outstanding creative and technical achievement using the collective brain power of all members.    The Team Spirit and Crowd Fave awards went to " Team Bletchley Park" for their innovative project preserving vote results integrity with a fraud and tamper-proof voting system on the Ethereum blockchain! You can read more about their work here: "Hackathoning   with IoT, Blockchain, and AppDynamics  by Swathi Bhat".   Team "It's EC2 Save Money" also won the Crowd Fave as well as the Power of the Brains award for unlocking infrastructure cost efficiencies via node-level cost transparency and cost-savings recommendations.    Team "It's EC2 Save Money"   Another Crowd Fave was team "Extensionators" for their smart recommendations for tailored AppD monitoring solutions via backend detection and a next-gen extensions platform.    Team "Electric Boogaloo" was awarded a Power of the Brains award for enforcing conference room etiquette by integrating room calendars with “smart” light bulbs.    Power of the Brains went to team "Say Cheese" for working on gauging employee sentiment around the office with emotion detection via live video.   Team "Cask-Lager-Ipa" won a Team Spirit award for making AppD configuration and daily performance status checks easier for the IT crowd...taking our product to where they are...command line interface for AppD.    And team "Mad Hatters" came up with an idea to have fun and do good by sewing and selling hats to raise donations for the Children’s Hospital.    There were many more teams working on fun, innovative projects who participated in the hackathon.   Mad Hatters     Q. What sort of feedback did you receive after the hackathon?   A. We heard from many participants that the event was fun, had a great turnout, and encouraged people to learn new skills. The format of the hackathon encouraged technical and non-technical participants to mix. The downside was timing, as some reported that they were not able to participate due to a heavy workload. In the future, we'd like to provide more feedback on the judging scores and also keep the originality of ideas in mind.       ... View more

Q. The agent is  not reporting to the Controller. What steps should I take to troubleshoot this issue?   A. Look for the following error messages in the agent logs and  check these items on the machine where the agent is installed: How to troubleshoot agent not reporting to the Controller     Q. How do I resolve the Standalone Machine Agent not reporting metrics as expected?   A. There are numerous reasons why a Machine Agent would not report to the Controller. Find solutions to the most common issues here: Machine Agent Not Reporting     Q. What do I do when the  agent log file is empty or nothing is being reported to the Controller?   A. The solution could be anywhere from directory installation to network connectivity. Find solutions to the most common issues here: App Agent Not Reporting     Q. What happens when the agent is not able to communicate with the Controller due to connectivity issues?   A. If you are seeing the following error message in the logs, check the SSLEnabled property in controller-info.xml file or any parameters configured in startup files. Ensure that SSLEnabled is set to true   for HTTPS, and false for HTTP.   [Thread-0] 22 Nov 2017 09:09:43,469 ERROR ConfigurationChannel - Exception: athenetest.saas.appdynamics.com:443 failed to respond org.apache.http.NoHttpResponseException: athenetest.saas.appdynamics.com:443 failed to respond     Q. What are some other common reasons why an agent would not report to the Controller?   A. If the agent logs are empty, it is possible that the user starting the application server does not have correct permissions on the agent folder. Check permissions in the agent folder and sub-folders.   Additionally, check for any connectivity issues, c heck the application name, tier name  and  node name, and  make sure the Controller host and port are correct. If multi-tenant Controller, check the account settings.        Q. What would cause the Machine Agent to report all metrics as zero?   A. Use the following steps to troubleshoot the issue: Verify in the machine agent log that the Machine Agent is only collecting zero-values. Change permissions on the machine agent installation folder for the user ID which the Machine Agent was started under  chmod -R 777   <machine-agent-install > . Restart the Machine Agent and verify that the zero values are no longer being reported. If restarting does not help, then d isable Sigar Hardware monitoring located in the  JavaHardwareMonitoring  directory. In the monitor.xml file (located at: <machine-agent-install>/monitors/JavaHardwareMonitor/* ) change the enabled property to false. Example: <monitor>  <name>SigarHardwareMonitor</name>  <type>managed</type>  <enabled>true</enabled> ---> change this to false –> <enabled>false</enabled> And enable OS-specific hardware monitoring found in the HardwareMonitoring directory.  In monitor.xml file (location: <machine-agent-install>/monitors/HardwareMonitor/* )  <monitor>  <name>HardwareMonitor</name>  <type>managed</type>  <enabled>false</enabled> -- -> change this to true – > <enabled>true</enabled> Restart the Machine Agent and verify that the zero values are no longer being reported.   More Troubleshooting Guides: FAQs Installation and Start-up     Published 2/26/2018 ... View more

This collection of articles will help users who are installing a new agent, upgrading an existing agent, or trying to resolve start-up failure.    Installation     Q. I get the following error message when installing a Machine Agent. How should I proceed?  Failed to find Premain-Class manifest attribute in /usr/share/appdynamics/machineagent.jar Error occurred during initialization of VM   A. The error message shows that the Machine Agent jar was used as a Java Agent. >java -javaagent:d:\AppDynamics\installs\3.5.6\MachineAgent\machineagent.jar -version Failed to find Premain-Class manifest attribute in d:\AppDynamics\installs\3.5.6\MachineAgent\machineagent.jar Error occurred during initialization of VM.  Agent library failed to init: instrument   Verify the command that is being used to start the Machine Agent. Follow the documentation to Install the Standalone Machine Agent. Open a separate command prompt and start the Machine Agent using  java -Dmetric.http.listener=true -jar machineagent.jar command.     Q.  Is there a way to install the appDynamics database agent without the wizard, such as simply configuring the installation XML files?  I want to monitor a client but I am having trouble when installing the agent. I cannot log in to the DB from outside the network and opening up the listener port on the DB for me right now is an issue. Is there a workaround?   A. You do not necessarily have to install the Database Agent on the database server. The database agent can reside on any server as long as that server can communicate to the database host. See Database Visibility System Requirements.    Q. I'm having trouble installing a PHP Agent. What steps should I take?   A. After PHP Agent installation there are a few cases when the agent does not report to the Controller. Use the following article for basic troubleshooting steps to verify the installation and place the agent configuration files in proper place. PHP Agent installation troubleshooting guide.   Q. Is it possible to deploy an agent using Puppet?   A. Every deployment is different,  so this task doesn't lend itself to one master Puppet script maintained by AppDynamics. However, here is a sample that can help you get started on your own Puppet module. Agent deployment using Puppet.   Upgrade   Q. What are the steps to upgrade a mobile agent?   A. For iOS and Android, follow these steps. Upgrading Mobile Agents   Q. Where can I find information on how to upgrade an agent?    A. The following documentation links will walk you through the requirements and steps for upgrading an agent. Upgrade the .NET Agent Upgrade the Node.js Agent Upgrade the Java Agent Upgrade the PHP Agent Upgrade the Python Agent Upgrade Analytics Agent Upgrade the Database Agent Upgrade the Standalone Machine Agent Upgrade the Apache Agent     Start-up   Q. My Analytics Agent fails to start and throws the following error message. What steps should I take? java.lang.NoSuchMethodError: io.dropwizard.logging.LoggingFactory.bootstrap   A. This error message is normally caused by the presence of a duplicate jar file from a different version of the Analytic Agent.    It can occur when the user unzips a new version of their machine-agent.zip file into the same directory where the older Machine Agent resides, creating a duplicate presence of the same jar files.   The solution is to first take a backup of the old agent, then delete or rename the Machine Agent, and then unzip the new machine-agent.zip  .      Q. My Analytic Agent fails to start and shows the following error message in the logs. What do I do? Analytics agent failed to connect to the controller registration endpoint.   A. This issue occurs when using a self-signed certificate for the Controller. The Analytics Agent uses a truststore for the JRE to validate the certificate of the connecting server. The truststore does not have self-signed certificates added to the store by default.   Import the client certificate to the JRE truststore, which is located in the Java directory used by the Analytics Agent by default. Example:   JAVA_HOME/lib/security/cacerts        More Troubleshooting Guides: FAQs Reporting and Connectivity     Published 2/26/2018     ... View more

 Q. Is there a way to  delete an abandoned Machine Agent when it is not attached to any APM agents?    A. Yes, t here are two ways to delete the abandoned machine agent, either from controller UI by creating a dummy application or through a manual step to delete the agent from the database. The steps for both are located here:  Delete abandoned machine agent.     Q.  What is the best way to setup Alerts for App Agent Status and Machine Agent Status?    A. There are two solutions for this, found here: Agent Status on .NET      Q. What does the error message "account key doesn't match with the current account for machine id" mean?   A. The message indicates that account key generated by AppDynamics controller during the registration may be changed at this point in time. This could happen if you migrate the Controller without shutting down the agents. Restart the agent to resolve this error.     Q. How to resolve Agent causing overhead or crash.   A. Follow these steps to troubleshoot: Check the agent logs for any communication errors between Agent and Controller. Make sure ALD has not been turned on for a long time. The Agent requires 30 MB of permgen space. Set this parameter ( -XX:MaxPermSize=<value> ) to a higher value (256m at least) and try again. Get the crash report and heap dump for analysis. Check the agent log level is set to INFO (default log level value if not changed to debug/trace) at agent_dir/conf/logging/log4j.xml . Make sure periodic sampling and occurrence request sampling is not set to a low value (default 100 requests and 10 min, update/disable as per requirement). Check the agent logs if any repetitive exceptions are logged. Disable the agent from the node dashboard and monitor the heap usage whether the overhead/crash is still happening to find whether agent/webapp instrumentation is raising the issue.     Q. When requesting agent logs in <Application> -> Servers -> App Servers -> <tier> -> <node> -> Agents, the log status times out. One possible cause is someone first requested the log and reset the agent before the log sends to the Controller.   A. Reset agent before requesting the agent log.         Q. Is it possible to assign a Machine Agent to multiple applications on the same server?   A. We need to run   one machine agent process for each application   on a machine. Add the VM parameter ( -Dappdynamics.agent.uniqueHostId=<<some unique value per application>> ) to both Machine Agent and the App Agent. Provide the same value to the machine - app agent pair. For example, if you have 2 nodes (n1 and n2 ) reporting to applications (a1 and a2). Let's say m1 and m2 are your machine agents for the corresponding nodes. You will need to pass   -Dappdynamics.agent.uniqueHostId=a1   for n1 and m1. and   -Dappdynamics.agent.uniqueHostId=a2   for n2 and m2. The uniqueHostId value can be any unique value but should be different for each application. After restarting the JVMs with the above options you should see machine agents associating with corresponding applications.      Q.  I want to get the list of AppD agents. I mean, we can see those list of agents in settings, but I want to get all of those agents to an Excel sheet. Is there any way that we can export the list of agents and details?    A. It's not out of the box,  but there is a workaround. Create small program which can read  /controller/rest/applications - List of all applications and get list of nodes from /controller/rest/applications/<application-name|application-id>/nodes Go to AppDynamics Agents page. Take screenshot. Trim the picture. Copy it to One Note. Right-click and Copy to Text.   More Troubleshooting Guides: Installation and Start-up Reporting and Connectivity   Published 2/26/2018 ... View more