Discussion Feed

Some users may have multiple on-premise Controllers running on standby in order to limit downtime. These users may wish to use a load balancer, which decides how traffic should be channeled and protects each server from the risk of being overloaded.    Users with security concerns who wish to filter outgoing traffic to safe sources may wish to use a proxy or a firewall as a gateway to allow only certain traffic through.    This document aims to answer some of the most common questions received by the AppDynamics Support Team in regards to firewalls, proxies, and load balancers.   Examples: High Availability Controller - Load Balancer Requirements and Considerations Events Service cluster - NGINX sample configuration Events Service cluster - High Availability proxy sample configuration    Related Links: Controller Deployment - Network Requirements How to inject the EUM Javascript Agent with an F5 load balancer   FAQ:    Q.  I am unable to enable End-User Monitoring from an on-premise Controller. The error message says it didn't receive a response from <server>.eum-appdynamics.com .    A. The most common reason for this is due to a firewall in place between the Controller and the EUM Cloud. If you intend to use EUM in SaaS, open port 443 to allow the on-premise Controller to communicate with the EUM Cloud. For more information, including how to configure proxy settings for the Controller, review the documentation on "Connection Problems."     Q. How do I  set up a Controller behind a reverse proxy?   A. Follow these instructions when deploying an AppDynamics Controller behind a NGINX or Apache reverse proxy.     Q. What are the port settings for on-premise deployments behind a network firewall or load balancer?   A. SaaS users only need to adjust their infrastructure to accommodate the HTTPS port provided to you by AppDynamics. On-premise users may need to make additional adjustments outlined here: Port Settings.     Q. What are the IP addresses for a specific SaaS Controller? I'd like to monitor outgoing traffic from my application's environment to the AppDynamics SaaS Controller.   A. Find the SaaS IP ranges listed here.       Do you still have questions? Start a new discussion or look up existing conversations in the community's Infrastructure (Server, Database) Discussion Forum. ... View more

Question:   What is the AppDynamics process for performing maintenance on SaaS Controllers? How often does it happen and is there downtime involved?   Answer:     AppDynamics uses the following policy to alert customers to maintenance and upgrades.           Question:   Where can I view the SaaS service status?   Answer:   Find the SaaS Service Status       Question:   What kind of communications does AppDynamics use to send alerts to customers?   Answer:   The primary account admin is notified via email about any maintenance activity beforehand, and also after the activity is completed.       Related Links: The Top 5 Advantages of SaaS-based Application Performance Management SaaS Controller Secure Connections ... View more

Question:   Injecting special headers into an application payload is disturbing application functionality. How do I resolve this?   Answer:   Disable the exit call correlation based on the exit call that is being sent to the payload. There are two ways to disable exit call correlation in order to stop writing correlation headers to the payload. Refer to the following screenshot for disabling exit call correlation for IBM MQ and other backends.     Or, register a node property to disable the exit call correlation for some of the backend types. Node Property: disable-exit-call-correlation- for Type: String Default value: none (you could select HTTP, JMS,RMI) Platform: Java, .NET    For more information: App Agent node properties reference   Question:   I don’t see some of the async threads tracked on environments where Camel, Mule, JBoss ESB, Tibcow, or Scala are used. Please help!   Answer:   First, stop the Agent. Navigate to the app-agent-config.xml file located under agent-install-dir>/conf folder Add a list of async environment names separated by commas to the config file. This will enable async transaction detection for supported environments, which include all of the above. The snippet will look like this: <!--   Add comma-separated list of async environment names.     These will be used to enable async transaction detectionfor supported environments.     Current supported async environments: mule,camel,jbossesb,tibcobw, scala                    --> <!--     property name="enable-async- correlation-for" value="mule,camel,jbossesb, tibcobw,scala"/ --> Then add the following code above or below the previous snippet: <property name="enable-async-correlation-for" value="scala"/> Start the Agent.       Question:   How should one manage the fact that the read-only nature of JMS messages doesn’t allow for writing headers for attaining correlation?   Answer:   To modify the read-only JMS message, add the object's package (org.apache.activemq.ra) to the jms-packages-allowed-for-rewriting node property.   Example message: [WorkManager(2)-50] 17 Jul 2017 12:49:03,065 ERROR ReadonlyRewriting_ JMSMessageReflector - MessageNotWriteableException caught while setting transaction header.   Option 1 -  Register a node property with the following values: Property Name: jms-packages-allowed-for- rewriting Type: String Value   :<object’s package>   Example 1: Property Name   : jms-packages-allowed-for- rewriting Type: String Value   :org.apache.activemq.ra    Option 2 - Disable JMS correlation on an upstream tier, so that the Agent will not be allowed to write any special headers to the JMS payload. This option will not allow the user to see the correlation between producer and consumer.     Or register a node property to disable exit call correlation.  Node Property: disable-exit-call-correlation- for\ Type: String Default value: none (you could select HTTP, JMS,RMI) Platform: Java, .NET   For more information:  App Agent node properties reference   Question:   Why is there a correlation missing between Web Service calls between two different technologies? For example, sometimes when Java talks to .NET the correlation is not there.   Answer:   Register a node property with the following values:   Property Name: enable-soap-header-correlation Type: Boolean Value: true   This property controls the correlation with web service transactions. When enabled, a node which receives a web service transaction may correlate that transaction with any downstream transactions.   The ability to correlate depends on the particular web service framework. Currently, correlation is supported only by Apache Synapse and CXF frameworks.   When disabled, the Agent will not perform correlation through any web service tiers.     Related Links: JVM Language Frameworks Support JVM Support Monitor JVMs     ... View more

Those who use the  Self Service Buy  option can update their billing information, cancel a subscription, change the number of Agents for an account, switch from monthly to annual payments, or subscribe/unsubscribe to email notifications.    Here's how: Manage a Subscription or License ... View more

  The following article explains the considerations and challenges when using any application that is a heavy user of I/O, like AppDynamics Controllers running on Virtual Machines (VMs). It is applicable to AppDynamics version 4.3.x.   AppDynamics’ metric intake is sensitive  to storage latency at scale.   Specifically: AppDynamics controllers receive metrics every minute from every   Agent,   and are marked with a minute number. When the controller determines that it has buffered up packets for all the data for a given minute, it puts that data into an insert buffer. The insert buffer contains all the metrics (plus derived metrics), from all the nodes, for the same minute. The metric buffer is then sent to MySQL. The controller then issues SQL queries to roll the new data up to the tier level, collapse out the node IDs, then roll up to the application level, collapsing out the tier IDs, leaving Business Transactions (BTs). This all happens within a   minute,   since there is data dependency from node to tier to   application.   The result is if the 60 second (one minute) timeframe is exceeded by the execution of the path, system performance will degrade  nonlinearly .   The rule of thumb is that the filesystem must be able to absorb 200MB/sec per million metrics/min measured with fio. This is difficult to achieve on VM infrastructure above a few million metrics per minute.   We have a benchmark using FIO that closely approximates our workload available on request. Contact Professional Services or Support for assistance.   Recommendation:   At this time we do not recommend running large or extra large AppDynamics Controllers on virtual machines.    Scalability is a key differentiator of the AppDynamics compared to competing solutions. We continue to invest significant resources in improving metric ingestion at scale and are working on changes that will limit sensitivity to I/O latency.       Related Links: Prepare Virtual Machines for the Controller Hardware Requirements   by   Profile How to install Fio to measure disk I/O performance ... View more

AppDynamics licenses for on-premise Controllers are tied to the MAC address of the machine on which the Controller is installed. In rare cases, users may need to move the Controller and its license to a new machine.   In this event, you can update the MAC Address associated with your license on your subscription page at AppDynamics.com.       Update MAC Address   Users with an  on-premise paid permanent license can see the Manage MAC Address tab on their subscription page.     The primary user for the license in the account can see both the Edit and History action links. For other users, only the History action is available.   To change the MAC address for the license, click the Edit link. The Edit Mac Address pop-up window appears.       Change the MAC Address and click the Save button.         It is important to note that the MAC Address can be changed up to 3 times within a 12-month period. Contact your Account Owner if you have to change it more than 3 times per year.      The History action link shows a history of past MAC Address updates for the license.  ... View more

Symptoms   When starting the Events Service, the database does not report the events data. The following error message appears in the Controller UI under the Databases tab.       Additional troubleshooting in the Controller's server.log file indicates that the Events Service process has stopped.   0500|SEVERE|glassfish3.1.2|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=120;_ThreadName=Thread-5;|Encountered runtime exception com.appdynamics.analytics.shared.rest.exceptions.ClientException: Could not execute request to http://localhost:9080/v2/events/dbmon-wait-time atcom.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.getResponse(GenericHttpRequestBuilder.java:224)   atcom.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.executeAndReturnRawResponseString(GenericHttpRequestBuilder.java:238) atcom.appdynamics.analytics.shared.rest.client.eventservice.DefaultEventServiceClient.registerEventType(DefaultEventServiceClient.java:132)     Diagnosis and Troubleshooting   1. Find the process ID from the output of the ps -ef | grep -i event-service command 52676 11537 1 0 03:58 pts/3 00:00:11 /opt/AppDynamics/Controller/jre/bin/java -Xmx6144m -Xms6144m -Xss256k -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -verbose:gc -XX:GCLogFileSize=256m -XX:NumberOfGCLogFiles=4 -XX:+UseGCLogFileRotation -Xloggc:/opt/AppDynamics/Controller/events_service/bin/../logs/events-service-api-store-gc.log -DAPPLICATION_HOME=/opt/AppDynamics/Controller/events_service/bin/.. -classpath /opt/AppDynamics/Controller/events_service/bin/../lib/* com.appdynamics.analytics.processor.AnalyticsService -p /opt/AppDynamics/Controller/events_service/conf/events-service-api-store.properties -y /opt/AppDynamics/Controller/events_service/bin/../conf/events-service-api-store.yml 52676 19976 9765 0 04:38 pts/3 00:00:00 grep events-service   2. Check the health state of the Events Service. It did not respond to the request.   curl http://<event-service-host>:9081/healthcheck?pretty=true   3. Check if the host and port of the Events Service are bound correctly using the netstat command.  There was no response to the request.   netstat -anp | grep 9080   4. If the process had hung during startup, use the kill -3 52676 (kill -3 <PID>) command to capture thread dumps and look for the hung process.   Note: Java thread dump output does to "stdout" and will be written to the  nohup .out stdout file.   5. Within the thread dumps, find the hung process.   Example:   "main" #1 prio=5 os_prio=0 tid=0x00007fdb84011000 nid=0xe280 runnable [0x00007fdb88425000] java.lang.Thread.State: RUNNABLE at sun.nio.fs.UnixNativeDispatcher.stat0(Native Method) at sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286) at sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70) at sun.nio.fs.UnixFileStore.devFor(UnixFileStore.java:55) at sun.nio.fs.UnixFileStore.(UnixFileStore.java:70) at sun.nio.fs.LinuxFileStore.(LinuxFileStore.java:48) at sun.nio.fs.LinuxFileSystem.getFileStore(LinuxFileSystem.java:112) at sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(UnixFileSystem.java:213) at sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(UnixFileSystem.java:224) - locked <0x000000065610cb50> (a sun.nio.fs.UnixFileSystem$FileStoreIterator) at org.elasticsearch.env.NodeEnvironment.getFileStore(NodeEnvironment.java:267) at org.elasticsearch.env.NodeEnvironment.access$000(NodeEnvironment.java:62) at org.elasticsearch.env.NodeEnvironment$NodePath.(NodeEnvironment.java:75) at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:140) at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:165) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticSearchSingleNode.(ElasticSearchSingleNode.java:49) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticSearchSingleNode$$FastClassByGuice$$7b182632.newInstance() at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) - locked <0x00000006534a90f0> (a java.lang.Class for com.google.inject.internal.InternalInjectorCreator) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:75) at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) at com.appdynamics.common.framework.Loaders.internalPrepareAndPreStart(Loaders.java:181) at com.appdynamics.common.framework.Loaders.loadAndInitializeModules(Loaders.java:127) at com.appdynamics.common.framework.AbstractApp.run(AbstractApp.java:311) at com.appdynamics.common.framework.AbstractApp.run(AbstractApp.java:59) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:42) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:76) at io.dropwizard.cli.Cli.run(Cli.java:70) at io.dropwizard.Application.run(Application.java:72) at com.appdynamics.common.framework.AbstractApp.callRunServer(AbstractApp.java:267) at com.appdynamics.common.framework.AbstractApp.runUsingFile(AbstractApp.java:261) at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:248) at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:167) at com.appdynamics.analytics.processor.AnalyticsService.main(AnalyticsService.java:71)   6. When analyzing the stack trace, as the operating system admin, look for an issue with the Network File System (NFS) Mount.   7. Confirm that the NFS Mount was indeed hung. This was due to the NFS server migrating to the new host.     Solution Unmounting and remounting the NFS server with the correct mount point resolves the issue.   ... View more