To configure Apache Solr version 6.5.x with AppDynamics Agent (preferably version 4.2.x or higher), use the following steps:
Navigate to /<solr-home>/bin/solr.in.sh on the Controller machine.
Add the following snippet to the solr.in.sh file in order to add the Java Agent.
SOLR_OPTS="$SOLR_OPTS -Dappdynamics.agent.applicationName=Solr -Dappdynamics.agent.tierName=SolrTier -Dappdynamics.agent.nodeName -javaagent:/Users/XXX/Documents/Agent/AppServerAgent-220.127.116.11/ver18.104.22.168/javaagent.jar"
... View more
A legacy Machine Agent and Java Application Agent are installed on a machine. Despite correct installation and configuration, one or both of the Agents do not report. The first step in troubleshooting this issue is to validate the Unique Host ID, which is a problem more common with Machine Agents.
Machine Agents use the Java API to get the Unique Host ID. Sometimes the same JVM can return different values for the same machine each time the Machine Agent is restarted. One possible cause is the Unique Host ID of the App Agent and the Machine Agent are not in sync. In this case, the Machine Agent does not know which application to register.
Search for the application or the Unique Host ID on both the Application and Machine Agents, and make sure that the values are the same.
To confirm the issue, go to the Agent settings page located at either of the following URLs:
http:// <host:Port> /#/location=SETTINGS_MACHINE_AGENTS
Also, verify that the Machine Agent is associated with the application.
If there is any discrepancy between the two Agents, resolve the issue by providing the same Unique Host ID value for the App Agent and the Machine Agent using the following steps.
Note: If it is not feasible to stop the App agent, it is possible to sync the Unique Host ID value of the Machine Agent with the App Agent, and then apply the property to the App Agent on the next restart.
Stop both Agents.
Start the App Agent with the following property:
-Dappdynamics.agent.uniqueHostId=<your-host-id> system property
Confirm that there are no new errors or issues with the App Agent.
Start the Machine Agent with the same property:
- Dappdynamics.agent.uniqueHostId=<your-host-id> system property
Another option is to add a <unique-host-id>your-host-id</unique-host-id> element within the <machine_agent_install>/conf/controller-info.xml file.
... View more
Custom metrics allows the user to specify a SQL query to run on a monitored DB every minute, and collect the results of the query.
This document specifies restrictions and limitations, as well as some examples and debugging techniques.
Restrictions and Limitations:
The query must always return a positive integer number.
The query should not return more than one column.
The user can define up to 20 custom metrics per monitored DB.
This limit is configurable through admin.jsp with the following property; however, it should be increased graciously.
For Security reasons, we only allow SELECT queries.
How to debug?
If the Agent encounters an error while running the query, the Agent will send an error event to the Controller.
View these error events by clicking on Database -> Events in the Controller UI.
For more details on an error, view the database Agent logs located at <db_install_dir>/logs .
The errors are self-descriptive and can be searched using the keywords " CustomMetricsCollector," "ERROR," or "WARN."
[<MyTestMysql>-Scheduler-3] 29 May 2017 11:04:59,702 ERROR CustomMetricsCollector - Error collecting data for database 'MyTestMysql' java.sql.SQLException: Invalid custom metric: test4 , actual data type is non-numeric and expected data type is numeric (Custom Metric : test4)
09 Jun 2017 13:40:08,889 ERROR [<Database>-Scheduler-1] CustomMetricsCollector:130 - Error collecting data for database 'Database' java.sql.SQLException: DB2 SQL Error: SQLCODE=-952, SQLSTATE=57014, SQLERRMC=null, DRIVER=4.16.53 (Custom Metric : TestMetric)
Query Examples with MySQL:
1.select COUNT(1) "total" from dual; - Worked Output: 1 2. select 9+ ' ' +':00' num from dual; - Worked. Output: 9 3. select name from user where id=3 - Not working as expected Output: root
4. select id as node_id from application_component_node where id NOT in (select distinct(node_id) from metricdata_hour where ts_min > ((select max(ts_min) from metricdata_hour)) - 60*2) - Worked/Not Worked based on output
Output: Fails if returns multiple records
Database monitoring metrics
Configuring custom metrics
... View more
We will no be able to update QUERYTIMEOUTINTERVAL under DBagent. Is it possible for you to run it through DB2 Command line?
db2 UPDATE CLI CFG FOR SECTION COMMON USING QUERYTIMEOUTINTERVAL 0
I would not recommend it to do if databtase is production but would prefer that query can be modified to validate.
Also, I believe it must be single connection as numeric metrics are created on per database instance basis but I will get confirmation from our dev team to be 100% sure
Please keep us posted on how it goes once the query execution time is minimzed. For testing purpose, you can also create a simple custom metrics and validated. We believe it should report seamlessly.
... View more
Applogies for confusion .A custom metrics allows the user to specify a SQL query to run on the monitored DB every minute, and collect the results of the query in a custom metric. From the error it seems like query might be taking longer than that leading to metrics not being collected for that minute. Complex long running statement may fail with SQL0952N error message. I am not sure about the query as it's not shared but Can we try and index the query further?
Also, while searching for the error, I could see two alternatives
1. Increasing QUERYTIMEOUTINTERVAL to a larger value based on how long it is expected for the SQL statement to complete.
2. Disable query timeouts itself. QUERYTIMEOUTINTERVAL=0
Please let us know if this helps. Also,it would be great if you could share db agent logs in debug mode and screenshot of config or query pattern with us.
... View more
Could you please enable debug logging?
Please edit the file <MachineAgent>/conf/logging/log4j.xml . Change the level value of the following <logger> elements to debug .
Please let it run for 10 mins and then zip and upload the All the logs in the directory MachineAgent/logs/*
Also, can you please share screenshot of custom metric and screenshot from metric browser?
... View more
Linux users can use the following steps to determine which of their Java threads is consuming the most CPU.
From your computer terminal:
1. First, find the Java process ID.
ps -ef |grep java
2. Use the Java process ID (PID) to pull the lightweight processes into a file named lwp.txt .
ps -eLo pid,lwp,nlwp,pcpu,etime,args|grep <pid> > lwp.txt
3. Use the PID to generate a Java thread dump.
kill -3 <pid>
4. Choose the lightweight process (LWP) that is consuming the most CPU from the lwp.txt file created in step 2, and convert the LWP ID from decimal to hexadecimal using the tool of your choice (for example, binaryhexconverter.com).
Example: Convert LWP DEC 4235 to HEX 108B.
5. Search the thread dump created in step 3 for the hexadecimal LWP ID to find the Java thread that is consuming the most CPU.
6. If the problematic Java thread is part of AppDynamics, contact support for further assistance. If it is an application thread, contact your development/application team to debug the issue further.
... View more
Use the following recommendations if high CPU usage occurs after attaching the AppDynamics agent:
Disable aggressive slow snapshot collection
Exclude specific hotspot interceptors
Disable unwanted instrumentation from transaction detection
Turn off Turbo custom exit point interceptor
1. Disable aggressive slow snapshot collection
If aggressive slow snapshot collection is enabled, the agent will retain selected call graph segments that precede the detection of a slow, stall or error transaction condition. Disabling the aggressive slow snapshot collection will significantly reduce overhead.
In the Controller UI, click on Configuration in the left navigation bar.
Click the double arrows in the top right to expand the menu.
Click on Call Graph Settings.
Uncheck the checkbox next to "Enable Aggressive Slow Snapshot Collection."
Click the save button.
2. Exclude specific hotspot interceptors
Editing the node level property "exclude-interceptors" to exclude hotspots will reduce CPU usage.
To determine if your agent has enabled hotspots, search the agent.* log files for "Enabled hotspots" and search the ByteCodeTransformer*.log for "Applying method interceptor diag.snapshot.BoundHotspot."
Edit a node property:
In the controller UI, click on Tiers and Nodes in the left navigation bar.
Double click on the node you'd like to configure, and click Agents > App Server Agent > Configure. This will open the App Server Configuration window.
At the node level, click on the "Use Custom Configuration" button.
Either search for the exclude-interceptors property and double click on it, or click on the gray plus sign to create a new agent property if the exclude-interceptors property does not already exist.
Click the save button after adding the new configuration. A "saved successfully" message will appear in the top right corner of the window.
Name - exclude-interceptors Description - exclude-interceptors Type - String Value - com.singularity.BoundHotspotInterceptor
Note: If there are already some class names added to this property, use a comma between them.
3. Async Instrumentation
Too many async interceptors can cause high CPU utilization.
You may have unnecessary application framework classes instrumented that are not needed for visibility or monitoring. Get a list of classes/methods which are applying async interceptor by looking at the ByteCodeTransformer*.log or search "async.handoffAsyncHandOffExecutionTracker." E xclude the packages 1-1 to improve CPU utilization.
The package/class can be excluded either from fork-config OR using node property.
Fork-config changes (recommended) - Requires JVM restart.
Navigate to <agent_install_dir>/<ver4.X.x>/conf/app-agent-config.xml .
Search for <fork-config> keyword.
Add the desired packages to be excluded.
<excludes filter-type="STARTSWITH" filter-value="<package/>"/> or fully qualified class name Few Examples: <excludes filter-type="STARTSWITH" filter-value="com.arjuna/"/> <excludes filter-type="STARTSWITH" filter-value="com.netflix/"/> <excludes filter-type="STARTSWITH" filter-value="com.bea/,com.weblogic/,weblogic/,com.ibm/,net/sf/,com/mchange"/>
Node Property - See step 2 to create or edit a node property.
Name: thread-correlation-classes-exclude Description: thread-correlation-classes-exclude Value:Type: <fully.qualified.package.ClassName>,<package.name>.*
Note: Excluding packages and classes could result in loss of visibility.
4. Disable unwanted instrumentation from transaction detection
Spring Bean and EJB interceptors are CPU intensive. If you are not looking for any spring or EJB entry points, disable Transaction Detection.
From the controller UI, click on Applications.
Click Configuration in the left navigation.
Under the Transaction Detection tab, uncheck both of the checkboxes next to Spring Bean and EJB to disable them.
5. Turn off Turbo custom exit point interceptor
Turbo exit points are special interceptors that handle a high volume of calls. To determine if these interceptors are causing high CPU usage, search the ByteCodeTransformer*.log for "exit.TurboCustom." If the classes are not adding visibility, exclude them and disable the interceptors by creating node-level properties (see "Edit a node property " in step 2 ).
Name - exclude-interceptors Description - exclude-interceptors Type - String Value - com.singularity.TurboCustomExitPointInterceptor
Name - disable-ootb-turbo-interceptors Description - disable-ootb-turbo-interceptors Type - boolean Value - true
If these troubleshooting recommendations do not improve your high CPU usage, contact our support team for further assistance.
... View more
For saas controllers, you will not have access to controller logs. I did a quick validation on your saas controller https://548.saas.appdynamics.com/controller. It looksl you have installed machine agent and not app agent. Business Transaction Discovery is supported on Java + .NET agents only and not on machine agents. Please find supporting screenshot attached.
More details on transaction discovery tool can be found on below link
We hope it clarifes your doubt.
Please keep us posted.
... View more
Also Matt, it seems Annie Pidgeon have opened a support case for the same
https://appdynamics.zendesk.com/agent/tickets/91212 - subject : Re: Failure setting up transaction discovery
It would be great if you could reply on the support case with above mentioned details. I will keep you in loop so you should receive an email related to issue.
... View more