Hi Arun, I am unable to access https://panjal.opsramp.net/ Hence not able to check the root cert that signed the private key for this URL. Could you please share a screenshot of certificate chain and also attach the root and intermediate certificates? Regards, Saradhi
... View more
You can modify connection settings through the Enterprise Console UI. The Enterprise Console will automatically update all occurrences in the controller for you. You do not need to manually update all the files and manage the sequence to restart services.
You can also edit the ports manually by editing configuration files used by the application server for the Controller domain. Updating the ports manually, however, will cause the Enterprise Console to have no visibility into the updates and cause health check errors.
... View more
1) In admin.jsp 😞 https://docs.appdynamics.com/display/PRO45/Access+the+Administration+Console)
setting "disable.historic.transactional.flow:" to "false" if it is set to "true" in "controller settings" used to enable this
2) But in 4.5.2 even with this property set to false there was an issue loading the flowmaps beyond 24 hours. This is being fixed as part of jira UIADMIN-1238 with high priority.
... View more
This means that minimal certificate validation is done. Which means, that the notBefore and notAfter are still checked. If this property is set to true full certification chain validation is done.
So there is no way to fully disable the validation of notBefore and notAfter dates as that defies the purpose of SSL.
When this property is set to false, X509Certificate.checkValidity method is called, that would validate the certificate start and expiry dates.
Either the cert needs to be corrected or you need to connect on non-SSL port.
I will ask the author of this article to explain the usage of this flag in full detail so that this confusion could be avoided further.
... View more
Health rules are based on one or more of an application's performance metrics and let you specify the parameters for what is consider “normal” for your application. When the health rule is violated, an event is raised that triggers a health policy. Health policies can also be triggered by other types of events (e.g. errors). In sum, health rules work on metrics, policies work on events, and the evaluation of health rules triggers events on which policies take action. For a diagram of how this functions, see Alert and Respond . Below are the most common reasons and steps to take if your health policies aren’t triggering.
Confirm that the health rule violation is being triggered
Confirm that the event that triggers the policy is occurring
What to check if the event exists
What to check if the event doesn't exist
1. Confirm that the health rule violation is being triggered
If your policies are triggered by health rule violation events and your health rules aren’t configured properly, violations may not be fired.
Review your health rule configuration - Ensure you followed the workflows outlined in Configure Health Rules when creating your health rule.
Check your Health Rule Evaluation configuration - This is the “Use the last <x> minutes of data when evaluating the health rule” field and the default is 30 minutes. This is the window of time in which the violation is detected, rather than a range over which the violation must persist. See the "Health Rule Evaluation Window" section in this document for additional information.
Check your Wait Time after Violation configuration - This value is used to avoid alert storms after a violation has occurred. When a violation is detected, the health rule is not re-evaluated until after this wait time has expired. If the erroneous conditions correct themselves before that window of time expires, you won’t receive any additional alerts. See the "Health Rule Wait Time After Violation" section in this document for details.
2. Confirm that the event that triggers the policy is occurring
Policies can be fired by health rule violation events or events that are automatically raised under certain conditions, such as errors and slow transactions. Events can also be manually registered or created programmatically using the REST API . For a list of events, see Events Reference . If you are on an older version of AppDynamics, refer to the events list for your specific version. If you do not see the expected event in the Controller UI, evaluate the following:
Slow transactions - Check the defined threshold and the baseline. Not allowing sufficient time for the baseline to be calculated or a lack of load on the application can affect the ability to define the baseline. If there is no baseline, we can’t monitor of the deviations from the baseline .
Errors - Check if the agent logs contain the expected error. If the error is found but the event didn’t trigger, contact AppDynamics Support.
Code Problems - Resource Pool Limit Reached is a predefined event that can trigger a policy. Policies for this event are not configured by default, so you would need to create one. The event is triggered by 80% usage of EJB-related resource pools (e.g. connection pools and thread pools), but that threshold can be configured using the jms-metric-threshold-percentage node property. Changing the value requires a restart of the agent JVM to apply this change.
Create discrete policies for health rule events - Rather than create one all-encompassing policy, create more specific ones so you can better understand the scope of the policy configuration. Before creating a policy, read: Configuring Policies
For more information, see: Policy Triggers
3. If the event exists, check the following:
Evaluated object - When the health rule is defined, you select the objects that you want monitored. If you create a policy based on the health rule, you can also select the objects that the policy applies to. If there is a mismatch between the health rule definition and the policy definition, the policy will not work as expected. If the event was not on the expected object, edit the health rule and/or policy configuration.
Check the expected policy action - You can assign the actions that are taken when a policy is triggered:
The user running the script has executable permissions
The dependent files or directory exist
The Machine Agent is up and running or reachable by the Controller
If the expected action is to send an email, check if the email server is working.
If the expected action is to run a remediation Machine Agent script, check the following:
If the expected action is to take a thread dump, check if the agent node is running.
If the expected action is to run a Custom Action, validate your configuration.
4. If the event doesn’t exist, check the following:
Evaluation time - Metrics that are being evaluated against a baseline need time and load to build up the baseline. If the timeframe or the load is insufficient, evaluation against the baseline can't occur.
Confirm metrics exist - When metrics are discovered, they are registered with the Controller. If something happens to the entities that identify the source of specific metrics (e.g., the application, tier, or node), the health rule evaluation will fail and there won’t be a rule violation to trigger the associated policy.
Symptoms (besides no event or "alert") include log entries in Controller logs similar to the following:
[#|SEVERE|glassfish3.1.2|com.appdynamics.RULES.PROCESSING|_ThreadID=75;_ThreadName=Thread-6;|An Error occured while evaluating Policy com.singularity.ee.controller.api.exceptions.ObjectNotFoundException: Metric not found: <metric_id> at com.singularity.ee.controller.beans.manage.policies.PolicyProcessorBean$SingleThreadedRuleProcessorBean.evaluateLeafCondition(PolicyProcessorBean.java:1895)
Run the following query on the Controller database: mysql> use controller; mysql> select id, name, application_id from metric where id in (<metric_IDs from controller log errors>); . The query should return empty set (no results) if the metrics actually don't exist. If a tier, node or specific metric used in a health rule has been deleted, the behavior may be uncertain. The metric must exist or the health rule can’t operate properly.
From the command line, change to the Controller's bin directory and use the appropriate script to log in to the Controller database:
For Windows: controller.bat login-db
For Linux: sh controller.sh login-db
If an incorrect metric was used, select the correct one and re-apply load on the application.
Last Updated: 3/4/19
... View more
In some cases, database corruption can result in java errors, deadlocks, sporadic Controller shutdowns, and Controller inaccessibility. Below is a list of errors that may indicate database corruption. If you encounter one of them or other issues that you think could indicate corruption, please contact AppDynamics Support so we can assist you.
InnoDB: cannot calculate statistics for table controller/<table_or_partition_name> InnoDB: because the .ibd file is missing. For help, please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html 180302 17:40:45 [ERROR] MySQL is trying to open a table handle but the .ibd file for table controller/<table_or_partition_name> does not exist. Have you deleted the .ibd file from the database directory under the MySQL datadir, or have you used DISCARD TABLESPACE? See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html how you can resolve the problem.
InnoDB: Error: page <page_number> log sequence number <log_sequence_number> InnoDB: is in the future! Current system log sequence number <log_sequence_number>. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: for more information.
[ERROR] mysqld.exe: Incorrect key file for table 'sql.MYI'; try to repair it [ERROR] Got an error from unknown thread, ..\..\..\mysqlcom-pro-5.5.40\storage\myisam\mi_write.c:<line_number> InnoDB: Error: tablespace id is <tablespace_id> in the data dictionary InnoDB: but in file <file_name>.ibd it is <tablespace_id>! InnoDB: Assertion failure in thread <thread_id> in file fil0fil.c line <line_number> InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery.
InnoDB: Assertion failure in thread <thread_id> in file mtr0log.c line <line_number> InnoDB: Failing assertion: DATA_ROLL_PTR_LEN == dict_index_get_nth_col(ind, DATA_ROLL_PTR - 1 + n_uniq)->len InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery.
InnoDB: Error: trying to access page number <page_number> in space <space_id>, InnoDB: space name <ibd_file_name>, InnoDB: which is outside the tablespace bounds. InnoDB: Byte offset <offset>, len <length>, i/o type <type_number>. InnoDB: If you get this error at mysqld startup, please check that InnoDB: your my.cnf matches the ibdata files that you have in the InnoDB: MySQL server.
Prepare to contact Support
It is important that you do not attempt to fix database corruption without hands-on assistance from our Support team. If you notice one of the errors above, please contact Support and prepare the following information:
Please share all the platform log files: Platform Log Files
If the Controller version is older than 4.4, provide Controller log files: Controller Log Files
If you have an HA installation, provide HA log files from both primary and secondary servers: HA Log Files
Is the Controller server on a physical machine or a virtual machine? If on a virtual machine, is the RAM fully reserved? If so, please share the screenshot of memory reservation: Fully Reserved RAM
What is the Controller profile and does the hardware meet the system requirements: System Requirements
If your Controller is on Linux, provide the /var/log/messages file for the date of the database crash and the event viewer logs for the MySQL service.
Information about series of events like a Controller migration, Controller restart or a server reboot that happened before the database corruption.
... View more
To make sure you have all of the information you need to understand license consumption, rules, errors, and usage, we've indexed several articles below that will guide you through how to resolve common issues and answer frequently asked questions.
These articles will provide you with a basic understanding of how licenses are consumed and how to best use license rules to strategically allocate units:
How does AppDynamics license consumption work?
How is Java Agent license consumption calculated?
How is .NET Agent license consumption calculated?
What are license rules and how do they work?
What happens if I delete a license rule?
What are some best practices for license management?
If you encounter any notifications or errors related to licensing, please review the troubleshooting resources linked below:
Why am I seeing license limit notifications?
How do I resolve "Agent License Request Denied" license limit notifications?
Viewing License Consumption and Usage History
The resources below will guide you through how to view license consumption from different perspectives to help you better understand your usage:
How do I view license usage and history on the Subscription page?
How do I check my license usage at the account level?
How do I find license usage by applications for each controller?
... View more
The certificates are imported to the trusstore. So the test run for HTTP request template should now succeed.
But when you actually call the request template as part of action, you would still see the same exception because Controller appserver is not restarted yet. OPS team is working on coordinating a downtime.
... View more
Following article explains how you configure machine agent for SSL and how to extract the certs.
If your configuration is to make machine agent connect through a reverse proxy which in turn connects to the Contoller, then the SSL endpoint would be your proxy and in that case you would need to import the certs that are configured on your proxy into the machine agent truststore (cacerts.jks).
... View more