custom action error in server.log

[Sundarapandian.Ponniah]

Hi,

We are using custom action to route health rule events to a another home grown tool that uses a filter to route the events to a ticketing tool.

The custom action script has been tested in QA and it does fine. However in prod the custom action fails and could see some erros in controller server.log, at end of the actual event

SAMPLE:

          [#|2017-07-10T09:12:00.211-0500|SEVERE|glassfish3.1.2|com.appdynamics.actionaas.executor.CustomActionExecutor|_ThreadID=6665;_ThreadName=Thread-6;|Process [[/d/d1/appd/controller/appserver/glassfish/../.././custom/actions/watchtower/send_to_wt.sh, "BPM-PS2-PR-PADT", "12", "Mon Jul 10 09:11:55 GMT-05:00 2017", "1", "ERROR", "send_to_wt.sh", "High Database/Remote Services Average Response Time", "2697", "10", "BACKEND", "NP194001-IBM DB2 DB-NY20DB15.NEW-YORK.MS.COM-SQL10015", "493", "1", "APPLICATION", "BPM-PS2-PR-PADT", "12", "1", "APPLICATION", "BPM-PS2-PR-PADT", "12", "condition 1", "15901", "EQUALS", "BASELINE_STANDARD_DEVIATION", "true", "3.0", "3.0", "High Database/Remote Services Average Response Time triggered at Mon Jul 10 09:11:55 GMT-05:00 2017. This policy was violated because the following conditions were met for the NP194001-IBM DB2 DB-NY20DB15.NEW-YORK.MS.COM-SQL10015 Backend for the last 10 minute(s):   For Evaluation Entity: BPM-PS2-PR-PADT Application - condition 1 is within the 'default' baseline by 3.0 standard deviation(s). Observed value = 3.0", "326521", "http://iapp687.howard.ms.com:8090/controller/#location=APP_INCIDENT_DETAIL&incident=", "POLICY_CONTINUES_CRITICAL", "ACCOUNT", "customer1_493f8c89-5444-40fd-96de-bf52e0056277"]] error msg:             'status' => 'REACHED_LIMIT',|#]

Similarly, can see few other errors

.....error msg:             'errorMsg' => 'Rejecting.  Not enough room on the backlog',|#]

.....error msg:             'quark' => {|#]

.....error msg:                          'payload' => {},|#]

.....error msg: $VAR1 = [|#]

Are these any know issues with custom actions or are these AppDynamics related errors?

The controller version we have is 4.2.11.0

Please advise

Thanks,

Sundar

[asha.ramegowda]

Hi Sundar,

One of the possible causes could be the lack of privileges to execute this script in the target environment.

The stack trace provided here is not sufficient enough to analyze the issue and identify the root cause. Please provide us with the server.log so that we can look at this issue within the context.

Thanks and regards,

Asha.

[Sundarapandian.Ponniah]

This turned out to be soem issue with the resource collector of the home grown tool, restarting the collector resolved the issue. Thanks