Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
05-10-2022 12:23 AM
Hi Everyone,
We have Ping Directory application (LDAP) running on the Linux server. We have java appAgent and machine agent installed and configured on the Linux server. The issue is that everyday 1.05 AM my appAgent restarting and that JVM restart time is showing on the AppDynamic dashboard instead of the Ping Directory JVM status. I have spoken to internal AppD team and raised a support ticket but not satisfied with their responses. I understand their argument that javaagent will not start standalone as it is part of Ping Directory JVM. I'm trying to chase the mystery what is causing the javaagent restart at exactly 1.05 AM everyday. Appreciate any help in troubleshooting this issue.
Thanks,
Anand Gulla
05-10-2022 05:48 PM
Hi @Anand.Gulla ,
I see a disconnect, have you checked the support update on the root cause of the April 29 update ticket? I manage support in India and I confirm from supoort ticket#316533? Why have not the user come back in ticket? The root cause with evidence is already shown but i am listing here as you are reaching in community, I suggest you reach any application dev team, You can see there are two JVM PID (javaagent is a thread (premain program) part of app JVM PID and it will not create JVM PID). I have "NOT" shared theory but shared evidence it is the customer starting second JVM PID here.
If you see in same host "dld201013230254
" at 1:05 AM two app JVM PID started which is using same -Dappdynamics.agent.nodeName= value and you can see the evidence of difference in application runtime args difference, this args are from app not javaagent. (example app arg -Dcom.unboundid.directory.server.scriptName).
Next steps: I have offered web call in tickets/316533 see if you can join we can discuss over your concerns, Also, I will leave it to others to also comment in the community to comment on my analysis.
dld201013230254.obm.nix.srv.westpac.com.au"
AD Agent init] 28 Apr. 2022 01:05:35,054 INFO AgentKernel - JVM Runtime Name: 57279@dld201013230254.obm.nix.srv.westpac.com.au
[AD Agent init] 28 Apr. 2022 01:05:35,045 INFO AgentKernel - JVM Args : -Xmx256m | -Xms128m | -javaagent:/opt/appdynamics/appagent/java/javaagent.jar | -Dappdynamics.agent.runtime.dir=/app/pingdirectory/racdirectory/logs/ | -Dappdynamics.agent.tierName=PingDirectory | -Dappdynamics.force.default.ssl.certificate.validation=true | --add-opens=java.base/java.io=ALL-UNNAMED | --add-opens=java.base/java.lang=ALL-UNNAMED | --add-opens=java.base/java.lang.invoke=ALL-UNNAMED | --add-opens=java.base/java.lang.reflect=ALL-UNNAMED | --add-opens=java.base/java.security=ALL-UNNAMED | --add-opens=java.base/java.text=ALL-UNNAMED | --add-opens=java.base/java.util=ALL-UNNAMED | --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED | --add-opens=java.desktop/java.awt.font=ALL-UNNAMED | -Dcom.unboundid.directory.server.scriptName=export-ldif |
[AD Agent init] 28 Apr. 2022 01:05:39,642 INFO ConfigurationChannel - Auto agent registration attempted: Application Name [PING-A00B2B-DEV] Component Name [PingDirectory] Node Name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
[AD Agent init] 28 Apr. 2022 01:05:39,642 INFO ConfigurationChannel - Registration information received Node ID[19447446] Component ID[226630] Application ID [61972]
[AD Agent init] 28 Apr. 2022 01:05:45,845 INFO AgentKernel - JVM Runtime Name: 59819@dld201013230254.obm.nix.srv.westpac.com.au
[AD Agent init] 28 Apr. 2022 01:05:45,827 INFO AgentKernel - JVM Args : -Xmx2g | -Xms2g | -XX:+UseParallelGC | -XX:NewRatio=8 | -XX:+HeapDumpOnOutOfMemoryError | -javaagent:/opt/appdynamics/appagent/java/javaagent.jar | -Dappdynamics.agent.runtime.dir=/app/pingdirectory/racdirectory/logs/ | -Dappdynamics.agent.tierName=PingDirectory | -Dappdynamics.force.default.ssl.certificate.validation=true | --add-opens=java.base/java.io=ALL-UNNAMED | --add-opens=java.base/java.lang=ALL-UNNAMED | --add-opens=java.base/java.lang.invoke=ALL-UNNAMED | --add-opens=java.base/java.lang.reflect=ALL-UNNAMED | --add-opens=java.base/java.security=ALL-UNNAMED | --add-opens=java.base/java.text=ALL-UNNAMED | --add-opens=java.base/java.util=ALL-UNNAMED | --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED | --add-opens=java.desktop/java.awt.font=ALL-UNNAMED | -Xlog:gc*:file=/app/pingdirectory/racdirectory/current/logs/jvm/gc.log:time,uptime,level,tags:filecount=3,filesize=100m | -Dcom.unboundid.directory.server.scriptName=export-ldif |
[AD Agent init] 28 Apr. 2022 01:05:50,251 INFO ConfigurationChannel - Auto agent registration attempted: Application Name [PING-A00B2B-DEV] Component Name [PingDirectory] Node Name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
[AD Agent init] 28 Apr. 2022 01:05:50,251 INFO ConfigurationChannel - Registration information received Node ID[19447446] Component ID[226630] Application ID [61972]
-Xlog:gc*:file and XX:NewRatio
Conclusion: The agent getting same node id because we are using manual node as per recent logs shared still. It is clear from OS and application perspective with above logs and PS output user does initiate two PIDs with same agent config parameters (-javaagent and -Dappdynamics) args with 10 seconds period gap and hence and agent shows "JVM startup time" discrepancy as same node name used for two different PIDs initiated by OS which is "Invalid config" as per doc specs.
[AD Agent init] 28 Apr. 2022 01:05:45,299 INFO XMLConfigManager - Full Agent Registration Info Resolver using node name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
Optional suggestion: As the issue is reproduced and confirms to be user initiating pingdir+ two PIDs 57279 and 59819
(PS output is the evidence) at same minute the only suggestion we have is to not use manual node name, if your team cannot have dedicated JVM startup args two both JVM process and use reuse node name arg approach that is below:
Step-1: Remove node-name value in agent controller-info.xml at path below and save xml:
/opt/appdynamics/appagent/java/conf/controller-info.xml
/opt/appdynamics/appagent/java/ver21.11.2.33305/conf/controller-info.xml
Step-2: Now in same startup file at host "dld201013230254.obm.nix.srv.westpac.com.au]" where you are passing arg -javaagent:/opt/appdynamics/appagent/java/javaagent.jar pass additionally below and that should get node name different for both the JVMs getting started with same node name earlier.
-Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.agent.reuse.nodeName.prefix=CloudActivator_ -Dappdynamics.cron.vm=true
Regards,
Arun
Support Manager
05-10-2022 06:49 PM
Hi Arun,
I thought of posting here in community to see if anyone have seen the behaviour with the java appagent specific to Ping Directory or any other LDAP components. I'm not convinced with the java appagent logs because I showed you that the Ping Directory process being running for more than a week but not sure what initiating the java appagent to restart its process.
Anyway I updated the configuration files and Ping Directory process file based on your recommendations. I will catchup with you tomorrow at 3 PM AEST.
Thanks,
Anand Gulla
05-10-2022 06:57 PM - edited 05-10-2022 06:59 PM
Hi Anand,
Javaagent does not have a timer to start exactly at 1:05 AM, neither javaagent will start PID, it is the app PID. I have given you evidence with the difference in JVM PID args example below arg is one of the diff arg from app PID and still we are claiming it as javaagent PID. Let's discuss this on web call tomorrow. I am sure if you try to reuse the node name approach after removing nodeName in agent or -D arg, this issue proves otherwise. So I request to try suggestions (reuse node name option) before web call or preferably before next 1:05 AM period today on target host, so call itself will not be needed.
-Xlog:gc*:file and XX:NewRatio
Just to inform, no other customer ever have faced an issue at exactly 1:05 AM IST and also the PingDirectory app team confirmed too they do not face any issues with any other environment with javaagent, on top of that I shared JVM PID getting started from the app with the different app -D args (i am not referring to agent args here) with 10 secs gap at 1:05 am, all this proves to how PingDirectory app processes start.
We want to address your concerns, so let us continue discussion in help ticket or in call tomorrow as i requested please try suggestions it will clear all your doubts.
Regards,
Arun
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form