Cisco AppDynamics Community

Vikash.Vardhan · ‎08-02-2024

I have installed AppDynamics locally with a private Synthetic server + PSA Agent in the same host and Controller, EUM in separate hosts.

Now my issue is that whenever I try to schedule a job in AppDynamics in the User Experience dashboard the agent hits the provided URL with some response. It is displayed on the session tab but only one hit is being registered and showed despite scheduling the job to run in every minute.

We have looked into the scheduler-shepherd logs where we're getting below logs:

INFO 2024-08-01 10:13:24,121 SyntheticBackgroundScheduler_Worker-16 JobRunShell

Job background-job.com.appdynamics.synthetic.scheduler.core.tasks.ClusterWatcher threw a JobExecutionException:

! java.io.IOException: error=2, No such file or directory

! at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)

! at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)

! at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)

! at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)

! ... 12 common frames omitted

! Causing: java.io.IOException: Cannot run program "jps" (in directory "."): error=2, No such file or directory

! at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)

! at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)

! at java.base/java.lang.Runtime.exec(Runtime.java:594)

! at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)

! at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:254)

! at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:319)

! at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)

! at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)

! at com.appdynamics.synthetic.cluster.local.PSCluster.findAllInstanceIds(PSCluster.java:58)

! ... 4 common frames omitted

! Causing: com.appdynamics.synthetic.cluster.Cluster$ClusterException: Failed to query active schedulers.

! at com.appdynamics.synthetic.cluster.local.PSCluster.findAllInstanceIds(PSCluster.java:67)

! at com.appdynamics.synthetic.cluster.local.PSCluster.findHealthyInstanceIds(PSCluster.java:46)

! at com.appdynamics.synthetic.scheduler.core.tasks.ClusterWatcher.execute(ClusterWatcher.java:49)

! ... 2 common frames omitted

! Causing: org.quartz.JobExecutionException: Exception while attempting to find healthy instances

! at com.appdynamics.synthetic.scheduler.core.tasks.ClusterWatcher.execute(ClusterWatcher.java:52)

! at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

! at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

WARN 2024-08-01 10:13:27,484 dw-583 - PUT /v1/schedule/02a54278-0b5f-4a0e-95f1-9299cd9b4dbc spi idx-requestId=96a2de91531b226a, idx-account=*********

javax.persistence.spi::No valid providers found.

INFO 2024-08-01 10:13:27,502 dw-583 - PUT /v1/schedule/02a54278-0b5f-4a0e-95f1-9299cd9b4dbc LicenseHelper idx-requestId=96a2de91531b226a, idx-scheduleId=02a54278-0b5f-4a0e-95f1-9299cd9b4dbc, idx-appKey=*********, idx-account=*********

Job with id 02a54278-0b5f-4a0e-95f1-9299cd9b4dbc and description http://172.16.21.70 previously used 1 units of license

INFO 2024-08-01 10:13:27,502 dw-583 - PUT /v1/schedule/02a54278-0b5f-4a0e-95f1-9299cd9b4dbc LicenseHelper idx-requestId=96a2de91531b226a, idx-scheduleId=02a54278-0b5f-4a0e-95f1-9299cd9b4dbc, idx-appKey=*********, idx-account=*********

Job with id 02a54278-0b5f-4a0e-95f1-9299cd9b4dbc and description http://172.16.21.70 now has 1 pages and 1 locations and requires 1 units of license

INFO 2024-08-01 10:13:27,505 hystrix-SchedulerGroup-13 SynthJobRunner

Will run 1 measurement(s) for job=Schedule{account='*********', appKey='*********', id='02a54278-0b5f-4a0e-95f1-9299cd9b4dbc', description='http://172.16.21.70'}

INFO 2024-08-01 10:13:27,505 hystrix-SchedulerGroup-13 SynthJobRunner

Requesting measurement for job=Schedule{account='*********', appKey='*********', id='02a54278-0b5f-4a0e-95f1-9299cd9b4dbc', description='http://172.16.21.70'}, location=NOD:*********, browser=IE11

INFO 2024-08-01 10:13:27,512 hystrix-SchedulerGroup-13 SynthJobRunner

Updating state for schedule

WARN 2024-08-01 10:13:33,178 okhttp-eventsource-stream-[]-0 DataSource

Error in stream connection (will retry): java.net.UnknownHostException: stream.launchdarkly.com: Name or service not known

INFO 2024-08-01 10:13:33,178 okhttp-eventsource-stream-[]-0 DataSource

Waiting 22193 milliseconds before reconnecting...

WARN 2024-08-01 10:13:33,178 okhttp-eventsource-events-[]-0 DataSource

Encountered EventSource error: java.net.UnknownHostException: stream.launchdarkly.com: Name or service not known

WARN 2024-08-01 10:13:40,054 hz.epic_ramanujan.cached.thread-9 MulticastService

[172.18.0.1]:5701 [dev] [5.3.2] Sending multicast datagram failed. Exception message saying the operation is not permitted usually means the underlying OS is not able to send packets at a given pace. It can be caused by starting several hazelcast members in parallel when the members send their join message nearly at the same time.

! java.net.SocketException: Message too long

! at java.base/sun.nio.ch.DatagramChannelImpl.send0(Native Method)

! at java.base/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:901)

! at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:863)

! at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:821)

! at java.base/sun.nio.ch.DatagramChannelImpl.blockingSend(DatagramChannelImpl.java:853)

! at java.base/sun.nio.ch.DatagramSocketAdaptor.send(DatagramSocketAdaptor.java:218)

! at java.base/java.net.DatagramSocket.send(DatagramSocket.java:664)

! at com.hazelcast.internal.cluster.impl.MulticastService.send(MulticastService.java:309)

! at com.hazelcast.internal.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:113)

! at com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75)

! at com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42)

! at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77)

! at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)

! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)

! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

! at java.base/java.lang.Thread.run(Thread.java:840)

! at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)

! at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)

INFO 2024-08-01 10:13:40,409 Account Credential Cache-1 AccountCredentialCache

Refreshing account credential cache