Issues deploying Events Service node (Linux on-prem install)

[Brian.Malin]

I'm trying to deploy an Events Service node on a Linux machine to move that functionality off the controller.  I have gotten passwordless SSH connectivity configured and am up to step 5 on https://docs.appdynamics.com/display/PRO43/Install+the+Events+Service+on+Linux

But everytime I try to add the platform I get the following:

[root@<controller> platform]# platform-admin/bin/platform-admin.sh add-hosts --hosts <eventsservice.fqdn> --credential appd-analytics
Adding new hosts to platform admin and making configuration updates.
Overall progress(6/9) : Set orcha modules root log location : STATUS_OK
Overall progress(7/9) : Copying JRE to the remote host : STATUS_APP_FATAL_ERROR
Platform Admin host expansion failed:
Failure occurred at stage: Bootstrap hosts
Error message:
Task failed: Copying JRE to the remote host
on host: <eventsservice.fqdn>
with message: Failed to upload file: java.net.ConnectException: Connection refused (Connection refused)

Any ideas?

[Mohammed.Rayan]

Hi bmalin,

The message is clear that you can't connect over SSH to the remote host as it says connection refused.

Task failed: Copying JRE to the remote host
on host: <eventsservice.fqdn>
with message: Failed to upload file: java.net.ConnectException: Connection refused (Connection refused)

When you say that you have configured the SSH passwordless configs, Have you manually tested that you can able to connect to the remote host using the ssh command manually.

Can you please verify it using the below command

ssh <eventsservice.fqdn>

 Regards,

Mohammed Rayan

[Brian.Malin]

Yes, I confirmed that works.

[Brian.Malin]

If it helps, SSH is running on an off-port per our corporate security policy (P 2222).

Logs show:

INFO  [2017-10-09 09:25:42,155] class com.appdynamics.platformadmin.core.job.JobProcessor: Running job add_hosts_job with description Add hosts eventsservice.host to Platform Admin.
INFO  [2017-10-09 09:25:42,156] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 1, stage name Check new hostnames, stage id: 134
INFO  [2017-10-09 09:25:42,158] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 2, stage name Add hosts to Platform Admin, stage id: 135
INFO  [2017-10-09 09:25:42,161] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 3, stage name Initialize Orcha Cluster and save Cluster configuration, stage id: 136
INFO  [2017-10-09 09:25:42,179] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 4, stage name Add hosts to Orcha Cluster, stage id: 137
INFO  [2017-10-09 09:25:42,183] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 5, stage name Unpack orcha-modules artifact, stage id: 138
INFO  [2017-10-09 09:25:42,241] com.appdynamics.orcha.core.OrchaRunner: Running playbook /var/opt/appdyn/platform/platform-admin/playbooks/pa_orcha/stage-artifact.orcha with variables file /var/opt/appdyn/platform/platform-admin/playbooks/pa_orcha/pa.groovy
INFO  [2017-10-09 09:25:42,241] com.appdynamics.orcha.core.executor.DefaultPlaybookExecutor: Executing : Unarchive binary in staging
INFO  [2017-10-09 09:25:42,259] com.appdynamics.orcha.modules.modules.UnarchiveExec: Expanding: /var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha-modules/2.0/orcha-modules.zip into /var/opt/appdyn/platform/platform-admin/conf/../../staging/unpack1069811259495274419
INFO  [2017-10-09 09:25:42,478] com.appdynamics.orcha.core.executor.TaskExecutor: Task [Unarchive binary in staging] completed executing in [228] ms.
INFO  [2017-10-09 09:25:42,483] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 6, stage name Configure orcha logging, stage id: 139
INFO  [2017-10-09 09:25:42,524] com.appdynamics.orcha.core.OrchaRunner: Running playbook /var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha/4.3.2.1/playbooks/configure-orcha-logs.orcha with variables file /var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha/4.3.2.1/playbooks/bootstrap.groovy
INFO  [2017-10-09 09:25:42,526] com.appdynamics.orcha.core.executor.DefaultPlaybookExecutor: Executing : Set orcha modules root log location
INFO  [2017-10-09 09:25:42,557] com.appdynamics.orcha.modules.modules.XmlUpdateExec: Successfully write to xml
INFO  [2017-10-09 09:25:42,558] com.appdynamics.orcha.core.executor.TaskExecutor: Task [Set orcha modules root log location] completed executing in [23] ms.
INFO  [2017-10-09 09:25:42,571] class com.appdynamics.platformadmin.core.job.JobProcessor: Sequence number 7, stage name Bootstrap hosts, stage id: 140
INFO  [2017-10-09 09:25:42,623] com.appdynamics.orcha.core.OrchaRunner: Running playbook /var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha/4.3.2.1/playbooks/bootstrap.orcha with variables file /var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha/4.3.2.1/playbooks/bootstrap.groovy
INFO  [2017-10-09 09:25:42,624] com.appdynamics.orcha.core.executor.DefaultPlaybookExecutor: Executing : Copying JRE to the remote host
INFO  [2017-10-09 09:25:42,648] com.appdynamics.orcha.utils.SshTransfer$ChannelWrapper: Connecting to remote host eventsservice.host...
INFO  [2017-10-09 09:25:42,653] com.appdynamics.orcha.core.executor.TaskExecutor: Task [Copying JRE to the remote host] completed executing in [15] ms.
ERROR [2017-10-09 09:25:42,654] com.appdynamics.orcha.core.executor.DefaultPlaybookExecutor: Error executing task: TaskSpec[name=Copying JRE to the remote host,runLocal=false,localModule=true,ignoreErrors=false,registerOutputAs=<null>,operationSpecList=[UploadSpec[filePermissions=0755,src=/var/opt/appdyn/platform/platform-admin/jre,dest=/var/opt/appdyn/platform/jre/1.8.0_121,overwrite=true,fileType=DIRECTORY,hostConnectionInfo=HostConnectionInfo[host=eventsservice.host,orchaCredential=com.appdynamics.orcha.shared.OrchaCredential@6f7d1912],requiresConnection=true,baseDir=/var/opt/appdyn/platform/platform-admin/conf/../archives/platform-configuration/../orcha/4.3.2.1/playbooks,name=upload,isLocalOperation=true,assertionSpecList=[]]],timeout=0,async=false,longRunning=false,updateThreshold=30,uid=<null>,deleteDir=true]
ERROR [2017-10-09 09:25:42,663] com.appdynamics.orcha.core.OrchaRunner: Error executing:
ERROR [2017-10-09 09:25:42,665] class com.appdynamics.platformadmin.core.job.JobProcessor: Stage Bootstrap hosts failed Task failed: Copying JRE to the remote host
on host: eventsservice.host
with message: Failed to upload file: java.net.ConnectException: Connection refused (Connection refused)
ERROR [2017-10-09 09:25:42,670] class com.appdynamics.platformadmin.core.job.JobProcessor: Cleanup Stage of job add_hosts_job failed
! java.lang.NullPointerException: null
! at com.appdynamics.platformadmin.orcha.manager.PAOrchaServiceRegistryImplV1$5.applyOperation(PAOrchaServiceRegistryImplV1.java:279)
! at com.appdynamics.platformadmin.core.tasks.ClusterOperationStage.buildContextAndApply(ClusterOperationStage.java:101)
! at com.appdynamics.platformadmin.core.tasks.ClusterOperationStage.execute(ClusterOperationStage.java:76)
! at com.appdynamics.platformadmin.core.job.JobProcessor.run(JobProcessor.java:168)
! at com.appdynamics.platformadmin.core.job.JobProcessor.run(JobProcessor.java:59)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run(FutureTask.java:266)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:745)

[Mohammed.Rayan]

Hi bmalin,

Thanks for this  additional info and yeah,we use the default port number 22 for SSH and so this can be the reason why its failing.

Regards,

Mohammed Rayan

[Brian.Malin]

Thanks - I reconfigured SSHD to run listen on 22 and was able to deploy the Events Service to the other host.

But after reboot when I go to check the health I get:

[root@controller.host platform-admin]# bin/platform-admin.sh show-events-service-health
Events Service cluster is not healthy.
Failure occurred at stage: es_cluster_health_stage
Error message:
Error reaching cluster

Ideas?

[Mohammed.Rayan]

Good to know that the events-service is installed.

May be the events-service is not running or its cluster state is in RED

is it possible for you to share the logs of the cluster and also you can run the below curl command and verify the cluster status as well

curl http://localhost:9081/healthcheck?pretty=true

Regards,

Mohammed Rayan