The Events Service cluster may not function correctly and its health state will be listed as red if there are unwanted indices either in the unassigned or initializing state.
Example:
In a browser or terminal, run the following:
curl http://localhost:9081/healthcheck?pretty=true
The output of the healthcheck command will show cluster state [RED].
"events-service-api-store / elasticsearch-singlenode-module" : {
"healthy" : true,
"message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [48], relocating shards: [0], initializing shards: [0], unassigned shards: [5], timed out: [false]"
},
These unwanted indices are created as a result of the Nessus Vulnerability Scanner that gets run on the cluster port (default 9200).
Diagnosis:
Run the following command on the Events Service machine or in a browser to verify if there are unwanted indices.
http://<events_service_machine>:9200/_cat/shards?v [in Browser]
OR
curl http://<events_service_machine>:9200/_cat/shards?v [in the terminal of the Events Service machine]
Output example:
index shard prirep state docs store ip node
vrvsgs1.html 4 p STARTED 0 114b 192.x.y.z node-xyzxyzxyz
vrvsgs1.html 4 r UNASSIGNED 0 168b 192.x.y.z node-xyzxyzxyz
vrvsgs1.html 0 r UNASSIGNED 0 791b 192.x.y.z node-xyzxyzxyz
vrvsgs1.html 0 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz
vrvsgs1.html 3 r STARTED 0 105b 192.x.y.z node-xyzxyzxyz
gvanws1.html 4 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz
gvanws1.html 4 r STARTED 0 128b 192.x.y.z node-xyzxyzxyz
gvanws1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
gvanws1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
gvanws1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
oapssw1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
oapssw1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
oapssw1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
nrawos1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
nrawos1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
nrawos1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
nrawos1.html 0 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
spipe 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
spipe 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
spipe 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
spipe 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
psaoon1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
psaoon1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
psaoon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
psaoon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
arenon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
arenon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
arenon1.html 3 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
sprreo1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
sprreo1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
sprreo1.html 2 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz
vepper1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
vepper1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
vepper1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
perl 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
perl 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
perl 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
gaowsa1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz
greprs1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
greprs1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
enoson1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
enoson1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
enoson1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
vpnwwg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
vpnwwg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
arenon1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
soaeap1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
soaeap1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
pwvwgg1.html 1 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
pwvwgg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz
pwvwgg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz
enawae1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
enawae1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz
sawmill6cl.exe 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz
sawmill6cl.exe 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
rogwsw1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
Solution:
Delete these unwanted indices using the following commands:
- Delete all unwanted indices using the following curl command in the machine terminal:
curl -XDELETE http://<events-service_hostname>:9200/<index_name>- Example:
curl -XDELETE http://localhost:9200/rogwsw1.html
- Verify if unwanted indices have been removed using the following commands:
curl http://<events_service_machine>:9200/_cat/shards?v curl http://<events_service_machine>:9200/_cat/indices?v
- Work with a network or security team to configure the Nessus Vulnerability Scanner toexclude scanning the Events Service cluster port (9200-9300).
