cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Auto instrument not working

sattha.puangput
Creator

Hi All,

 

Currently, I try to implement cluster-operator and cluster-agent on K8s. the cluster metric shows under the "Server/Cluster" menu correctly. However, I would like to config auto-instrument for Java application but no hope follow the validate guide here is still not much helpful...

 

https://docs.appdynamics.com/21.5/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-a...

 

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "TEST-CLUSTER"
  controllerUrl: "https://xxx.appdynamics.com"
  account: "xxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my.own.repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: "xxx-dev|xxx-qa"
  appNameStrategy: label
  defaultAppName: not-specific
  defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "xxx-dev"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "xxx-qa"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

 

But I got this....

 

[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-qa matches Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-qa  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile] is false
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-dev matches Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-dev  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries] is false

 

Please advice

8 REPLIES 8

Mario.Morelli
Architect

Hi Sattha

 

You have set the appNameStrategy : label

And in the instrumentation rules, you state that it should use the below label of the Java pods to name the Application.

 

Do you have a label in your java app's deployment spec that is named "app"? As it will look for that label to set the Application Name. If it's empty it wont register. Your logs show that the instrumentation rule is matching the pod so the rest seem to be correct.

appNameLabel: app

 

I would suggest to set the following, to just validate testing, before switching back to using labels to set the AppD Application Name

 

 

appNameStrategy: manual

Change 

appNameLabel: app to appName: XXX_DEV

                                             appName: XXX_QA

 

Then redeploy and let us know

 

Ciao

 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Learn more about me in the Community Member Spotlight Q & A

Thank you for your advice, I update configure to this one now... but still got the same log output

 

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxx.saas.appdynamics.com"
  account: "xxx"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://xx.xx.xx.xx:80"
  proxyUser: "xxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  instrumentContainer: first
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

 

Log

[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false


[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-dev matches Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-dev  map[] SEACMA-TH-DEV  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile] is false

 

What I try...

  1. check K8s deployment config to make sure that config is applied to the current "k8s-cluster-agent" pod
  2. delete and recreate application deployment but no init-container config apply (it should not I think since Instrument still false

 

Current dashboard screenshot...

clusterclusterapplicationapplication

Thanks Sattha

 

Could you remove all instances of the following from the cluster agent yaml

instrumentContainer: first

and add the following to each of the 2 instrumentation rules

containerMatchString: .*

 

The redeploy, and check if it instruments the pods now



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Learn more about me in the Community Member Spotlight Q & A

I try to add the following config but it does not work...

 

instrumentContainer: select
containerMatchString: .*

 

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  ### Uncomment the following two lines if you need pull secrets
  #imagePullSecrets:
  #  name: "<your-docker-pull-secret-name>"
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  instrumentContainer: select
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

  Still same error

[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java select .*  JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:pvnexusho001th.dmz.th-tesco.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false

 

What I try

  • Update config according to your suggestion
  • Delete cluster-operator  K8S deployment
  • Delete cluster-agent K8S deployment
  • recreate both cluster-operator and cluster-agent
  • check deployment configuration change to what we need
  • delete application deployment and recreate it --- not thing happen no init-container

Hi Sattha

 

Can you try the below YAML Config

 

 

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitorRegex: appdynamics|cma-dev|cma-qa
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Learn more about me in the Community Member Spotlight Q & A

I try your .yaml file but when creating `cluster-agent.yaml` there is the following error

error: error validating "appdynamics/cluster-agent-test.yaml": error validating data: ValidationError(Clusteragent.spec): unknown field "nsToMonitorRegex" in com.appdynamics.v1alpha1.Clusteragent.spec; if you choose to ignore these errors, turn validation off with --validate=false

 

Thanks Sattha

 

Can you just confirm a couple of things

 

1. Which version of K8S are you running?

2. What version of the Cluster Agent Operator are you running? If running version 1.14 or less you should have deployed the Operator with the below YAML File

cluster-agent-operator-1.14-or-less.yaml

 

You can try the below YAML and advise

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrument:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.

Learn more about me in the Community Member Spotlight Q & A

Pooja.Patel
Explorer

did it work @sattha.puangput ? I am also facing the same issue with my containerized app.