I found some documentation.
That states this:
The lightweight Standalone Machine Agent consumes minimal resources of your computer.
AppDynamics recommends the following additional Heap and PermGen space to accommodate the agent:
Maximum heap size (-Xmx): 100 MB
Maximum PermGen heap size (-XX:MaxPermSize): 20 MB
Looks similar to what you tried, but perhaps it may help.
... View more