The embedded machine agent which comes with .net agent periodically collects machine snapshots.
When you recieve alerts for high memory usage, check the time frame and machine it is orginating from and go to Machine Snapshots tab on application dashboard. Filter by machine name and time frame and you would get snapshots with high memory usage.
Review few of these snapshots to identify which of the processes taking high memory. After identifying which process is taking lots of memory, indentify if it is an instrumented .net process or any other process running on the server taking high memory.
For .net based instrumented processes -
Check if the process is consistently taking a high memory or if it is increasing with time or if periodically it is going down ? Based on above details you can pinpoint if there is any memory leak issue or not ?