Oh, I completely understand why the rule is firing. It's correct that those rules are firing.
My complaint is that I can't put a condition on the health rule so that it doesn't fire when baseline is 0 or when Std Dev is 0. They cause false alarms for us, every single time. And they're very sensitive, since it only takes 1 thing above baseline for it to fire.
Ideally, I'd love to be able to put a condition that compares the value of the baseline of a metric, and a different one for comparing the value of the baseline std. deviation of the metric.
Short of that, the most viable workaround I've found is to require a minimum level of traffic (calls/minute), a minimum number of Std Deviations above baseline, and a minimum percentage of the baseline above the baseline. But even that still fails when there's sufficient traffic, and baseline and std dev are both 0. One single thing will be both infinite std deviations above baseline and infinite percentage above baseline. And then a bunch of us get useless alerts waking us up in the middle of the night.
... View more
I'm going crazy with this, even with several weeks of data in our controller already. We're getting a lot of health rules that fire due to edge cases when baseline is 0 or when standard deviation is 0. The math is awful:
When baseline is 0 (or even very close to it), even a lax rule like 1000% increase over the baseline will fire with even a very small value. We can set rules that check for minimum volumes of calls/errors/slow calls in some cases, or minimum values of response time in some cases, but those don't completely solve the issue.
When baseline standard deviation is 0 (or very low), then any rule, even something huge like 10 standard deviations above baseline, is destined to fail as soon as anything higher than the baseline comes along. Again, sometimes there are some other conditions that can be added, but they don't completely solve the issue, because in some types of rules, those other options are extremely limited, like setting a hard minimum on the value in question.
Are there other solutions I'm missing?
I've seen posts here in the community going back to 2015 that ask about this, with lackluster responses from staff. Is there any chance of getting this seemingly simple feature onto the roadmap?
... View more