More monitoring is not necessarily better monitoring
What gets measured, gets fixed.
Are we measuring the meaningful stuff, or incidentals
Meaningful metrics: quality, quantity and cost
Incidentals – swap, disk space, anything targetted at a specific “problem” condition
There should be (at least) 3 levels of “severity” with appropriate corresponding actions
Major site problem – VIP not working, broken images, error messages, can’t place orders, etc.
Minor site problem – Slow response, error in rare cases, monitoring broken
Not customer-facing – One server or a small number have a problem but VIP is OK
Minor – Node is not down but some resources are out-of-bounds. (swap, disk, cpu etc)
Forensic, correlating – Should not notify anyone, but can be used for additional information
“Critical” is not necessarily important (e.g. NTP can be “critical” but not worth notifying anyone)
Too many minor alarms can mask important problems and dull our response

Problem: monitoring environment too complex for one person to fully understand
Recent monitoring changes put us at risk
Best subject matter experts don’t understand the monitors well
Monitoring specialist doesn’t know the specifics of each service
Monitoring specialist is not the best person to have the primary responsibility for monitoring
Someone should review the service and tune the monitors, using monitoring specialist as a resource
Every monitor should be well described, including impact, probable causes, troubleshooting info

Lots of monitors in red/yellow/orange state for a long time means it’s tough to find interesting/relevant problems.
Chronic, frequent problems that don’t need a response are good candidates for nuking.
Sheer number of monitors makes it hard to evaluate whether monitoring is working right
Periodic review of disabled monitors (automatic report would be better).
Disabled notifications should be tracked by a ticket; when closing, reopen if the monitor is still disabled.
Don’t disable “checking”, only disable “notifications”, system should keep checking/recording.