Summary
When infrastructure issues cause cascading failures across many monitors, users would typically receive dozens or hundreds of individual notifications. Alert Storm Protection detects this pattern and suppresses the flood of individual alerts, replacing them with a single notification.
Configuration
Alert storm protection settings are found under notification settings :
Users can configure storm protection settings at two levels:
Global level: Applies to all monitors not assigned to specific groups
Monitor group level: Applies only to monitors within a specific group
Key configuration options:
Threshold type: Can be based on absolute count or percentage of monitors in the pool
Threshold value: Number of monitors (e.g., 10) or percentage (e.g., 25%)
Time window: Duration in minutes over which failures are counted (default is 5 minutes)
Enable/disable: Users can turn the feature on or off
How It Works
Detection Phase
When a monitor goes down, the system records the event in a time-based bucket (per minute). It tracks the count of down monitors within the configured time window for each user's monitor pool.
Storm Activation
If the number of down monitors meets or exceeds the configured threshold within the time window, an alert storm is confirmed. At this point:
Individual "Down" notifications are suppressed for that pool
A single storm confirmation notification is sent to all configured alert contacts
The system tracks which alert contacts should receive storm notifications
During Active Storm
While a storm is active:
New monitor down events are counted but don't trigger individual alerts
Scheduled threshold and recurrence jobs are absorbed (cancelled)
The storm gate evaluates each notification and decides to pass through or absorb
Recovery Phase
When monitors start recovering:
The system checks if the number of still-down monitors has dropped below threshold
If threshold is no longer met (or all monitors are up), the storm is resolved
A single storm recovery notification is sent
Normal notification behavior resumes
Notifications
Notification Channels
The feature supports all notification channels for storm alerts:
Email with detailed storm information
Teams, Slack, Telegram, PagerDuty notifications
SMS and voice calls with storm status messages
All other alert channels
Storm notifications include:
Monitor group name (if applicable)
Threshold configuration (e.g., "10 within 5 minutes")
Number of monitors impacted
Storm start/end timestamps
Link to view all incidents