Day 8: Alerts and Reports
What You Will Learn Today
- Scheduled searches
- Creating and sharing reports
- Configuring alerts
- Alert actions
- Search macros
Scheduled Searches
Scheduled searches let you run searches automatically on a recurring basis.
flowchart LR
Schedule["Schedule<br>Hourly / Daily / Cron"]
Search["SPL Search<br>Execution"]
Result["Results"]
Action["Action<br>Email / Alert"]
Schedule --> Search --> Result --> Action
style Schedule fill:#3b82f6,color:#fff
style Search fill:#22c55e,color:#fff
style Result fill:#f59e0b,color:#fff
style Action fill:#8b5cf6,color:#fff
Cron Expressions
| Field | Range | Example |
|---|---|---|
| Minute | 0-59 | */5 (every 5 minutes) |
| Hour | 0-23 | 9 (9 AM) |
| Day | 1-31 | 1 (1st of month) |
| Month | 1-12 | * (every month) |
| Day of week | 0-6 (Sun-Sat) | 1-5 (weekdays) |
# Every hour on the hour
0 * * * *
# Daily at 9 AM
0 9 * * *
# Every hour on weekdays
0 * * * 1-5
# Every 5 minutes
*/5 * * * *
# Midnight on the 1st of each month
0 0 1 * *
Creating Reports
From the Web UI
- Run a search
- Click Save As > Report
- Enter a title and description
- Optionally configure a schedule
Report Settings
| Setting | Description |
|---|---|
| Title | Report name |
| Description | What the report covers |
| Time Range | The search time window |
| Schedule | When the report runs |
| Trigger Actions | Post-execution actions |
Scheduling a Report
Title: Daily Error Summary
Schedule: 0 9 * * * (daily at 9 AM)
Time Range: Previous day (-1d@d to @d)
Report Permissions
| Permission Level | Description |
|---|---|
| Private | Visible only to the creator |
| This app only | Shared with users of the same app |
| All apps | Shared across all apps |
Configuring Alerts
Alert Types
flowchart TB
subgraph AlertTypes["Alert Types"]
Scheduled["Scheduled Alert<br>Periodic checks"]
Realtime["Real-time Alert<br>Instant detection"]
end
style Scheduled fill:#3b82f6,color:#fff
style Realtime fill:#ef4444,color:#fff
| Type | Description | Use Case |
|---|---|---|
| Scheduled | Runs a search on a schedule | Daily reports, periodic health checks |
| Real-time | Monitors continuously | Critical anomalies requiring immediate response |
Creating an Alert
- Run a search:
index=main sourcetype=access_combined status>=500
| stats count
| where count > 10
- Click Save As > Alert
- Configure the alert settings
Trigger Conditions
| Condition | Description | Example |
|---|---|---|
| Number of Results | Trigger based on result count | > 0 |
| Number of Hosts | Trigger based on host count | > 3 |
| Number of Sources | Trigger based on source count | > 5 |
| Custom | Custom SPL condition | Defined in the search |
Throttling
Throttling prevents the same alert from firing repeatedly in a short period.
| Setting | Description |
|---|---|
| Suppress | Silence the alert for a specified duration |
| Suppress fields | Suppress per unique field value |
Suppress for: 1 hour
Suppress if field value matches: host
-> The same host triggers the alert at most once per hour
Alert Actions
Alert actions define what happens when an alert fires.
flowchart TB
Alert["Alert<br>Triggered"]
Email["Send Email"]
Webhook["Webhook"]
Script["Run Script"]
Log["Log Event"]
Ticket["Create Ticket<br>ServiceNow, etc."]
Alert --> Email
Alert --> Webhook
Alert --> Script
Alert --> Log
Alert --> Ticket
style Alert fill:#ef4444,color:#fff
style Email fill:#3b82f6,color:#fff
style Webhook fill:#22c55e,color:#fff
style Script fill:#f59e0b,color:#fff
style Log fill:#8b5cf6,color:#fff
style Ticket fill:#3b82f6,color:#fff
Email Action
| Setting | Value |
|---|---|
| To | ops-team@example.com |
| Subject | [ALERT] Server Error Count Exceeded |
| Message | Summary of results with a link |
| Include | CSV attachment, inline results, etc. |
Webhook Action
{
"text": "Alert: $name$ triggered at $trigger_time$",
"result": "$result$"
}
Webhooks integrate with Slack, Microsoft Teams, PagerDuty, and other services.
Script Action
#!/bin/bash
# $SPLUNK_HOME/bin/scripts/alert_handler.sh
echo "Alert: $1" >> /var/log/splunk_alerts.log
# Additional logic: create a ticket, trigger remediation, etc.
Search Macros
Search macros let you save frequently used SPL fragments as reusable shortcuts.
Creating a Macro
Navigate to Settings > Advanced search > Search macros > Add new
Name: get_errors
Definition: index=main sourcetype=access_combined status>=400
Using a Macro
`get_errors`
| stats count by status
Macros with Arguments
Name: get_errors(1)
Arguments: min_status
Definition: index=main sourcetype=access_combined status>=$min_status$
`get_errors(500)`
| stats count by host
`get_errors(400)`
| timechart span=1h count
Useful Macro Examples
# Error rate calculation
Name: error_rate
Definition: stats count(eval(status>=400)) AS errors, count AS total | eval error_rate=round(errors/total*100, 2)
# Usage
index=main sourcetype=access_combined
| `error_rate`
# Time-of-day label
Name: time_label
Definition: eval time_label=case(date_hour>=6 AND date_hour<12, "Morning", date_hour>=12 AND date_hour<18, "Afternoon", date_hour>=18 AND date_hour<22, "Evening", 1=1, "Night")
Workflow Actions
Workflow actions add custom context-menu options to events in search results.
Workflow Action Types
| Type | Description |
|---|---|
| GET link | Opens an external URL |
| POST link | Sends data to an external URL |
| Search | Runs a new search |
Example: IP Investigation
Name: Investigate IP
Type: GET link
URI: https://www.virustotal.com/gui/ip-address/$clientip$
Apply to: field = clientip
Hands-On: Automating Operations Monitoring
1. Error Spike Alert
index=main sourcetype=access_combined status>=500
| bin _time span=5m
| stats count by _time
| where count > 50
Configuration:
- Schedule:
*/5 * * * *(every 5 minutes) - Trigger: Number of Results > 0
- Suppress: 30 minutes
- Actions: Email + Slack Webhook
2. Daily Report
index=main sourcetype=access_combined earliest=-1d@d latest=@d
| stats
count AS total_requests,
dc(clientip) AS unique_visitors,
avg(response_time) AS avg_response_time,
count(eval(status>=400)) AS errors
| eval error_rate = round(errors/total_requests*100, 2) . "%"
| eval avg_response_time = round(avg_response_time, 3) . "s"
Configuration:
- Schedule:
0 9 * * *(daily at 9 AM) - Action: Email with CSV attachment
3. Disk Usage Monitoring
index=_internal source=*metrics.log group=per_index_thruput
| stats sum(kb) AS total_kb by series
| eval total_gb = round(total_kb/1024/1024, 2)
| where total_gb > 10
| sort -total_gb
Summary
| Concept | Description |
|---|---|
| Scheduled search | Automatically runs a search on a recurring schedule |
| Report | A saved, shareable search result |
| Alert | A notification triggered by search conditions |
| Alert action | Email, webhook, script, or ticket creation |
| Throttling | Prevents duplicate alerts in a short window |
| Macro | A reusable SPL fragment |
Key Takeaways
- Scheduled searches automate routine monitoring tasks
- Alerts need well-tuned conditions and appropriate actions
- Throttling prevents alert fatigue
- Macros improve SPL reusability and consistency
Exercises
Exercise 1: Basic
Create an alert that checks for error events (status >= 500) every 5 minutes and logs an entry when the count exceeds 10.
Exercise 2: Applied
Set up a daily report that summarizes the previous day's statistics and sends it via email at 9 AM.
Exercise 3: Challenge
Create a parameterized search macro error_summary(2) that accepts sourcetype and threshold as arguments. The macro should return results only when the error count for the given sourcetype exceeds the threshold.
References
Coming up next: In Day 9, you will dive into Splunk administration -- managing indexes, roles, knowledge objects, and configuration file precedence.