Alerting and SLOs

Exam guide§3.4

Reliability is defined, measured, and defended with three linked terms - SLI, SLO, SLA - and an error budget. Alerting is how you defend it: build a policy that fires on the right condition, then spend the budget wisely by paging a human only when it matters. Metrics and uptime checks come from Cloud Monitoring; this page covers the alerting policies built on top of them and the SLOs that drive them.

SLI, SLO, and SLA

One sentence, all three terms: 'maintain an error rate of less than 0.3% for the billing system' - error rate is the SLI (what you measure), 0.3% is the SLO (the target), and the promise as a whole is the SLA.

SLI

Service Level Indicator

What you measure - a metric of one reliability aspect, ideally the ratio good events / all valid events, with a near-linear tie to user experience.

SLO

Service Level Objective

The internal target you hold the SLI to, just short of 100% (99.9% = "three nines"). SLI + a goal.

SLA

Service Level Agreement

The external promise to customers, with consequences (refunds/credits) when you miss it. Alert tighter than the SLA so you fix problems before the promise breaks.

The three terms build on each other: the SLI is the measurement, the SLO wraps it in a target, and the SLA turns that target into a customer contract with consequences.

The three terms on one latency example: the Indicator is what you measure (a request's latency), the Objective is the internal target (200 ms), and the Agreement is the external promise (300 ms) - miss it and you get an unhappy customer.

⚠GotchaSLI is measured, SLO is the target, SLA is the promise

Classic ordering trap: the SLI is the number (error rate), the SLO is the target on it (< 0.3%), and the SLA is the customer contract that pays out when you miss. Example: "maintain < 0.3% error rate for billing" - error rate is the SLI, 0.3% is the SLO.

SMART SLOs

You can't measure everything, so choose SLOs that are S.M.A.R.T.

S

Specific

"The 95th percentile of results return in under 100 ms" - not the subjective "is the site fast enough?".

M

Measurable

Built on an indicator that is a number or a delta - something you can put in a mathematical equation, not a feeling.

A

Achievable

100% availability sounds good but cannot be obtained, let alone held over a long window. Pick a target you can actually sustain.

R

Relevant

It has to matter to the user and help meet application goals. If it does not, it is a poor metric.

T

Time-bound

99% available - per year? month? day? A fixed Sunday-to-Sunday window or a rolling last-7-days? Without the window it cannot be measured accurately.

A good SLO passes all five tests: a specific, measurable indicator set to an achievable, relevant target over a defined time window.

Error budgets

The error budget is everything the SLO does not require. A 99.9% SLO leaves a 0.1% budget to spend on failures, risky rollouts, and maintenance. Alert when you are on track to burn it early.

An error budget is 100% - SLO. If the SLO is "90% of requests return in 200 ms", the budget is 10%. The budget is what you're allowed to spend on failures and risky changes. A great time to alert is when you're trending to spend the whole budget before the time window ends - not when it's already gone.

⚠GotchaBeing out of SLO must have consequences

Just like breaching an SLA, being out of SLO should trigger concrete action - slow the rate of change, redirect engineering to reliability - and needs executive backing to enforce. An SLO nobody defends is decoration.

Alerting policies

An alerting policy = a name + condition(s) + notification channels + documentation. Notification channels are configured once and reused; direct-to-human ones are email, SMS, Slack, mobile app push, and PagerDuty, while Pub/Sub and webhook hand off to third-party systems.

The documentation field is a first-class part of the policy, not an afterthought: it travels with every notification and gives the responder context - a runbook/playbook, links to the right dashboard, and dynamic labels (${metric.label...}, ${resource...}) that fill in the specifics of the firing incident. Treat it as the "easy button" that tells whoever is paged what to do.

User-defined labels you attach to a policy also flow into the notification, so downstream systems can route and filter incidents by them (e.g. team, severity, service).

A policy can hold multiple conditions (e.g. one on CPU usage, another on reserved cores). A policy trigger decides how they combine: fire when all conditions are met (AND) or when any is met (OR).

Each condition reads IF metric CONDITION threshold FOR duration - the duration keeps a momentary spike from paging you.

An alerting condition reads: IF a metric CONDITION a threshold FOR a duration. The FOR window is what stops a single spike from paging you - the value has to stay breached for the whole duration before the policy fires.

⇄CompareAll vs any changes when the alert fires

allTwo conditions only page you when both are breached at once.

anyPages on the first breach.

Picking the wrong one is the difference between a noisy pager and a missed outage.

CommandsCreate a notification channel and inspect policies

gcloud beta monitoring channels list

gcloud alpha monitoring policies list

⚠GotchaNotification targets are channels, not a service

A classic trap: valid targets for an alert or uptime notification are channels - email, SMS, webhook, Pub/Sub, and third-party services (Slack, PagerDuty). An EC2 service (or any compute resource) is not a notification channel.

Metric-based vs log-based policies

Two policy types:

• Metric-based - track metric data collected by Monitoring (e.g. page a human when a VM's latency stays high). This is the default.
• Log-based - notify whenever a specific message appears in a log. Created from the Logs Explorer in Cloud Logging or via the Monitoring API. Example: alert when a human user accesses a service account's security key.

⚠GotchaLog-based alerting = "tell me when THIS log line appears"

"Alert me if anyone reads the service-account key" is a log-based alerting policy built in Logs Explorer - not a metric threshold. Metric-based alerts watch numbers; log-based alerts watch for a matching log entry.

Condition types

There are three condition types for metric-based alerts:

• Metric-threshold - fires when the metric is above/below a threshold for a duration window (the common case, see the diagram above).
• Metric-absence - fires when there are NO measurements for a duration window (the service stopped reporting).
• Forecast - predicts future behaviour from past data and fires when a time series is predicted to violate the threshold within a forecast window (warns you before the breach).

⚠GotchaAbsence and forecast, not just threshold

"Alert if the metric stops arriving" = metric-absence (not a threshold of 0). "Warn me before I run out of disk" = forecast. Reaching for a plain threshold for these is the classic wrong answer.

Incidents and snooze

When a policy's conditions are met, an event occurs and Monitoring opens an incident. Incident states:

• Firing - open; conditions still met (or no data proving otherwise). Usually a new/unhandled alert.
• Acknowledged - a technician has marked it as being handled, signalling others.

A snooze temporarily suppresses new notifications for a policy or incident - useful during a known, escalating outage so you're not buried in repeat pages.

#NumbersWays to create an alerting policy

• Console, gcloud CLI, Monitoring API, and Terraform all create policies.
• Trick: build one in the Console, then gcloud monitoring policies list + describe to get the JSON/YAML definition.
• A policy can hold up to 6 conditions; a Terraform policy needs display_name, combiner (how conditions combine), and conditions.

Alerting strategy

An alert is an automated notification sent through a channel to a person, system, or ticket when something needs to change (a service is down, an SLO is at risk). A good strategy balances four attributes.

Evaluating alerts

P

Precision

Of the alerts that fired, the share that were relevant. Lowered by false alerts. This is exactness.

R

Recall

Of the situations that should have alerted, the share that did. Lowered by missed alerts. This is completeness.

D

Detection time

How long until the system notices. Long detection burns error budget; too-fast triggers false positives.

T

Reset time

How long alerts keep firing after the issue is resolved. A long reset causes confusion about what is still broken.

Four measures grade an alert: precision and recall trade exactness against completeness, while detection and reset time govern how fast it reacts and how cleanly it clears.

Window length

The alert window is a subdivision of the SLO period. Short windows detect and reset fast but false-positive more; long windows are precise but spend more error budget before firing.

?DECISIONHow long should the alert window be?

Short window (e.g. 10 min)→fast detection + reset, but noisy

Long window (e.g. 36 hr)→precise, but slow to fire and clear

On a 99.9% availability SLO over 30 days: a short 10-min window alerts on a full outage in ~0.6 s and spends only ~0.02% of the budget, but it also fires on brief blips. A long 36-hr window catches a full outage in ~2 min and rarely false-alarms, but by the time that single alert fires it has already spent a full ~5% of the budget.

Pick this when: short = fast + noisy; long = precise + slow

Strategies that get both precision and recall

1

Short window + successive-failure count

One bad window will not page; N in a row will. Spots issues fast but treats a single blip as an anomaly. Risk: errors that spike up and down may never trip it.

2

Multiple conditions

Combine conditions in one policy (and even multi-step logic via Pub/Sub to Cloud Run) to get good precision, recall, detection, and reset at once. You do not have to pick just one.

3

Prioritize by customer impact / SLA

Use severity levels so only critical alerts reach humans - high-priority to Slack/SMS/PagerDuty, low-priority logged, emailed, or filed as a ticket. Severity travels in the notification (Email, Webhook, Pub/Sub, PagerDuty) for downstream routing.

Three ways to raise both precision and recall: react to sustained failure, combine conditions in one policy, and route by severity so only critical alerts reach a human.

⚠GotchaAlert on symptoms, keep humans for what matters

Alert on symptoms (failing queries) rather than causes (the database is down), and use multiple notification channels (email and SMS) so there's no single point of failure. Customize each alert to its audience with the action to take. And don't page a human unless the alert crosses a criticality threshold - alerts fired on everything get ignored (alert fatigue), so if an alert doesn't lead to a specific action, question whether it should exist.

Service monitoring

Modern apps are many services, and one failure can look like many. Service Monitoring helps you define services, set SLOs and alerts on them, and see their health in one place. It can auto-discover candidate services of type: GKE namespaces, GKE services, GKE workloads, and Cloud Run services.

Request-based vs windows-based SLOs

Request-based SLOs count good vs total requests; windows-based SLOs count good vs bad time windows. Windows hide bursts - a short daily outage can pass a windows-based SLO while failing every request in that window.

?DECISIONRequest-based or windows-based SLO?

Request-based→good requests / total requests (e.g. 99.9% of requests under 100 ms). Direct, per-request.

Windows-based→good windows / total windows (e.g. 99% of 10-min windows are compliant). Each window is one data point - can HIDE short bursts of total failure.

Pick this when: request = per-request ratio; windows = per-interval ratio, can hide bursts

⚠GotchaWindows-based SLOs can hide bursts

If a system returns nothing but errors every Friday 9:00-9:05 but is perfect otherwise, a windows-based SLO averaged over long windows may never register a violation - even though real users hit the outage. Request-based counts every bad request.

Burn-rate alerts

Service Monitoring can alert when a service is trending to violate its SLO. It uses a lookback window to examine recent trend and a burn-rate threshold to decide. A burn rate of 1 would spend 100% of the error budget exactly by the end of the compliance period; a higher threshold means you're burning faster than sustainable. Example: a 60-minute lookback on a 7-day period, alerting if the trend would burn the whole budget in 1/10th of the period or faster. Compliance periods are calendar-based or rolling.

≡FactsCreating an SLO in Service Monitoring

• Pick an SLI metric: Availability (successful responses / all responses), Latency (calls under a threshold / all calls), or Other (build your own in Metrics Explorer).
• Choose request-based or windows-based.
• Set a compliance period (calendar or rolling) and length, and a performance goal (%); the goal sets your error budget.
• Optionally attach a burn-rate alerting policy with a lookback window.