Metrics, logs & alerting platform

First, prevent it: enforce per-tenant and per-metric limits on active series and label values at the ingester, and reject over-limit series with a counter the owning team can see. Second, detect it: track series growth per metric name so we can spot the label that exploded, usually a user ID, pod name churn or raw URL path. Third, fix the source: drop or rewrite the label with relabelling in the collector, bucket URL paths into route templates, and move per-request detail to logs or traces, which handle high cardinality better. For genuine needs, use recording rules to pre-aggregate. The key principle is that one team's bad deploy must never degrade ingestion or alerting for everyone else.

I alert on symptoms users feel, ideally SLO burn rates with a fast and a slow window, rather than every resource threshold. Rules use a 'for' duration to skip short blips. Rulers run replicated so a single failure does not miss evaluations, and Alertmanager clustering deduplicates the resulting duplicate notifications. Alertmanager groups alerts by service and cluster so a database outage becomes one page, not fifty, and inhibition suppresses downstream alerts when an upstream one fires. Routing uses team and severity labels, mapped to on-call schedules with escalation if nobody acknowledges. Finally, a heartbeat alert to an external service proves the pipeline itself is alive, and we review noisy alerts after every rotation.

Agents on every node scrape Prometheus endpoints and tail container logs. They attach cluster, namespace and service labels, and buffer to local disk if the pipeline is down. They publish compressed batches to separate Kafka topics for metrics and logs, so a traffic spike lands in the queue instead of the stores. Metrics ingesters enforce per-tenant limits, replicate writes and hold recent data in memory and a write-ahead log. Compactors then move blocks to object storage and build downsampled rollups. The log pipeline parses fields, redacts secrets and writes compressed chunks with a small label index. Rulers evaluate recording and alert rules. Alertmanager groups, deduplicates, silences and routes alerts to on-call schedules. Grafana ties dashboards and logs together for whoever gets paged.

I run meta-monitoring on a separate, minimal stack, ideally in another region or account, so it does not share failure modes with the main platform. It watches ingestion rate, Kafka consumer lag, ingester health, rule evaluation failures and missed iterations, and Alertmanager notification failures. I also use a dead man's switch: an always-firing Watchdog alert sent to an external service. If that heartbeat stops arriving, the external service pages us, because silence means the alert path is broken. I add end-to-end synthetic probes too. A canary writes a known metric and log line, and a checker confirms both are queryable within a freshness target. That catches cases where every component looks healthy but data is not actually flowing.

Load usually peaks exactly when production is broken, so I design shedding to follow priority. Alert-relevant data gets protected capacity: a separate high-priority Kafka topic or quota for metrics that SLO and alert rules use, and dedicated ruler and Alertmanager resources. I shed in this order: expensive ad-hoc queries and long-range dashboards, then debug-level logs, then low-priority tenants' excess series. Every drop increments a visible counter per tenant and reason, so nothing is lost silently. Query concurrency limits and timeouts stop one heavy dashboard from starving the rule evaluator. Agents keep buffering to disk, so delayed data still arrives later. The aim is to degrade exploration first and keep detection and paging working.

I state assumptions first. Say 10 million active series scraped every 15 seconds. That is about 667,000 samples per second. Prometheus-style compression stores roughly 1 to 2 bytes per sample, so raw data is around 1.3 MB per second at most, about 115 GB a day before replication. With replication factor 3 in ingesters, hot storage triples, though object storage blocks are deduplicated after compaction. Memory usually dominates: each active series in the head block carries labels and chunk overhead, so active series count, not sample rate, drives ingester sizing. Then I add Kafka retention for a few hours of replay, plus query capacity. Finally I plan headroom for spikes and for losing one zone.

Defence in depth, as early as possible. First, application logging libraries and code review discourage logging tokens, passwords or personal data. Second, agents or the first pipeline stage run redaction: known secret patterns such as API keys, bearer tokens and card numbers, plus field-based rules that mask fields like email or phone. Redaction happens before data touches Kafka retention or storage, because deleting it later is hard. Third, access is restricted by tenant and team with audit logging. I also limit retention for logs that may contain personal data, which matters for obligations like India's DPDP Act. Finally, I sample stored logs for leaks and treat any leak as an incident: rotate the secret, then purge the affected chunks.

Burn rate is how fast you consume the error budget compared with spending it evenly over the SLO window. A burn rate of 1 uses the whole budget exactly over the window. A fast-burn alert pages at a high burn rate measured over a long window, such as one hour, and also requires a short window, such as five minutes, to agree. That combination catches serious incidents quickly and resolves soon after recovery. A slow-burn alert uses a lower burn rate over hours and creates a ticket instead of a page. A raw threshold like 'errors above 1 percent' ignores traffic volume and budget context. It is either noisy or too late. Burn-rate alerts tie paging directly to user impact.

The queue decouples producers from storage. Traffic spikes during incidents, exactly when observability matters most, so Kafka absorbs bursts and lets ingesters catch up rather than dropping data. It also lets us restart or upgrade the TSDB without losing writes, replay data after a bug, and fan the same stream to multiple consumers like a security pipeline. The costs are extra latency, another stateful system to operate, and storage for retention on the brokers. Ordering only holds within a partition, so I partition by series hash to keep samples for one series together. For alerting freshness, I watch consumer lag closely, because a lagging ingester delays alerts as well as dashboards.

I size the buffer to cover the longest pipeline outage I want to survive without loss, for example an hour. The rough formula is node ingest rate times outage duration, compressed, plus headroom, and I check it against node disk quotas. The buffer must be bounded so it can never fill the node disk and hurt the workload. When it fills, the agent drops with a policy: oldest or lowest-priority data first, debug logs before error logs, never silently. It exposes buffer usage and dropped-bytes metrics. Meta-monitoring alerts when buffers keep growing across many nodes, because that means the pipeline, not one node, is down. On recovery, replay is rate-limited so a thousand agents do not flood Kafka at once.

A full-text index can be as big as the data and costs a lot of CPU at write time. Most log queries start from a known service, namespace or time range anyway. With a small label index, I store compressed chunks cheaply in object storage and filter them with brute-force parallel scans at query time. Ingest becomes cheap and retention affordable. The trade-off is slower needle-in-haystack searches across huge time ranges, and labels must stay low-cardinality, so no request IDs as labels. It is the wrong choice when users mainly run arbitrary full-text search across everything, like security analytics or audit investigations. In those cases a search engine such as Elasticsearch or OpenSearch, or a columnar store, fits better.

Isolation sits at every layer. At ingest, per-tenant limits cover samples per second, active series, labels per series, label value length and log bytes per second, with clear rejection counters. In Kafka, partitioning by tenant and series hash stops one tenant pinning a single partition, and big tenants get more partitions or shuffle sharding across ingesters. On the query path, per-tenant concurrency, series-fetched and bytes-scanned limits apply, plus fair queuing in the query frontend. Rulers can be sharded so one tenant's heavy rules do not delay another's evaluations. Tenant identity comes from authentication, not a client-supplied header. Each tenant gets a usage dashboard, so teams see their own cost and limits before they hit them.

Each series hashes onto a ring, and the write goes to several ingesters, typically three. It succeeds when a quorum, two of three, acknowledges. A single ingester failure then loses no acknowledged data, and writes continue. Queries fan out to the replicas and merge the results, deduplicating identical samples. So the recent window is effectively read with quorum semantics: you see acknowledged data even if one replica is behind. Consistency is eventual at the edges. Data still in agent buffers or Kafka is not visible yet, so I track ingestion delay and expose freshness. After compaction, object storage blocks from the replicas are merged and deduplicated, which cuts long-term storage back to one copy.

Each replica adds external labels for cluster and replica. The distributor runs HA deduplication: for each cluster it elects one replica as leader and accepts only that replica's samples, dropping the other's. If the leader stops sending for longer than a failover timeout, the distributor switches to the other replica. A short gap or overlap can occur at failover, which is acceptable for monitoring. The replica label is stripped before storage so series identities match. Without this, you double storage and active series, and sum queries double count. The same problem appears for alerts: both replicas fire identical alerts, which is why Alertmanager dedupes by label set.

A query frontend splits the range into smaller intervals, often by day, checks a results cache and runs the pieces in parallel. Each querier fetches recent data from ingesters and older data from store gateways, which read block indexes and chunks from object storage and cache index headers locally. Results are merged and deduplicated where the windows overlap. For long ranges the querier can pick downsampled blocks based on the step, so a 30-day graph with a one-hour step reads 1-hour rollups instead of raw samples. Limits on series and bytes fetched protect the system. Caching index lookups, chunks and query results matters most, because object storage latency is high.

A single average per window destroys information. You lose spikes, and you cannot correctly compute rates or re-aggregate. So each rollup stores several aggregates: count, sum, min and max, plus counter-aware values so rate() still works across resets. Queries then choose the right aggregate, for example max for latency peaks. Histograms must keep bucket counts, not precomputed percentiles, because percentiles cannot be averaged. Users should know a 1-hour rollup cannot show a two-minute outage, so dashboards auto-select resolution based on step. Downsampling saves query time more than storage. I keep raw data long enough for incident reviews, then expire it per tenant policy.

The 'for' duration keeps an alert pending until the condition has held across evaluations for that long. That filters brief blips but adds detection delay. Five minutes of 'for' means at least five minutes before anyone is paged. Evaluation interval sets resolution: a 1-minute interval with a 15-second scrape can miss short spikes between evaluations. Very short intervals cost ruler CPU and load the query path. For critical user-facing SLOs I keep 'for' short and rely on burn-rate windows for smoothing. For noisy infrastructure signals I use longer 'for' or keep_firing_for to avoid flapping. If the condition flickers, the pending timer resets, so I smooth with range functions rather than stretching 'for'.

Threshold alerts fail silently when the series disappears, because there is nothing to compare. So I add explicit absence checks. absent() or absent_over_time() covers critical series, and up == 0 or missing targets covers scrapes. Freshness alerts track ingestion delay per tenant. For pipelines I alert when Kafka consumer lag grows or when agents report rising buffer usage. I also use synthetic canaries that write known data and verify it end to end. Alerts should be written so 'no data' reads as a problem, not as 'all fine'. In Grafana-managed alerts, that means setting the no-data state deliberately instead of leaving the default.

I run rulers as a sharded group. Each rule group is assigned to an owner through a hash ring, and replicas can take over if the owner fails. Some setups run two evaluators per group on purpose and rely on Alertmanager to deduplicate, since identical label sets collapse into one alert. I monitor missed and slow evaluations, because a ruler that falls behind is effectively blind. Rulers query ingesters and storage directly, with reserved query capacity, so a dashboard storm cannot starve them. On failover, alert state such as pending timers may reset unless state is restored from the ALERTS_FOR_STATE series. That is why Prometheus persists it.

Alertmanager runs as a cluster of usually three replicas that gossip silences and notification logs. Rulers send every alert to all replicas. Each replica waits a position-based delay before sending, and checks the shared notification log first, so normally only one notification goes out. The guarantee is at-least-once, not exactly-once. During a network partition, two replicas may both notify. That is the right bias, because a duplicate page is far better than a missed one. Receivers such as paging tools should deduplicate on a stable key, like the group key. I also monitor failed notifications per integration and keep a secondary channel for the case where the primary paging provider is down.

Per-tenant and per-stream ingestion rate limits kick in at the agent or distributor. Excess lines are rejected or sampled, and each drop is counted by tenant and reason. Kafka absorbs the short-term surge, so other tenants are unaffected. I give priority to error and warn levels over debug, so the useful signal survives. The tenant gets an automatic notification with a usage dashboard showing what is being dropped. Agents can apply dynamic sampling or filter rules pushed from central config without a redeploy. Longer term, cost attribution per team and log-level defaults in libraries prevent repeats. The key point is that the platform degrades that one tenant visibly, not everyone silently.

Agents authenticate with mTLS or short-lived tokens tied to workload identity. The gateway derives the tenant from that identity and ignores any client-claimed tenant header, which prevents writing into or reading another tenant's data. All traffic is encrypted in transit. Kafka uses ACLs per topic, and object storage is encrypted at rest with scoped access for platform services only. On the query side, SSO with RBAC maps users and teams to tenants, with audit logs of who queried what, especially for logs. Alert webhooks and notification secrets are kept in a secret manager. Rate limits protect against abusive or runaway queries. I also treat the observability store as sensitive, because logs often contain more than people intend.

I treat the paging provider as a dependency with its own failure mode. Alertmanager metrics for failed notifications per receiver trigger an alert that goes through a different channel, such as a second provider, SMS gateway or chat. Critical routes can fan out to two receivers at the same time, and the receivers dedupe. The dead man's switch heartbeat also detects paging-path failure from the outside. Escalation policies make sure an unacknowledged page reaches a secondary on-call. Teams keep a documented manual fallback, such as a status channel and phone tree, which is tested in game days. Notifications retry with backoff, so alerts queue rather than vanish.

I track consumer lag per partition, both in messages and in time. Time lag matters more because it shows how stale dashboards and alerts are. Alerting on lag time, not just message count, tells me when detection is degraded. To recover, I scale ingesters or consumers, keeping partition count at least equal to consumer parallelism, and check whether one hot partition is the bottleneck. If lag is beyond Kafka retention, data is lost, so retention must exceed the worst expected outage. During catch-up, ingesters must accept older samples. That requires an out-of-order window, or prioritising the newest data for alerting and backfilling the rest. Rules should flag low data freshness to on-call rather than fire false 'all clear' results.

I use pull at the node level: agents scrape local Prometheus endpoints. Then I push from agents into Kafka. Pull gives health for free through the up metric, lets operators control scrape rate, and makes it easy to curl a target to debug. Push from agents to the central pipeline works better across clusters and networks, avoids central scrapers reaching into every cluster, and suits the buffering and batching that Kafka needs. For short-lived batch jobs that end before a scrape, I use push through something like a Pushgateway or OTLP. OpenTelemetry collectors support both models, which helps standardise. Mostly it is a hybrid design with clear ownership.

Correlation depends on consistent identity. Agents attach the same resource labels, cluster, namespace, service and pod, to metrics, logs and traces, ideally following OpenTelemetry semantic conventions. Grafana can then jump from a panel to logs filtered by the same labels and time range. Exemplars attach trace IDs to histogram samples, so a latency spike links straight to example traces. Logs include trace IDs as a parsed field, not a label, to avoid cardinality issues. Every alert carries links to its runbook, dashboard and a pre-filtered logs query, so the engineer does not start from a blank screen. Time synchronisation across nodes also matters, because skew breaks correlation.

Each tenant has a policy: how long raw data stays hot, how long downsampled blocks stay in object storage, and when everything expires. Compactors enforce expiry by deleting whole blocks past the window, which is cheap because blocks are time-partitioned. Log chunks expire through object storage lifecycle rules or the compactor. Targeted deletes, such as purging logs that leaked personal data, are harder. I support delete requests that mark series or streams through tombstones, filter them at query time, and rewrite the affected blocks during compaction. Deletion takes time, so I document the SLA. Kafka retention, caches and backups also need bounded lifetimes, or deleted data lingers somewhere.

Ingesters hold recent samples in memory, so restarts need care. Every write goes to a write-ahead log on persistent disk first, so a restarted ingester replays it and recovers. With replication factor 3 and quorum writes, restarting one ingester at a time keeps writes succeeding. I roll one zone at a time with zone-aware replication and wait for each ingester to become ready, WAL replayed and back in the ring, before moving on. Kafka helps further: if ingesters consume from Kafka, they can resume from committed offsets after restart. I watch ingestion errors, ring health and query errors during rollout, and stop automatically if they rise.

Alert rules live in version control and go through review, like code. CI runs syntax checks and unit tests, for example promtool test rules, with synthetic series to confirm an alert fires and resolves as expected. Every paging rule must have an owner team label, a severity, a runbook link and a summary that says what is broken for users. Rules are deployed per tenant through an API, not edited by hand in production. I backtest new rules against historical data to estimate how often they would have fired. Rules that never fire or always fire are reviewed regularly. Game days inject failures to prove the important alerts actually page.

I track pages per on-call shift, the share of pages that were actionable, time to acknowledge, time to resolve, and how often alerts flap or auto-resolve before anyone acts. I also track incidents that were found by customers or humans instead of alerts, which counts as a missed detection. Each page gets a quick post-shift label: actionable, noise or duplicate. That data drives tuning: delete, demote to ticket, adjust 'for', or convert to burn-rate. Grouping effectiveness shows whether one incident caused one notification or fifty. Alert quality should be reviewed like reliability, with clear owners, because alert fatigue directly causes missed real incidents.

I keep ingestion regional. Agents write to the nearest region's Kafka and storage, so a cross-region link failure does not blind a region, and data residency rules can be honoured. Each region runs its own rulers and Alertmanager for local alerts, so paging keeps working if another region fails. Global views come from a query layer that fans out across regions, accepting partial results with a clear warning. Object storage can be replicated cross-region for durability of long-term blocks. Meta-monitoring for each region runs from a different region. I define RPO and RTO for the platform explicitly and test region failover, including where on-call notifications come from.

End to end it is at-least-once. Agents retry on failure, Kafka producers may resend, and consumers may reprocess after a rebalance, so duplicates happen. For metrics this is mostly harmless. A sample is identified by series and timestamp, so the TSDB deduplicates identical samples and rejects conflicting duplicates. Replicated writes and HA pairs are deduplicated at query or compaction time. For logs, duplicate lines are possible. Some stores drop exact duplicates within a stream, and otherwise I accept rare duplicates. Idempotent Kafka producers reduce resends. Exactly-once across agents, Kafka and stores is expensive and unnecessary here. Losing data is the real risk, so I bias toward retries and visible drop counters.

Standardise the resource labels centrally. The agent attaches cluster, namespace and service, so teams cannot omit or misspell them. I publish a naming guide: base units in metric names, _total for counters, and no unbounded values like user IDs, emails or full URLs as labels. Relabel rules at the agent drop or rewrite known-bad labels. Per-tenant limits on labels per series and label value length catch the rest. A cardinality explorer shows teams their top series and labels. CI linting on instrumentation and dashboards helps too. Golden libraries give teams pre-labelled HTTP and RPC metrics with route templates rather than raw paths, so most teams get it right by default.

1. Ahttp_method
2. Bstatus_code
3. Cuser_idCorrect answer
4. Dregion

Why: A user ID has an unbounded number of values, so each new user creates new time series, while the others have small fixed sets.

1. ATo suppress all other alerts during maintenance
2. BTo measure query latency in Grafana
3. CTo test the paging tool's sound settings
4. DTo detect when the alerting pipeline itself stops working, because the heartbeat goes missingCorrect answer

Why: An external service expects the heartbeat continuously. If it stops arriving, something in the alert path is broken, and the external service pages.

1. AIt sends a page and repeats it every 5 minutes
2. BIt fires immediately and resolves after 5 minutes
3. CIt enters a pending state and fires only if the condition stays true for 5 minutesCorrect answer
4. DIt is evaluated only every 5 minutes

Why: The alert stays pending until the condition has held continuously across evaluations for the 'for' duration.

1. A500
2. B1,500Correct answer
3. C300
4. D108

Why: Series count is the product of label value combinations: 3 × 5 × 100 = 1,500.

1. ATo reduce storage and speed up long-range queriesCorrect answer
2. BTo deduplicate alerts from replicated rulers
3. CTo encrypt data at rest
4. DTo make recent alerts fire faster

Why: Downsampling replaces many raw samples with aggregated rollups, cutting storage and the number of points scanned for long time ranges.

1. AInhibition rulesCorrect answer
2. BRecording rules
3. CSilences
4. DGrouping

Why: Inhibition rules mute matching alerts while a specified source alert is firing; silences are manual and time-bound, and grouping only batches notifications.

1. AGzip compression of each individual sample
2. BStoring only the average value per minute
3. CFull-text indexing of label values
4. DDelta-of-delta encoding for timestamps and XOR encoding for float valuesCorrect answer

Why: Gorilla-style compression stores timestamp delta-of-deltas and XORs consecutive values. Regular scrape intervals and slowly changing values make both very compact.

1. AGlobal ordering across all partitions of a topic
2. BOrdering within a single partition onlyCorrect answer
3. CNo ordering guarantees at all
4. DOrdering only within a consumer group across topics

Why: Kafka orders messages only within a partition. Partitioning by series hash keeps each series' samples in order.

1. AThe rule evaluation interval
2. BHow long to wait before sending the first notification for a new group, so related alerts can be batchedCorrect answer
3. CHow often a firing alert is re-sent
4. DHow long a silence lasts

Why: group_wait delays the first notification of a new group, so alerts that arrive shortly after can join the same notification.

1. AAbout 2% of the monthly error budgetCorrect answer
2. BAbout 14.4% of the monthly error budget
3. CThe entire monthly error budget
4. DAbout 0.1% of the monthly error budget

Why: One hour is 1/720 of 30 days, and 14.4/720 equals 0.02, or 2% of the budget. The Google SRE Workbook uses this as a fast-burn paging threshold.

1. Arate() removes high-cardinality labels
2. Brate() converts counters into gauges permanently
3. Crate() computes per-second increase and handles counter resets after restartsCorrect answer
4. DRaw counters cannot be stored in a TSDB

Why: Counters only increase and reset on restart. rate() detects resets and gives a meaningful per-second rate.

1. AOnly the last value
2. BCount, sum, min and max, plus counter-aware valuesCorrect answer
3. COnly the median
4. DOnly the average

Why: Keeping several aggregates lets queries compute averages, peaks, troughs and rates. A single average loses spikes and breaks rate calculations.

1. ATake the maximum of each pod's p99
2. BSum bucket rates across pods by le, then apply histogram_quantileCorrect answer
3. CTake the median of each pod's p99
4. DAverage each pod's p99

Why: Percentiles cannot be averaged. Aggregating bucket counts first and then computing the quantile gives a correct estimate.

1. A3
2. BAll ingesters in the ring
3. C1
4. D2Correct answer

Why: A quorum of 3 is 2, so a write survives one ingester failure and still succeeds.

1. ATo recover in-memory samples after a crash or restartCorrect answer
2. BTo store downsampled rollups
3. CTo index log lines
4. DTo speed up long-range queries

Why: The WAL persists incoming writes to disk, so the in-memory head block can be rebuilt on restart without losing data.

1. ATo silence noisy alerts
2. BTo precompute expensive or frequently used expressions into new series for fast dashboards and alertsCorrect answer
3. CTo record user sessions in Grafana
4. DTo log every query for auditing

Why: Recording rules evaluate queries on a schedule and store the results as new series. This makes dashboards and alert rules cheaper and more consistent.

1. AThey are identical features
2. BA silence deletes alerts from the ruler
3. CAn inhibition only applies to log alerts
4. DA silence is a time-bound, manually created mute matching labels; an inhibition mutes alerts automatically when a matching source alert is firingCorrect answer

Why: Silences are explicit and expire, which suits maintenance windows. Inhibitions are rule-based and depend on another alert currently firing.

1. ACrash the agent so Kubernetes restarts it
2. BSilently discard all new data
3. CKeep writing until the node disk is full
4. DDrop lowest-priority data first and expose dropped-data countersCorrect answer

Why: The buffer must never harm the workload. Prioritised dropping keeps the most important data, and counters make the loss visible.

1. Atopk()
2. Bincrease()
3. Crate()
4. Dabsent() or absent_over_time()Correct answer

Why: absent() returns a result when no matching series exists. A threshold rule on missing data would otherwise never fire.

1. AAlertmanager is down
2. BDashboards and alerts are evaluating stale dataCorrect answer
3. CAgents are dropping data
4. DStorage is out of space

Why: Lag means ingesters have not consumed recent data yet. Queries and rules see an older picture, so detection is delayed.

1. ADuring monthly compaction
2. BNever; access control is enough
3. CIn Grafana when displaying results
4. DAs early as possible, at the agent or first pipeline stage before durable storageCorrect answer

Why: Early redaction stops sensitive data from spreading into Kafka retention, caches, storage and backups, where removing it is hard.

1. ARandomly, for load balancing
2. BFrom a header the client sets freely
3. CFrom the authenticated identity, such as an mTLS certificate or token, enforced at the gatewayCorrect answer
4. DFrom the metric name prefix

Why: Deriving the tenant from authenticated identity prevents a client from writing into or reading another tenant's data.

1. AAbout 15,000,000
2. BAbout 15,000
3. CAbout 66,700Correct answer
4. DAbout 1,000,000

Why: Each series yields one sample every 15 seconds, so 1,000,000 ÷ 15 ≈ 66,667 samples per second.

1. ADuplicate alerts
2. BData being encrypted twice
3. CLost ordering within a series
4. DA hot partition when one large tenant sends most of the trafficCorrect answer

Why: All of a big tenant's data lands on one partition and one consumer. Adding series hash spreads that load.

1. AA node's CPU is above 80% for one minute
2. BThe checkout SLO error budget is burning at a high rate over both 1h and 5m windowsCorrect answer
3. CA disk will be full in 30 days
4. DA deployment finished successfully

Why: Pages should be for urgent, user-impacting symptoms. A fast multi-window burn rate on a key SLO is both.

1. Aevaluation_interval
2. Bscrape_interval
3. Cgroup_interval
4. Drepeat_intervalCorrect answer

Why: repeat_interval sets the re-notification period for a group whose alerts have not changed. group_interval covers sending when new alerts join an existing group.

1. AEvery word in every log line
2. BOnly the stream labels, while log content sits in compressed chunksCorrect answer
3. CNothing at all; every query scans all data
4. DOnly timestamps

Why: Only stream labels are indexed. Line content is filtered by scanning chunks at query time, which keeps the index small and ingest cheap.

1. ABy comparing every sample value
2. BBy electing one replica per cluster, using cluster and replica labels, and accepting only its samplesCorrect answer
3. CBy dropping every other sample
4. DBy using Kafka exactly-once transactions

Why: The distributor tracks a leader replica per cluster label and drops samples from the other replica until failover.

1. AIt needs no service discovery
2. BIt removes the need for labels
3. CIt works best for jobs that finish in under a second
4. DThe scraper generates an 'up' metric, so a down target is detected automaticallyCorrect answer

Why: A failed scrape sets up to 0. That is an immediate health signal which push models must build separately.

1. AIncrease the scrape interval
2. BPush them through a Pushgateway or an OTLP pushCorrect answer
3. CIgnore batch jobs
4. DWrite them into log lines only

Why: Short-lived jobs may never be scraped. Pushing their final metrics to an intermediary or collector makes them observable.

1. ATrace IDs attached to specific metric samples, linking metrics to tracesCorrect answer
2. BSample dashboards for new tenants
3. CCompressed log chunks
4. DExample alert rules

Why: Exemplars attach a trace reference to a sample, such as a histogram bucket, so you can jump from a latency spike to a matching trace.

1. AIt is durable, scalable and cheap per GB for large volumes of immutable dataCorrect answer
2. BIt supports in-place sample updates
3. CIt removes the need for indexes
4. DIt has the lowest read latency

Why: Immutable blocks suit object storage's cheap, durable capacity. Caches and store gateways hide its higher latency.

1. AAt-most-once
2. BAt-least-once, so receivers should deduplicateCorrect answer
3. CBest-effort with no retries
4. DExactly-once

Why: Replicas coordinate through gossip but can both send during partitions. The system deliberately prefers a duplicate page over a missed one.