The easiest mining alert to build is a notification every time a number crosses a line. The harder task is making that notification useful to the person who receives it. An alert should identify an observed problem, show the relevant evidence, and make the next action clear. Otherwise a monitoring system can create more interruptions without making the underlying operation easier to manage.

This guide outlines an alerting workflow for mining telemetry, with attention to stale data, noisy thresholds, ownership, and recovery. The examples are design exercises, not manufacturer operating limits. Use the Mining Analytics overview to connect this workflow to your data model, and use the dashboard demo to rehearse states without interrupting real equipment.

Start with a symptom someone can act on

The Prometheus alerting guidance recommends simple alerts focused on symptoms, useful diagnostic consoles, and room for brief disturbances. Apply that principle by writing the action before the condition. Who should receive this alert, what can they inspect, and what would make an interruption worthwhile? An alert with no answer to those questions probably belongs in a chart or review report instead.

For example, “expected worker has no fresh observations” is a directly observable symptom. “The cooling fan has failed” is a diagnosis that may require additional evidence. Keep the message at the level the data supports. Link the symptom to the worker view and the collector status so the recipient can narrow the cause. That avoids teaching people to distrust confident messages that later turn out to be guesses.

Separate missing telemetry from low performance

A missing reading should not be converted into zero merely to simplify the alert expression. A zero-hashrate observation and an absent observation suggest different investigations. Define a freshness policy for each source and give it a distinct alert state. Store the original observation time so stale data remains recognizable even while the website itself is responding normally.

Consider a fictional collector expected to poll once per minute. You might test a freshness rule that becomes pending after several missed observations, but the actual interval must reflect your source and operational needs. Write the rule in terms of observation age, not page refresh time. Then test a collector outage separately from a worker outage. The system should help identify which evidence disappeared rather than collapsing both events into the same message.

Use persistence to avoid reacting to every blip

A condition that appears in one sample may disappear in the next. Decide how long a symptom must persist before notifying someone. The persistence period should match the consequence of the problem and the reliability of the signal. Do not copy a duration from an unrelated system and assume it is appropriate for every metric in a mining environment.

Distinguish the moment a condition first becomes true from the moment it is confirmed for notification. Store both if they matter to incident analysis. A pending state can be visible in the dashboard without producing an immediate interruption. Test the boundary cases: one bad sample, a sustained problem, and alternating good and bad observations. The rule should behave predictably in all three rather than flapping between urgent and resolved every minute.

Define recovery with the same care as triggering

An alert should not vanish merely because one good sample appears. Choose an explicit recovery rule and decide whether you need a different recovery threshold or a sustained healthy period. This separation, often called hysteresis when thresholds differ, can prevent repeated opening and closing around a boundary. Make the policy visible to the people using the alert.

For a freshness incident, recovery may require a valid new observation rather than just a successful TCP connection. For a performance incident, recovery might depend on a compatible measurement window. Use the signal that actually resolves the symptom. Record recovery time and retain the incident history. A resolved alert is still useful evidence when reviewing recurring issues, collector behavior, or the effect of an authorized operational change.

Make the notification small but complete

A useful message includes the worker or group, the observed symptom, the observation time, the source, and a link to the relevant view. Add an assigned owner or routing label when your process supports it. Keep secret values, private tokens, and unnecessary raw payloads out of the notification. A message may be forwarded or displayed on a locked-screen preview.

Describe what is known and what remains unconfirmed. “No valid observation for the configured interval; collector connection attempts timed out” gives the operator evidence. “Rig dead” is shorter but throws away both nuance and diagnostic value. Include a short next step such as checking collector reachability through an approved channel. The runbook can contain deeper context without forcing the notification to become a wall of text.

An upstream collection outage can produce many worker-level symptoms at once. Design grouping so a shared incident creates a manageable notification rather than a burst of nearly identical messages. Preserve the list of affected workers in the linked view. Grouping should reduce interruption, not erase the scope of the problem.

Think carefully about dependencies. If the collector has stopped, a lack of downstream observations may be explained by that known issue, but it does not prove that every worker is healthy. Show the dependency and the uncertainty. Avoid a rule that suppresses all worker information so completely that an unrelated issue becomes invisible. Test the grouping behavior with a fixture containing both a shared collector failure and an independently observed device problem.

Treat maintenance as a planned state

Planned work should have a time-bounded maintenance record, a clear target, and an owner. A permanent silence added during one repair can hide later incidents long after the original work is finished. Set an expiry and make the active maintenance state visible in the dashboard. Preserve observations during maintenance even when notifications are suppressed.

When maintenance ends, verify that normal collection and routing have resumed. Do not rely only on the expiry of a notification rule. A collector or credential may still need attention. Include a small return-to-service check in the runbook: fresh observation received, expected identity present, and no unexplained gaps in the relevant source. That is more useful than assuming the absence of alerts means every component is operating as intended.

Rehearse the runbook with fictional incidents

Create one fixture for each important state: healthy, pending, active, acknowledged, and recovered. Add cases for unavailable sensors and malformed responses. Ask a colleague to follow the runbook using only the information an on-call recipient would see. Where they need private background knowledge, improve the documentation or the message.

A good runbook explains how to confirm the symptom, identify the data source, inspect collection health, and record the outcome. It should not begin with disruptive steps such as restarting equipment before evidence is gathered. Keep equipment-specific operating actions in the authorized procedures for that environment. The ASIC telemetry guide helps define the source and capability information that makes those investigations more precise.

Review usefulness, not just alert volume

After a test or real incident, ask whether the alert reached the correct person, whether the evidence was current, and whether the next step was clear. Count duplicate interruptions and incidents where the initial message sent the operator in the wrong direction. Those observations help improve the system more than celebrating a large number of notifications.

Remove or downgrade alerts that consistently require no action. Split alerts that combine unrelated causes into an ambiguous message. Update fixtures whenever a new failure pattern appears, and replay them before changing the rules. Treat the alerting configuration as maintained application behavior, not as a collection of thresholds that nobody wants to touch after launch. A smaller set of trusted alerts is easier to operate than a noisy collection of unexplained warnings.

Conclusion: make every interruption earn its place

Mining alerts work when they preserve evidence, respect uncertainty, and point to a useful action. Separate stale telemetry from measured underperformance, use explicit trigger and recovery rules, and test the entire notification-to-runbook path. The objective is not constant urgency. It is a system that helps a responsible operator understand a problem quickly enough to choose the appropriate next step.