Device Reliability

Device Reliability tracks the operational health of every managed device in your fleet by collecting crash events, application hangs, service failures, hardware errors, and uptime data. The system computes a reliability score from 0 to 100 for each device using a weighted formula, identifies trend direction over time using linear regression, and surfaces the top issues affecting each device. Scores are recomputed automatically each time new telemetry arrives from the agent.

The Breeze agent collects reliability telemetry from platform-specific event sources: Windows Event Log, macOS system logs, and Linux journal/syslog. Each heartbeat cycle, the agent sends a snapshot of recent events to the API, which stores the raw history and then triggers an asynchronous score computation via BullMQ (with an inline fallback if the queue is unavailable).

---

Key Concepts

Data Collection Flow

1. Agent collects telemetry from the OS event log system. On Windows, this includes Event Log entries for BSODs, service crashes, hardware WHEA errors, and application hangs. On macOS, the agent reads system logs for kernel panics, application crashes, and launchd service failures. On Linux, the agent reads journald/syslog for kernel panics, OOM kills, systemd failures, and hardware errors.

2. Agent submits metrics to POST /agents/:id/reliability with the structured payload: uptime seconds, boot time, crash events, application hangs, service failures, and hardware errors.

3. API stores raw history in the device_reliability_history table. Each submission creates a new row, preserving the full event timeline.

4. Score computation is enqueued via BullMQ. If the queue is unavailable, the computation runs inline as a fallback.

5. The scoring service reads up to 90 days of history, aggregates events into daily buckets, computes sub-scores for each reliability factor, applies weights, and persists the result to the device_reliability table.

Reliability Score Bands

Band	Score Range	Meaning
Critical	0 – 50	Device has significant stability problems requiring immediate attention
Poor	51 – 70	Device is experiencing frequent issues that affect usability
Fair	71 – 85	Device is generally stable but has notable issues
Good	86 – 100	Device is operating reliably with minimal issues

---

Reliability Score

The reliability score is a weighted composite of five sub-scores, each calculated from the device’s event history over rolling time windows (7, 30, and 90 days).

Device-type-aware weighting

Not every device should be judged the same way. A server or network device is expected to stay powered on, so its uptime is a meaningful reliability signal. A laptop or desktop workstation is expected to sleep and shut down every day — penalizing it for that would produce a misleadingly low score.

Breeze therefore uses two weighting profiles based on device type:

• Servers, NAS, and network gear (and other always-on roles) — uptime counts for 30% of the score, as in the table below.
• Workstations (laptops and desktops) — uptime is not counted at all. The weight that uptime would have carried is redistributed across the fault factors that actually indicate a problem (crashes, hangs, service failures, hardware errors).

The practical effect: a laptop that is shut down overnight but otherwise crash-free scores near 100, instead of being dragged down for normal power-off behavior. The factor weights in the next table apply to servers and infrastructure; workstation-class devices carry no uptime weight and proportionally higher fault weights.

Factor Weights

Factor	Weight	Description
Uptime	30% (servers/infrastructure) / 0% (workstations)	Based on the device’s observed availability over the measurement window (how much of the time it was actually up and reporting in) since enrollment, up to a 90-day window – not just the length of the current boot session, so a single recent reboot doesn’t distort the figure. Score 100 at 100% availability, linearly down to 0 at 90% or below. Not counted for laptops and desktop workstations (see Device-type-aware weighting)
Crashes	25%	Penalizes crash events. Recent crashes (7-day) are weighted more heavily than 30-day crashes
Hangs	15%	Penalizes application hangs, with extra penalty for unresolved hangs
Service Failures	15%	Penalizes service failures, with partial credit for auto-recovered services
Hardware Errors	15%	Penalizes hardware errors by severity: critical (-30), error (-15), warning (-5) per event

Score Calculation

Each sub-score starts at 100 and is reduced as faults accumulate. The reduction is rate-normalized by observed up-days: faults are weighed against how many days the device was actually up and reporting, so a device that was offline for two weeks is not punished for events it could not have generated. Scores are also de-saturated so they spread across the full 0–100 range — heavily-faulting devices that previously all collapsed to the same low number now separate out, and sparse or newly-enrolled devices no longer read as artificially healthy.

The five sub-scores are combined using the weighted factors in the table above. The result is clamped to 0–100.

Note

Scores shift after the v0.85.0 agent update. Once agents in your fleet are promoted to v0.85.0, existing device scores will visibly change as the improved algorithm takes effect. This is expected behavior — not device degradation. If you have alert rules or thresholds based on reliability scores, re-baseline them after your fleet has updated and settled.

As of v0.86.0, the same classification cleanup applies to macOS and Linux devices, and runs server-side — so those scores shift when your server is upgraded to v0.86.0, regardless of agent version (the server also cleans historical data and data from devices still on older agents). The v0.85.0 Windows improvement, by contrast, takes effect as agents are promoted.

v0.89.0 adds two further server-side corrections: memory/disk event IDs are only counted as hardware errors when they come from a genuine hardware source (see Hardware Errors), and failures of Breeze’s own services no longer count against the service-failure factor (see Service Failures). Both apply on the next recompute — affected scores rise without an agent update.

Trend Direction

Trend direction is computed using linear regression over 30 days of daily reliability estimates. Each day’s events are scored independently, and a regression line is fitted to the daily scores.

Trend	Slope Threshold	Meaning
improving	slope > 2	Reliability is getting better over time
stable	-2 ≤ slope ≤ 2	Reliability is holding steady
degrading	slope < -2	Reliability is getting worse over time

The trendConfidence field (0.0 to 1.0) indicates how well the linear model fits the data, factoring in both R-squared and data coverage (at least 14 days of data for full confidence).

Mean Time Between Failures (MTBF)

MTBF is calculated from the 90-day window as:

mtbfHours = operatingHours / totalFailureCount

Where total failures include crashes, hangs, service failures, and hardware errors over the 90-day window. MTBF is null when there are zero failures or zero operating hours.

---

Tracked Metrics

Crash Events

System-level crashes that indicate an unexpected shutdown or critical failure.

Crash Type	Description	Platforms
bsod	Blue Screen of Death / bugcheck	Windows
kernel_panic	Kernel panic or oops	Windows, macOS, Linux
system_crash	General system or application crash	Windows, macOS
oom_kill	Out-of-memory kill	Linux
unknown	Unclassified crash event	All

Application Hangs

Detected when a process is reported as “not responding” or “hang” in system event logs.

Field	Type	Description
processName	string	Name of the hanging process
timestamp	ISO 8601	When the hang was detected
duration	integer	Duration of the hang in seconds (0 if unknown)
resolved	boolean	Whether the hang resolved without intervention

Service Failures

Detected when system services terminate unexpectedly or fail to start.

Field	Type	Description
serviceName	string	Name of the failed service
timestamp	ISO 8601	When the failure occurred
errorCode	string	OS-specific error code or event ID
recovered	boolean	Whether the service auto-recovered

Note

Failures of Breeze’s own agent, watchdog, and helper services do not count against the reliability score. These are most commonly logged during agent auto-updates, when the operating system briefly reports the Breeze service as “terminated unexpectedly” — indistinguishable at the OS level from a real crash. The events are still recorded and remain visible in the failure counts and the offenders drill-down, annotated “from Breeze services, not scored” — they just don’t lower the score.

Hardware Errors

Hardware-level errors from WHEA (Windows), MCE, disk I/O, and memory subsystems.

Hardware Type	Classification Criteria
mce	Machine Check Exception: WHEA source, “machine check”, or “mce” keywords
memory	Memory errors: EDAC or “memory” keyword; Event ID 13/50/51 only when the event comes from a hardware source
disk	Disk errors: I/O errors or “disk”/“blk_update_request” keywords; Event ID 7/11/15 only when the event comes from a hardware source
unknown	Hardware error that does not match known patterns

Note

An event that matches a memory/disk event ID alone (13/50/51 or 7/11/15) is only counted as a hardware error when it was actually logged by a hardware-related source — a disk controller, storage or memory driver, WHEA, and the like. Software services reuse those same event IDs: Windows’ Volume Shadow Copy service, for example, routinely logs its own event ID 13, which older versions miscounted as a memory error, unfairly dragging scores on healthy devices. Events whose message contains a hardware keyword (“machine check”, “memory”, “i/o error”, etc.) are still classified as hardware regardless of source. Previously miscounted data self-heals automatically on the next score recompute — no reinstall or data migration needed.

Hardware errors are further classified by severity:

Severity	Weight in Score
critical	-30 per event
error	-15 per event
warning	-5 per event

---

Platform Support

Windows

The Windows collector reads from the Windows Event Log via the EventLogCollector. Detected signals include:

• BSOD/Bugcheck: Event IDs 1001, 6008; messages containing “bugcheck”, “blue screen”, or “unexpected shutdown”
• Service failures: Event ID 7034; messages with “service terminated” or “service failed”
• Application hangs: Messages containing “hang” or “not responding”
• Hardware errors: WHEA events, disk errors, memory errors
• System crashes: Critical-level system events containing “crash”

Windows provides the richest reliability telemetry due to the structured Event Log system.

v0.85.0 classification improvements: Starting with the v0.85.0 agent, Windows event classification is more accurate. Routine component timeouts (such as driver or firmware response timeouts that Windows recovers from automatically) are no longer counted as application crashes. Additionally, System log entries that were previously categorized as hardware failures are now classified more precisely — generic System log noise no longer inflates the hardware error count. Fault counts will decrease for most Windows devices after updating to v0.85.0, which will raise their reliability scores.

v0.89.0 classification improvements: Memory and disk hardware errors identified by event ID alone (13/50/51, 7/11/15) now also require the event to originate from a genuine hardware source. Benign software events that reuse those IDs — most notably Volume Shadow Copy’s event 13 — no longer count as hardware errors. The fix applies server-side on the next score recompute, so previously affected scores rise without an agent update.

macOS

The macOS collector reads from system logs. Detected signals include:

• Kernel panics: Messages containing “kernel panic” or “panic(”
• Application crashes: Messages containing “application crash” or “crashed”
• Application hangs: Messages containing “hang” or “not responding”
• Service failures: launchd messages containing “exited” or “failed”
• System crashes: Critical-level system events containing “shutdown”
• Hardware errors: Messages with “i/o error” or “memory” keywords

v0.86.0 classification improvements: Breeze now filters macOS reliability data server-side. Non-hardware system-log noise (IOKit plugin chatter, App Store activity, and similar routine messages) no longer inflates the hardware-error count, and routine per-application crash reports are weighted far below a kernel panic. Expect macOS scores to rise after the server is upgraded to v0.86.0 — this applies even to devices still on older agents, because the cleanup runs on the server.

Linux

The Linux collector reads from journald/syslog. Detected signals include:

• Kernel panics: Messages containing “kernel panic”, “oops”, or “segfault”
• OOM kills: Messages containing “oom” or “out of memory”
• Service failures: systemd messages containing “failed” or “failure”
• Process hangs: Messages containing “hang”, “not responding”, or “blocked for more than”
• Hardware errors: Messages with “i/o error”, “edac”, or “mce” keywords

v0.86.0 classification improvements: Breeze now filters Linux reliability data server-side. Routine kernel chatter (USB and ACPI messages, and similar non-fault log noise) no longer inflates the hardware-error count, and routine per-application crash reports are weighted far below a kernel panic. Expect Linux scores to rise after the server is upgraded to v0.86.0 — this applies even to devices still on older agents, because the cleanup runs on the server.

Note

On all platforms, if the event log collector fails (permissions, missing log source, etc.), the reliability collector falls back to returning base metrics only (uptime and boot time) without any event data. This ensures the uptime score remains accurate even when event collection is unavailable.

---

Viewing Reliability Data

In the device list

The Devices list includes a sortable Reliability column. Each device shows a colored score badge (0–100) with a trend arrow — improving, stable, or degrading — colored by band. Click the column header to sort the fleet worst-first or best-first. Devices that don’t have a score yet (newly enrolled, or network devices) show a dash.

On the device detail page

Open a device and find the Reliability card. It is designed to explain the score, not just display it:

• All five factors as rows. Every factor (uptime, crashes, application hangs, service failures, hardware errors) is shown as a row with its points earned out of points available (e.g. “18 / 36 pts”), so you can see how the rows sum to the total score. Rows are sorted worst-first; hovering a row shows the exact figures. A weight-segmented score bar above the rows shows each factor’s share of the score at a glance.
• Humanized evidence per factor. Problem factors show a plain-language evidence line (e.g. “24 in 30d · 11 in last 7d · 3 recovered”). Where per-event data exists (service failures, hardware errors, hangs), a details link expands a drill-down listing the top offending services, hardware components, and processes with event counts and last-occurrence dates. Service failures caused by Breeze’s own agent/watchdog/helper services are annotated “from Breeze services, not scored.”
• Headline stat. Servers and always-on infrastructure show 30-day uptime as the headline stat. Workstations — where uptime carries no weight — instead show Biggest drag: the factor costing the most points and its 30-day event count, so the headline is always meaningful.
• Trend and MTBF are shown alongside the score, and the “At risk” badge (score ≤ 70) names the biggest-drag factor in its tooltip.

One action is available: Ask AI about reliability — hands the device’s reliability snapshot to the Breeze AI assistant for a plain-language diagnosis and suggested next steps.

Fleet Overview

List reliability scores for all devices in your organization, sorted worst-first by default:

GET /reliability?orgId=uuid&scoreRange=critical&trendDirection=degrading&page=1&limit=25

List Query Parameters

Parameter	Type	Description
orgId	UUID	Filter by organization
siteId	UUID	Filter by site
scoreRange	string	Filter by band: critical, poor, fair, good (also accepts legacy 0-50, 51-70, 71-85, 86-100 format)
trendDirection	string	Filter by trend: improving, stable, degrading
issueType	string	Filter by issue type: crashes, hangs, hardware, services, uptime
minScore	integer	Minimum reliability score (0-100)
maxScore	integer	Maximum reliability score (0-100)
page	integer	Page number (default 1)
limit	integer	Results per page (1-100, default 25)

The response includes a summary section with the average score, count of critical devices (score ≤ 50), and count of degrading devices:

{
  "data": [...],
  "pagination": { "total": 150, "page": 1, "limit": 25, "totalPages": 6 },
  "summary": {
    "averageScore": 78,
    "criticalDevices": 5,
    "degradingDevices": 12
  }
}

Organization Summary

Get a high-level reliability overview for an organization, including the 10 worst devices:

GET /reliability/org/:orgId/summary

The response includes:

Field	Description
devices	Total device count with reliability data
averageScore	Organization-wide average reliability score
criticalDevices	Devices with score 0-50
poorDevices	Devices with score 51-70
fairDevices	Devices with score 71-85
goodDevices	Devices with score 86-100
degradingDevices	Devices with a degrading trend
topIssues	Ranked list of most common issue types across the org
worstDevices	The 10 lowest-scoring devices with full reliability details

Single Device Detail

Get the full reliability snapshot and 30-day history for a specific device:

GET /reliability/:deviceId

The response contains two sections:

• snapshot – The current computed reliability state: overall score, all sub-scores, uptime percentages (7d/30d/90d), event counts, MTBF, trend direction and confidence, and top issues.
• history – An array of daily data points for the last 30 days, each containing sample count, max uptime seconds, crash/hang/service failure/hardware error counts, and a daily reliability estimate.

Device History

Retrieve daily reliability history for a configurable lookback window:

GET /reliability/:deviceId/history?days=90

Parameter	Type	Description
days	integer	Lookback window in days (1-365, default 90)

Each data point in the response represents one day and includes:

Field	Type	Description
date	string	Day in YYYY-MM-DD format
sampleCount	integer	Number of telemetry submissions that day
uptimeSecondsMax	integer	Highest reported uptime that day
crashCount	integer	Total crash events
hangCount	integer	Total application hangs
serviceFailureCount	integer	Total service failures
hardwareErrorCount	integer	Total hardware errors
reliabilityEstimate	integer	Estimated reliability score for that day (0-100)

---

AI Integration

The Breeze AI assistant can query device reliability data through its built-in tool system. The query_device_reliability tool allows natural language questions about fleet reliability to be answered with real data.

The AI tool supports the same filters as the list API: organization, score range, trend direction, issue type, and score bounds. When invoked, it returns the same paginated results with a summary section, allowing the AI to answer questions like:

• “Which devices have the worst reliability scores?”
• “How many devices are in a degrading trend?”
• “Show me all devices with hardware errors in the last 30 days”
• “What is the average reliability score for Contoso?”

Note

AI reliability queries are subject to the same organization-scoped data isolation as direct API access. The AI can only access reliability data for organizations the authenticated user has permission to view.

---

API Reference

Fleet Reliability

Method	Path	Description
GET	/reliability	List device reliability scores with filtering and pagination
GET	/reliability/org/:orgId/summary	Organization-level reliability summary with worst devices
GET	/reliability/:deviceId	Full reliability snapshot and 30-day history for a device
GET	/reliability/:deviceId/history	Daily reliability history with configurable lookback (?days=)

Agent Ingestion

Method	Path	Description
POST	/agents/:id/reliability	Submit reliability metrics from the agent (agent auth required)

Note

All reliability query endpoints require authentication and one of these scopes: organization, partner, or system. Organization-scoped tokens are automatically restricted to their own organization’s data. The agent ingestion endpoint uses agent bearer token authentication.

---

Troubleshooting

No reliability data for a device. Reliability data appears after the agent has submitted at least one telemetry payload via POST /agents/:id/reliability. Confirm the agent is online and the heartbeat cycle is running. The agent includes a 24-hour initial lookback on first collection, so the first submission should include recent events. If the device exists but has no reliability snapshot, the scoring computation may not have run yet – check BullMQ worker status.

Reliability score seems too low despite no visible issues. The score is a composite of five factors with different weights. A device can have a low score due to a single factor being severely penalized. Use GET /reliability/:deviceId to inspect the individual sub-scores (uptimeScore, crashScore, hangScore, serviceFailureScore, hardwareErrorScore) and identify which factor is dragging the score down. For example, a 90% uptime over 90 days produces an uptime sub-score of 0, which alone would reduce the overall score by up to 30 points.

A laptop’s reliability score looks fine even though it’s powered off a lot. This is expected. Laptops and desktop workstations are not scored on uptime — they’re expected to sleep and shut down — so normal power-off behavior does not lower their reliability score. Only servers and infrastructure are scored on uptime.

Trend direction shows stable with low confidence. Trend computation requires at least 3 days of data and achieves full confidence at 14+ days. If the device was recently enrolled or has sparse telemetry, the trend will default to stable with trendConfidence: 0. Allow the device to accumulate more history before relying on trend data.

Agent event collection failing on specific platform. On all platforms, if the event log collector encounters an error, the reliability collector gracefully falls back to base metrics (uptime and boot time only). Check agent logs for warnings like “reliability event log collection failed, returning base metrics only”. Common causes include insufficient permissions to read system event logs, missing log sources, or the event log service being stopped.

MTBF showing null. MTBF is only computed when there is at least one failure event (crash, hang, service failure, or hardware error) in the 90-day window AND the device has positive operating hours. A device with zero failures has no meaningful MTBF – this is the ideal state. A device with zero uptime data also produces null MTBF.

Score not updating after new events arrive. Score computation is enqueued via BullMQ after each telemetry submission. If the queue worker is down, the system falls back to inline computation, but this fallback may fail silently if the database is under load. Check BullMQ dashboard for failed or stalled device-reliability-computation jobs. The computedAt timestamp on the reliability snapshot indicates when the score was last calculated.

Organization summary showing stale data. The org summary endpoint computes results in real time from the device_reliability table. If individual device scores have not been recomputed recently (check computedAt), the summary reflects outdated data. Trigger a fleet-wide recomputation by ensuring all agents are submitting telemetry and the reliability worker is processing jobs.