mat/homeassistant-comwatt
1
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Activity Actions

🐛 fix: skip QUANTITY/HOUR reconciliation buckets with untrusted units #63

Merged
mat merged 1 commit from fix/reconcile-skip-unreliable into main 2026-07-15 15:51:03 +00:00
Conversation 0 Commits 1 Files changed 2 +160 -12
mat commented 2026-07-14 19:11:55 +00:00
Owner
Copy link

Context

The hourly energy reconciliation in _fetch_device_metrics snaps the live ∫W·dt total against the server's QUANTITY/HOUR bucket once per hour. It assumed the server value was in Wh. It is not, consistently: the Comwatt endpoint returns mixed units per device with no unit field anywhere in the API response or device metadata. A grid-injection child device returns kWh (server val=0.9 for an hour averaging ~900 W), so the reconciliation did live_total += 0.9 - 900 ≈ -899 and snapped ~900 Wh down every hour — a hundred-Wh jump on sensor.echange_reseau_injection_total_energy (and similarly on soutirage). This is the latent QUANTITY-unit bug the handoff describes, surfaced by the streaming reconciliation.

Investigation (live data, documented in the reconciliation docstring):

  • Bucket labeling is START-of-hour (confirmed: a device that ran only near the end of one hour carried that hour's label, not the previous one). The existing hour keying is correct — unchanged.
  • No unit field exists and deviceKind / threePhase / global do not predict the unit (solar is threePhase=True & Wh; injection is threePhase=True & kWh; batteries are threePhase=False & anomalous). No static kind map works.
  • The only reliable Wh reference is the live ∫W·dt in live_by_hour.

This is slice 1 of 2. Slice 1 stops the bleeding: buckets whose unit cannot be trusted are skipped (live total left untouched, high-water mark still advances). Slice 2 (stacked on this) will add kWh→Wh conversion so grid devices get a bounded drift correction too.

Impact

  • No *_total_energy sensor makes a hundred-Wh jump at the hourly snap anymore. The reconciliation either applies a bounded drift correction (Wh devices, ratio to live ≈ 1.0) or skips (kWh / no live reference / incoherent ratio). The live ∫W·dt path (Slice 4) is untouched and remains the real-time source of truth.
  • The fallback branch (live_total is None → state.total_wh += val, the pre-streaming path) is byte-for-byte unchanged — its latent unit bug is out of scope here.
  • Scope: only custom_components/comwatt/coordinator.py and tests/test_coordinator.py.

Considerations

  • Why skip rather than convert kWh here? Detecting kWh needs a live ∫ to compare against, which only exists in the reconciliation branch — but the conversion adds a second ratio band and more failure modes. Splitting lets slice 1 ship a guaranteed-safe "stop the jumps" fix immediately, with the kWh→Wh enhancement verified separately on the box in slice 2.
  • Skipped buckets still advance last_bucket_ts, so a bogus night value (solar/injection returning non-zero when idle) or an anomalous virtual-device aggregation (EV one-off, battery) is never reconsidered. This is intended: re-evaluating would just re-skip, and holding the high-water mark back would also stall the stale-live_by_hour prune.
  • Thresholds: _RECONCILE_MIN_LIVE_WH = 10.0 (below this, no reliable reference), Wh band ratio ∈ [0.5, 2.0] (drift tolerance). Tunable constants, kept narrow to stay conservative.

Tests

  • 3 new: kWh bucket is skipped (no ~900 Wh jump), no-live-reference bucket is skipped (+HWM advances), incoherent-ratio bucket is skipped (+HWM advances).
  • 1 existing test updated (test_reconcile_prunes_stale_live_by_hour) to give the reconciled hour a live reference, preserving its prune intent.
  • All other existing reconcile tests stay green unchanged (their Wh mocks land in-band).
  • 77/77 passing, ruff clean, mypy clean.
## Context The hourly energy reconciliation in `_fetch_device_metrics` snaps the live ∫W·dt total against the server's `QUANTITY/HOUR` bucket once per hour. It assumed the server value was in Wh. It is not, consistently: the Comwatt endpoint returns **mixed units per device** with **no unit field** anywhere in the API response or device metadata. A grid-injection child device returns kWh (server `val=0.9` for an hour averaging ~900 W), so the reconciliation did `live_total += 0.9 - 900 ≈ -899` and snapped ~900 Wh down every hour — a hundred-Wh jump on `sensor.echange_reseau_injection_total_energy` (and similarly on soutirage). This is the latent `QUANTITY`-unit bug the handoff describes, surfaced by the streaming reconciliation. Investigation (live data, documented in the reconciliation docstring): - **Bucket labeling is START-of-hour** (confirmed: a device that ran only near the end of one hour carried that hour's label, not the previous one). The existing `hour` keying is correct — unchanged. - **No unit field exists** and `deviceKind` / `threePhase` / `global` do **not** predict the unit (solar is `threePhase=True` & Wh; injection is `threePhase=True` & kWh; batteries are `threePhase=False` & anomalous). No static kind map works. - The only reliable Wh reference is the **live ∫W·dt** in `live_by_hour`. This is **slice 1 of 2**. Slice 1 stops the bleeding: buckets whose unit cannot be trusted are **skipped** (live total left untouched, high-water mark still advances). Slice 2 (stacked on this) will add kWh→Wh conversion so grid devices get a bounded drift correction too. ## Impact - No `*_total_energy` sensor makes a hundred-Wh jump at the hourly snap anymore. The reconciliation either applies a bounded drift correction (Wh devices, ratio to live ≈ 1.0) or skips (kWh / no live reference / incoherent ratio). The live ∫W·dt path (Slice 4) is untouched and remains the real-time source of truth. - The fallback branch (`live_total is None` → `state.total_wh += val`, the pre-streaming path) is byte-for-byte unchanged — its latent unit bug is out of scope here. - Scope: only `custom_components/comwatt/coordinator.py` and `tests/test_coordinator.py`. ## Considerations - **Why skip rather than convert kWh here?** Detecting kWh needs a live ∫ to compare against, which only exists in the reconciliation branch — but the conversion adds a second ratio band and more failure modes. Splitting lets slice 1 ship a guaranteed-safe "stop the jumps" fix immediately, with the kWh→Wh enhancement verified separately on the box in slice 2. - **Skipped buckets still advance `last_bucket_ts`**, so a bogus night value (solar/injection returning non-zero when idle) or an anomalous virtual-device aggregation (EV one-off, battery) is never reconsidered. This is intended: re-evaluating would just re-skip, and holding the high-water mark back would also stall the stale-`live_by_hour` prune. - Thresholds: `_RECONCILE_MIN_LIVE_WH = 10.0` (below this, no reliable reference), Wh band `ratio ∈ [0.5, 2.0]` (drift tolerance). Tunable constants, kept narrow to stay conservative. ## Tests - 3 new: kWh bucket is skipped (no ~900 Wh jump), no-live-reference bucket is skipped (+HWM advances), incoherent-ratio bucket is skipped (+HWM advances). - 1 existing test updated (`test_reconcile_prunes_stale_live_by_hour`) to give the reconciled hour a live reference, preserving its prune intent. - All other existing reconcile tests stay green unchanged (their Wh mocks land in-band). - `77/77 passing`, `ruff` clean, `mypy` clean.
🐛 fix: skip QUANTITY/HOUR buckets with untrusted units in reconciliation
All checks were successful
Validate / lint-ruff (push) Successful in 7s
Details
Validate / test-pytest (push) Successful in 3m1s
Details
Validate / type-check-mypy (push) Successful in 3m7s
Details
Validate / lint-ruff (pull_request) Successful in 7s
Details
Validate / test-pytest (pull_request) Successful in 3m1s
Details
Validate / type-check-mypy (pull_request) Successful in 3m9s
Details
d9bb299827 
The Comwatt QUANTITY/HOUR endpoint returns mixed units per device with no
unit field anywhere in the API. Reconciliation assumed Wh, so a kWh device
(e.g. grid injection, val~0.9 for an ~900 Wh hour) snapped the live total
down by ~900 Wh every hour.

Infer the unit from the ratio of the server value to the live integral W*dt
for the same hour: a Wh device lands near ratio 1.0. Skip buckets whose unit
can't be trusted - no live reference (live ~ 0) or an incoherent ratio - so
the live accumulator stays authoritative and the high-water mark still
advances so they are not reconsidered. Converting kWh to Wh is a later,
out-of-scope slice; this slice only skips.
mat changed title from WIP: 🐛 fix: skip QUANTITY/HOUR reconciliation buckets with untrusted units to 🐛 fix: skip QUANTITY/HOUR reconciliation buckets with untrusted units 2026-07-15 15:50:05 +00:00
mat merged commit 179322aa5d into main 2026-07-15 15:51:03 +00:00
mat deleted branch fix/reconcile-skip-unreliable 2026-07-15 15:51:04 +00:00
mat referenced this pull request from a commit 2026-07-25 16:37:12 +00:00
:bookmark: Release v0.7.0
mat referenced this pull request from a commit 2026-07-25 16:40:44 +00:00
:bookmark: Release v0.7.0
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
mat/homeassistant-comwatt!63
No description provided.