MQTT integration errors in Home Assistant since HA 2023.5 (wrong payloads)

Since updating Home Assistant to > 2023.4 the entities provided by OMG through MQTT create plenty of errors like these:

2024-07-19 21:06:15.263 ERROR (MainThread) [homeassistant.components.mqtt.models] Exception raised when updating state of sensor.xiaomi_smart_band_7_mac, topic: 'OpenMQTTGateway/ESP_OMG_ABC/BTtoMQTT/XXXXXXXXX' with payload: b'{"id":"XX:XX:XX:XX:XX:XX","rssi":-98,"distance":45.0994,"brand":"Xiaomi/Amazfit","model":"Mi Band/Smart Watch","model_id":"MB/SW","type":"BODY","device":"Xiaomi/Amazfit Tracker","mac":"XX:XX:XX:XX:XX:XX"}'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 581, in state
    numerical_value = int(value)
                      ^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'XX:XX:XX:XX:XX:XX'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/models.py", line 270, in process_write_state_requests
    entity.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 654, in _async_write_ha_state
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 596, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 585, in state
    raise ValueError(
ValueError: Sensor sensor.xiaomi_smart_band_7_mac has device class 'None', state class 'measurement' unit 'None' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'XX:XX:XX:XX:XX:XX' ()
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 585, in state
    raise ValueError(
ValueError: Sensor sensor.xiaomi_smart_band_7_mac has device class 'None', state class 'measurement' unit 'None' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'XX:XX:XX:XX:XX:XX' ()

Therefore sensors are unknown:

This is due to more strict handlings of entities in HA Core:
Following other cases like MQTT Discovery, unit_of_measure · Issue #93071 · home-assistant/core · GitHub or homeassistant.components.mqtt.models · Issue #93059 · home-assistant/core · GitHub it seems the sensor attributes are sent wrong.

How can one run into such an issue when upgrading to a release after more than one year…

I tried to play a bit with setting the UoM and state_class manually using customize.yaml, unfortunately it made no difference. It is configured wrongly before already looking at the MQTT payload:

state_topic: +/+/BTtoMQTT/XXXXXXXXXXXX
name: MB/SW-mac
unique_id: XXXXXXXXXXXXX-mac
value_template: '{{ value_json.mac | is_defined }}'
state_class: measurement
device:
  identifiers:
    - XXXXXXXXXXXX
  connections:
    - - mac
      - XXXXXXXXXXXX
  manufacturer: Xiaomi/Amazfit
  model: MB/SW
  name: Mi Band/Smart Watch-XXXXXX
  via_device: ESP_OMG_XYZ
platform: mqtt

So can you please take care and fix this in OMG so no rubbish data arrives at MQTT and in HA anymore?

Which version of OMG do you have ?

All OMGs currently use:

version "v1.7.0"

Could you search for -mac in MQTT explorer, remove the topics and restart your ESP32s.
image

And see if the topics ending with -mac are recreated.

Unfortunately, no. After ESP restarts I waited a bit, also performed a “BT: force scan” and waited a bit again.

:white_check_mark: Gets updated: device_tracker

:x: Deleted from MQTT explorer but not created again: -mac

:question: Also unknown but not deleted and not sure if the same behavior like with the mac:

  • steps
  • act_bpm
  • device

This seems to be the expected to don’t have sensors for the mac. Do you still see errors in the logs. for Sensor sensor.xiaomi_smart_band_7_mac ?

No, of course not: no sensor in HA with wrong configuration, no more complaints from HA about it.

Wait: why did they exist? Have they been removed? Or why/when have they been added at all? I don’t get it.

Your statement is a bit confusing tbh :slight_smile:

MAC has already been deleted, still left (with two more unknown) are:

They were added as sensors in the decoder to enable the Apple device to use the sensor’s MAC Address (for the Theengs App). They may have come from a prior or a developed version of OMG. And as the broker retain the message they were let there.

Did you ever had the BPM and steps with this tracker ?

Which Apple device?

I don’t recall if they were ever shown for the series 7. I think it was for the series 4. Series 8 not discovered by OMG at all but that‘s another thing.

  1. So should I remove all those topics (steps, activity/bpm/, device) so only the device tracker remains?
  2. As I noticed the device tracker entity‘s attribute still shows „gps“ instead of „bluetooth_le“ I‘d like to also delete the device tracker topic once. It should be recreated automatically, correct?

You can delete them; they are recreated automatically if the auto-discovery is activated.

Yes you can

Iphone, Mac, Ipad

I deleted all those topics. Now I’m heavily screwed:

I restarted the ESPs and then reloaded the MQTT integration.

All that spans to three issues now:

1) Sensor data back and still unknown

Now the device (Xiaomi Smart Band 7) came back as default “Mi Band/Smart Watch-XXXXXX” with these sensors:

So all those entities (and topics) come back!

  • -steps
  • -act_bpm
  • -device

Why do you recommend me to delete stuff if that’s not a permanent solution? Restarts and reloads happen from time to time. Only the -mac did not come back so far.

How can I fix this? I tend to disable those entities in Home Assistant instead. Seems like you can’t fix it on OMG/TheengsGateway/MQTT side, once again…

According to what I understood from HA side you would just need to fix the sensor definition data (unit_of_measurement and state_class) as shown in the payload example of the first post here.

2) Source type for one device tracker still wrong

And second bad news: the source type for the device tracker entity is still wrong.
MQTT:

Home Assistant:

3) Device Tracker topic for one BM2 not created anymore at all

For one BM2 device the -volt and -batt topics still exist (I did not delete them), but after deletion of one -tracker topic of a BM2 device, this topic is NOT created anymore. Which means I don’t have this important device tracker entity anymore in HA.

Here’s the /OpenMQTTGateway/presence/ESP_OMG_XYZ topic:

{"id":"XX:XX:XX:XX:XX:XX","name":"ZX-1689","rssi":-80,"txpower":0,"distance":9.521558,"brand":"GENERIC","model":"BM2 Battery Monitor","model_id":"BM2","type":"BATT","batt":100,"device":"BM2 Tracker"}

Unfortunately, no more /homeassistant/device_tracker/XXXXXXXXXXXX-tracker topic is created anymore now. Gosh… :sob:

HOW TO FIX THIS ASAP please? :frowning: :frowning: :frowning:
(just to make sure: fixing issue 3 is the most important one currently)


edit:

  • #3 is fixed (I had to enable the “SYS: auto discovery” switch for a short moment, completely forgot about that switch and it seems it is necessary), unfortunately here the source_type still is wrong, even it is correctly in the MQTT topic.

grafik

  • #2 is waiting for a new release, please see now almost one year old issue (it’s just one single blank space hahaaaaaa :slight_smile: ):

:exclamation::exclamation::exclamation:

  • #1 I disabled those entities, I don’t care anymore today what OMG sends to my MQTT broker and what rubbish is stored there, at least HA is clean in the end.

If we can’t fix it once again, maybe we should stop here and avoid spending time on this ;-).

There will be one by the end of the year at max.

OK so removing a single blank space (Presence detection for several devices: wrong source_type attribute in Home Assistant · Issue #1755 · 1technophile/OpenMQTTGateway · GitHub) seems to take more than one year, impressive :wink:

But that’s another topic (OMG/TheengsGateway release management), this topic was about completely different things (which I sorted out FOR ME mostly as written in the previous post).

Honest thanks for your great work, I really enjoy TG + OMG.