Create MQTT sensor for presence detection of BLE devices

e-raser · August 5, 2024, 10:36pm

I have two OMGs and one TheengsGateway I want to use for presence detection for a specific BLE device.

For this I have this sensor (multiple times of course):

    - name: "E-Bike (OMG 1)"
      unique_id: xyzxyzxyz
      state_topic: "OpenMQTTGateway/presence/ESP_OMG_1"
      value_template: "{{ 'home' if value_json.id == 'XX:XX:XX:XX:XX:XX' else 'not_home' }}"
      #expire_after: 10
      icon: mdi:bicycle-electric
      device:
        identifiers: XYZ
      entity_category: diagnostic

While this seems (!) to basically work, it seems like the presence topic for OMG and TheengsGateway is very different to the one I used before in another ESP based project (where the state topic contained a list of all devices currenctly around, so as long as the device was active it was part of the state topic, that way it was discovered “home” as long as it was nearby):

the topic only stores the latest device found, for a very short period of time (depending on the next new/strongest one detected).

Multiple discovery doesn’t matter as I have one collection sensor (contains OMG 1 + OMG 2 + TG + WiFi based sensor + previous Bluetooth sensors).

In short: how to use the OMG/TheengsGateway provided presence topics for doing some half-serious (no device_tracker entities needed at the end, but reliable discovery and ideally signaling how long the device is around) device tracking?

I know you natively implemented this in a very professional way (meaning in the end there’s a autodiscovery provided device_tracker entity in Home Assistant, just perfect) for many devices (BM2, fitness trackers etc.). Would this be “the way to go to do it fully professional”?

I’m currently looking at the best way to go. Focus:

reliability
speed (I can’t wait months to have this natively implemented and shipped in a stable OMG/TG release, that’s why I tried to use what I can with the existing MQTT topics)

Thanks a lot for your help on this. Unfortunately I found that doing device tracking on my own is quite complicated with bluetooth devices and the way the presence topics are build (ultra short-term lifetime/no list as I was used to).

DigiH · August 6, 2024, 4:51pm

How about just creating and publishing your own device tracker discovery message for the mentioned BLE device (E-Bike??)?

Then you would have a proper BLE device tracker entry for the bike.

Do you think you can cobble this together yourself by looking at the topic and payload of your other already existing device tracker discovery entries?

Which E-Bike is it, a very popular one, and is the BLE integration standard with it?

Could you also post some sample BLE messages you receive from it, with advertising data turned on? As this might be interesting to others as well, and the advertising data might include some decodable data, like battery level or such?

Does the bike have a native app showing some received information?

The actual presence option in OMG and TGW is only really applicable for room presence, but device trackers are the way to go if you just want a home/away presence detection.

e-raser · September 1, 2024, 4:39pm

How to achieve this?

It’s just the on-board computer (BOSCH), a Nyon 2 attached specifically to that E-Bike. It just “sends” BLE messages.
See Nyon: der Bordcomputer mit Navigation für Elektrofahrräder

Yes it does: ‎Bosch eBike Connect im App Store
Maybe that’s inside the manufacturerdata part below? At least it is not clear text. But my smartphone app is paired, the OMGs/TheengsGateway obviously not.

That’s exactly what I’m looking for.

presence:

{"id":"XX:XX:XX:XX:XX:XX","mac_type":0,"adv_type":0,"name":"Nyon-XXX","manufacturerdata":"a60210eb002500","rssi":-73,"txpower":12,"distance":4.756481}

BTtoMQTT:

{"id":"XX:XX:XX:XX:XX:XX","mac_type":0,"adv_type":0,"name":"Nyon-XXX","manufacturerdata":"a60210eb002500","rssi":-73,"txpower":12,"distance":4.756481}

I can’t use the BTtoMQTT one as this seems to be kind of persistent (not removed once the device goes away/off) and the presence one only contains the latest fetched one as mentioned in the OP.

Can you help me on this / work with the information provided @DigiH ?

DigiH · September 2, 2024, 4:19am

e-raser:

Can you help me on this / work with the information provided @DigiH ?

BTtoMQTT:
{"id":"XX:XX:XX:XX:XX:XX","mac_type":0,"adv_type":0,"name":"Nyon-XXX","manufacturerdata":"a60210eb002500","rssi":-73,"txpower":12,"distance":4.756481}

This was the information which was needed to allow me to add a device tracker decoder for the Bosch Nyon, and assuming this advertising data is sent out in regular intervals.

You should be able to test and use this additional decoder with your ESP32 OpenMQTTGateway gateway by installing the latest development version from

e-raser · September 2, 2024, 7:35am

Oh great! Thanks. I‘ll give it a try asap.

Please note I masked things like the MAC and the latter part of the name (usually „Nyon-XXX“ contains the last 3 characters of the MAC address), I forgot to mention this in the previous post. Not sure if this is relevant for the decoder - just to be sure.

DigiH · September 2, 2024, 2:26pm

I already assumed as much, with all the Xs It’s fine, they were not relevant for creating the decoder.

What you could monitor however is if the manufacturerdata changes at all over time, to see if there is any additional data encoded in it, like possibly the accumulated distance cycled or such.

e-raser · September 2, 2024, 6:41pm

I want to avoid using the web installer. Instead I tried to upgrade from the OMGs web interface (/up?):

Unfortunately, it stays on stable 1.7.0. Tried several times, after restart still 1.7.0 is flashed. Do you have a OTA URL for the dev version I shall test?

DigiH · September 2, 2024, 6:46pm

I think that might be due to the certificate change.

@1technophile might have a direct OTA URL for you.

Otherwise, using the web installer and unselecting Erase Flash should keep your current settings, unless there have been changes which require a new set up for the current development version.

e-raser · September 2, 2024, 7:03pm

OK. I’ll wait for the OTA URL then. It’s just so much easier flashing/updating them remotely, considering where the ESP(s) are physically located.

To be ready to test after switching to dev channel: what is needed to provoke the “catch” and create the device tracker? I remember back in the days when we set this up for the BM2 I had tough times achieving the creation of the device_tracker entity and spent a lot of unnecessary time then.

I guess I need to:

Enable “SYS: Auto discovery” temporarily
Enable “BT: Publish advertisement data” temporarily
Speed things up maybe by enabling “BT: Force scan” once?

Always enabled FYI: “BT: Publish HASS presence” as well as…

1technophile · September 2, 2024, 7:18pm

You still need to do an upload from :

To get the new certificate, it will not work to enter the new url here due to the Github certificate change.

e-raser · September 2, 2024, 7:21pm

So I need to use (Option 1) Upload from the web | Theengs OpenMQTTGateway DEVELOPMENT SHA:c5b9bc TEST ONLY, no remote update?

Therefore: how could I switch back to latest stable (1.7.0 release)? Again a local web flash (but this time using the stable web flasher (Option 1) Upload from the web | OpenMQTTGateway v1.7.0)?

DigiH · September 2, 2024, 7:22pm

And then press the SYS: Restart Gateway button. After the restart, once the device Tracker for the Bosch Nyon has been auto discovered you can disable SYS: Auto discovery again.

BT: Force scan doesn’t really help here, nor is BT: Publish HASS presence required to be enabled if you only want to use Home/Away device trackers.

You can keep BT: Publish advertisement data enabled for a few days, to see if the manufacturerdata changes at all, especially after having used your eBike for a ride. You could also tell us what battery level the Nyon shows in the app, as this might also be encoded in the manufacturerdata.

If the manufacturerdata doesn’t change at all just let us know here, and you can dsable BT: Publish advertisement data again. If iot does change we should investigate some more to see which data might be encoded there, to create a more detailed decoder for the Nyon.

1technophile · September 2, 2024, 7:34pm

Yes, no remote update.

You will be able to switch back easily as all the 1.7.0 binaries have been copied to the new OTA host.
But if you switch back you will still need to do a manual upload for the v1.8.0

DigiH · September 6, 2024, 5:41pm

@e-raser

Did you get a chance to test this with the development build at all?

Any feedback?

e-raser · September 6, 2024, 10:04pm

Not yet, scheduled for Sunday. A bit difficult to do remotely when you need to flash locally, so it took/takes longer than I was initially hoping for. I‘ll turn back.

e-raser · October 19, 2024, 5:06pm

So here I am - a bit late to the party, but I finally managed to flash one OMG to version "f86d89".

After doing exactly what you instructed to, nothing happens. Meaning: the Nyon is not showing up at all in the BTtoMQTT (also not on the web interface’s console) with BT: Force scan and SYS: Auto discovery and SYS: Restart gateway several times in different orders.

1. Detection time

Update: after powering the Nyon up a few times, it got discovered. Device tracker topic also created immediately. A bit strange as I tested how long it takes for the OMG to catch bluetooth signal of the Nyon: no idea when it happens after it is “coming home” (after disabling and enabling BT on the Nyon (like a leaving home / coming home simulation), but interestingly the MQTT topic timestamp changes once the Nyon gets turned off. Does that make any sense?

More testing (few / latest examples):

51 seconds to switch from {"id":"XX:XX:XX:XX:XX:XX","state":"offline"} to (see below for example) with the Nyon sitting 30 centimeters away from the OMG ESP. That’s a lot of time. I think even Wi-Fi might be quicker. Therefore I am really wondering how device tracking turns out to work in real-world everyday life…
25 seconds for the OMG (right next to the Nyon) while Wi-Fi took 20 seconds and the former Bluetooth discovery ESPs (another project) 2 seconds. So indeed not faster than Wi-Fi and much slower than the former project. Hmm
00 former bluetooth project + 15 seconds later OMGs + 27 seconds later Wi-Fi. Better (because quicker than Wi-Fi) but not rapidly fast.
30 seconds Wi-Fi, 49 seconds later () OMG…

2. Regarding manufacturerdata:

{"id":"XX:XX:XX:XX:XX:XX","mac_type":0,"adv_type":0,"name":"Nyon-XXX","manufacturerdata":"a60210eb002500","rssi":-40,"txpower":12,"distance":0.020515,"brand":"Bosch","model":"Nyon","model_id":"BOSCHNYON","type":"TRACK","acts":true,"track":true,"device":"Bosch Nyon Tracker","topic":"OpenMQTTGateway/presence/ESP_OMG_XXX1"}

I checked with iOS app “LightBlue” which showed some device information and bluetooth specific fields labeled with Properties: Indicate|Write|Notify|Write without Response|Read. The read property gives 0x022E02 which doesn’t match any notable number converted to dec. Also the manufacturerdata did not change since the beginning of September, so I think nothing (like hoping for total trip or sth. like that) relevant (nothing dynamic) is stored there.

3. HA entities

With the dev firmware I noticed the update entity of the OMG changed.
Stable (1.7.0) icon:
grafik

auto_update: false
installed_version: v1.7.0
in_progress: false
latest_version: v1.7.0
release_summary: 
release_url: https://github.com/1technophile/OpenMQTTGateway/releases/tag/v1.7.0
skipped_version: null
title: OpenMQTTGateway
entity_picture: https://github.com/1technophile/OpenMQTTGateway/raw/development/docs/img/Openmqttgateway_logo_mini_margins.png
friendly_name: ESP OMG XXX2 Firmware Update
supported_features: 1

Dev one (default icon):
grafik

auto_update: false
installed_version: v1.7.0
in_progress: false
latest_version: v1.7.0
release_summary: null
release_url: null
skipped_version: null
title: null
entity_picture: https://brands.home-assistant.io/_/mqtt/icon.png
friendly_name: ESP OMG XXX1 Firmware Update
supported_features: 1

Interestingly it still reports as installed_version: v1.7.0 next to the icon “issue”.

Updated the 2nd OMG ESP now too (device tracker HA topic seems to update the "via_device" fine so both OMGs should update the device tracker topic. Will check how device tracking works out in the next days especially for CH/LH. Hopefully all the other device tracking things work unchanged, too. So far +1: for the BOSCH Nyon device tracker! Looking forward to see that in the next stable release of OMG

DigiH · October 21, 2024, 3:54pm

With you having used several device trackers with OMG, I’m sure by now you now know the parameters defining the recognition time. If not reading through this other thread from this point onwards might help.

Thanks, so with the unchanging manufacturerdata it seems that currently the Nyon is only usable as a device tracker.

AFAIK it is a known issue with the firmware update icon.

e-raser · November 2, 2024, 9:03pm

OK. So far I tend to say it is working somehow reliable. Have not had many changes to test it in everyday life yet, but so far rather positive. So:

e-raser · November 16, 2024, 10:55pm

As my OMGs are already on a dev release (version f86d89) and I already did the manual flash for that dev version, following Over the Air server change in latest Release OpenMQTTGateway v1.8.0 · 1technophile/OpenMQTTGateway · GitHub is not necessary once again, isn’t it?

It’s just that all OMGs did not discover the new 1.8.0 release yet, so just wondering.

I’m so curious to experience

Tracker Synchronization: @DigiH introduced a powerful enhancement for tracker synchronization across multiple gateways. Now, “away” messages are only triggered if no gateways detect the tracker, ensuring more accurate notifications.

by the way!

e-raser · November 25, 2024, 4:56pm

@DigiH my OMGs still did not discover the stable 1.8.0 release.

They still show up as 1.7.0 (but of course run the dev version installed back then).

How to easily upgrade them to latest 1.8.0?