Presence Detection, and how does all this fit together?

I run Home Assistant on an Unraid VM. I use ESPHome for some things but stumbled on OpenMQTTGateway and it was the perfect solution for connecting several Thermometers. For My Hot Tub, Grill, etc. I had a few ESP32 Devices around which I loaded with OpenMQTTGateway and placed them around my house. All seems to be working great for the Themometers mentioned above.

I noticed that OpenMQTTGateway supported presence. I have a BlueCharm Device and a Tile device. Not sure how to get these working to display presence? Tried Several things and I can see the Tile Device Away or Home but not the BlueCharm Device. The Home Assistant device displays x y z movement information but not any home or away. I also assumed presence would be indicated at the OpenMQTTGateway Sensor Level in Home Assistant (meaning indicating the room or area in the house) not just home or away for the house?

Then I noticed on line the Theengs Gateway Integration for Home Assistant which I installed assuming it would talk to the indivicual gateways and improve the Presence. However, using MQTT Explorer the only topic I ever see for the home>TheengsGateway is LWT=Online? Nothing else so I am not really sure it is doing anything other than use CPU Cycles???

So, not really sure I understand how presence works and what to expect?? And I clearly do not understand the overall view and operational abilities of the combination of OpenMQTTGateway(s) to the TheengsGateway? Sorry for being a stupid noob here but I am kinda confused… Any help or clarification would be awesome…

Hi @Shesakillatwo

Lots of questions, lots of confusion, which I can understand when starting off with OpenMQTTGateway and Theengs Gateway.

Let me try and hopefully make a few things clearer.

• Theengs Gateway is basically much the same as an OpenMQTTGateway BLE gateway, but not running on an ESP32 but a Linux/RPi or Windows computer, as a stand alone installation, or as a Home Assistant Add-on, which you seem to be using.
This computer however needs a Bluetooth adapter, so that Theengs Gateway (Add-on) can actually received any BLE broadcasts to then decode them.
Does the computer you are running Home Assistant on with the Theengs Gateway Add-on have a Bluetooth adapter?
If yes, and you are still only getting the LWT message there might be a problem with Theengs Gateway correctly being able to access the BT adapter, which might need to be explicitly defined in the Configuration of the Theengs gateway Add-on in your case.
If the computer you are using HA on does not have Bluetooth capabilities, then Theengs Gateway will not be able to receive any BLE broadcasts on its ow, but …

• Even without any Bluetooth adapter to decode any Bluetooth broadcasts on its own, Theengs Gateway can also be used to act as a central decoding hub for received Bluetooth broadcasts which have been picked up by different ESP32s with OpenMQTTGateway installed. This allows to cover a large area with several OMG ESP32s, but lewtting one single Theengs Gateway do all the decoding work.
For this however all the ESP32s will need to have the esp32dev-ble-mqtt-undecoded binary installed - as the description for it says

BLE gateway with the decoding offloaded to Theengs Gateway

Both the above options allow you to recognise device tracker general Home/Away presence for all compatible devices listed as Presence Tracker in our compatibility list

which works fine for your Tile, but why it doesn’t work for your BlueCharm would need to be investigated, assuming it is one of the listed BlueCharm/KKM Beacon 08/04P/021 - K8/K4/K21.

Could you turn on Advertisement and advanced data in one of your ESP32 OMG gateway settings in HA and then post a message with all the data for the BlueCharm here for us to see what might be the case of the missing device tracker recognition for it?

Also please try turning On Discovery again on this or all other ESP32 OMG gateway and restart them, as discovery is automatically turned off after an initial 30 minutes and might have been the issue of the missing correct device tracker discovery of the BlueCharm.

I have specifically left out the distance/room presence detection here, as I do not know HA well enough to know if and how it integrates selecting the gateway with the closest distance to be the location the trakcer is at any particular time. Best for someone else with more knowledge of the distance/room presence tracking to answer this.

Hope this helps in understanding all things OpenMQTTGateway and Theengs Gateway a bit better.

P.S. Just saw your reply on the HA forum as well, best to keep any further discussions here in one place :slight_smile:

Just found out from @1technophile

The Home Assistant integration you are looking for to work with all the separate OpenMQTTGateways you have is

You just need to make sure that each of your gateways has the BT: Publish HASS presence option turned on, so that they publish on the presence topic with the distance property.

Using all the OpenMQTTGateways like that, having the additional Theengs Gateway installed on your HA system is not really required.

Thanks so much and I will digest your info and respond later. I will admit that I had originally loaded the esp32-m5attom-lite to my devices and got them working and could not find the esp32dev-ble-mqtt-undecoded binary initially, but found it this morning in the dropdown list. Stupid me. I loaded it on an m5atom and noted that it does indeed communicate with the TheengsGateway now, and I see a presence topic under home which I did not see until now. Not sure why I had not seen it as as I expected it with the individual OMG devices. Anyway, I will play around with the various settings later and return with my next round of dumb questions once I am totally lost! Have a great day and thanks for your efforts…

After playing a bit with the OpenMQTTGateway ESP 32 devices and I have figured most things out. I had enabled the “Publish Only Sensors” which I think was preventing the Presence from showing up from my OMG_## devices. I also looked at the mqtt_room from Home Assistant and after fighting with that for a bit got it working so as I walk through the house the value in Home Assistant updates to the nearest OMG_## device along with a distance. I think I am good for now. Thank you so much for all your assistance and guidance through this phase of the adventure.

One small suggestion for a tweak. I had enabled the “Publish Only Sensors” early on to eliminate much of the traffic I was seeing in MQTT Explorer. Through all of that I was able to see the Tile Tracker and the BlueCharm device under OMG_## gateways. However the Presence topic did not show up in MQTT Explorer until I disabled the “Publish Only Sensors”. Because I saw them there but could not get the presence topic it appeared to not be woking properly and contributed to my confusion. Maybe update that toggle to be “Publish Only Sensors & Trackers”??? Just a thought. And this suggestion is assuming my experience as stated above is fully accurate though it may not be because I changed so mony things over time I could simply be confused. Anyway, here is my BlueCharm Data you asked for if it is of any value now! Thanks so much!

{
“id”: “DD:34:02:06:71:E5”,
“mac_type”: 1,
“adv_type”: 0,
“name”: “Beacon JES 01”,
“rssi”: -69,
“distance”: 3.119501,
“servicedata”: “21010b0b2c1b0003b900d0011c”,
“servicedatauuid”: “0xfeaa”,
“brand”: “BlueCharm”,
“model”: “Beacon 08/04P/021”,
“model_id”: “KSensor”,
“type”: “ACEL”,
“track”: true,
“tempc”: 27,
“tempf”: 80.6,
“accx”: 953,
“accy”: 208,
“accz”: 284,
“volt”: 2.86
“servicedata”: “48010efe1355”,
“servicedatauuid”: “0x2080”
}

Hood ton hear you have the mqtt_room presence working in Home Assistant.

“Publish Only Sensors” should however not make any difference in the device trackers you are interested in, as both of them are actually defined sensors. So with the setting “Publish Only Sensors” on or off, they should be recognised and appear in the published MQTT messages just the same. Therefore you don’t need to woprry about this etting in theis regards, and can turn it on or off, depending on how you want other non-sensor defined devices to be published, as described in the docs.

You posted BlueCharm data does have the “track”: true tag, which signifies that it is a devices available for tracking as a device_tracker, but I have to admit that it looks like we have been missing to add the KSensor model_id group in the device_tracker discovery routine, for whatever reason :roll_eyes:

So rest assured, your BlueCharm not being discovered as a device_tracker is on us, and will be fixed in the next release and also be available in the development test builds in the next days. Then it will be correctly discovered just like your tile.

Thanks for making us aware of the BlueCharm issue.

So I specifically configured the BlueCharm to use the Ksensor based on instructions from your site. Not sure what it was before, but with that I get x,y,z, battery etc, but no home/away.

Also I did a test this morning and ran MQTT Explorer with the “Publish Only Sensors” disabled and then enabled in Home Assistant. With this enabled on all my OMG Devices I get a small number of topics and messages and no presence topic. With it disabled I get a bunch of topic and messages and the presence topic is visible. So it seems to make a difference for me. Maybe I have something else set wrong?

Image with Publish Sensors only enabled
MQTT EXPLORER 2024-01-25 062244

Image with Publish Sensors only disabled

EDIT: In looking over some of your links again I found this description which seems to imply that enabling “Publish Only Sensors” will disable presence?? :

Setting if the gateway publishes all the BLE devices scanned or only the detected sensors (default: false, available with HA discovery)

If you want to change this characteristic:
mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoBT/config -m '{"onlysensors":true}'

The gateway will publish only the detected sensors like Mi Flora, Mi jia, LYWSD03MMC… and not the other BLE devices. This is useful if you don’t use the gateway for presence detection but only to retrieve sensors data

With this enabled on all my OMG Devices I get a small number of topics and messages and no presence topic. With it disabled I get a bunch of topic and messages and the presence topic is visible.

Could it be that the bunch of topic and messages are what it actually means, that these are not sensors in the sense that there are not any decoders available for these devices, so that they are not regarded as sensors. Do they only have the 'rssi' key and possibly some of them a 'name', but not other properties at all?

Then it seems that your only real decoder recognised sensor is the DD3402XXXXXX device, which also showed up when “Publish Only Sensors” was enabled.

I found this description which seems to imply that enabling “Publish Only Sensors” will disable presence

Yes, this is currently the case, but we are thinking about changing this, as Only Sensors and Presence really should not be connected.

Could you try out a current test build for these two issue on one of your gateway, maybe OMG_01 from the examples above. This should now correctly publish presence messages, independent of the Publish Only Sensors” setting, and it should also correctly recognise your BlueCharm as a device tracker.

with SHA: 66f2ea

will be available in about 90 minutes from now.

I had a spare M5 Atom so I copied the config from OMG_01 and named it OMG_Test.
THis way I am available to load upcoming dev versions to help out on this or other issues. The “Publish Only Sensors” being enabled still seems to not allow the OMG_TEST data to be added to or updated in the presence topic. Sorry, but not sure how to validate that the BlueCharm as a device tracker. Give me a hint and I will check…

The test build has just finished this minute, available as a web upload at the link I gave above.

Just install it on your spare M5 Atom and then test it’s presence messages publishing with Publish Only Sensors on and off. It should publish independent of this setting now.

You can also see about your BlueCharm when you activate Discovery on the test M5 Atom. Then you should see a new entry under the device_tracker sup-topic of the homeassistant topic with the MAC address/id of your BlueCharm.

OK, so the Bluecharm is indeed recognized as a tracker now and I have the Home/Away entity in Home Assistant. THis can be seen in the image below.

image

The “Publish Only Sensors” being enabled now allows the OMG_TEST data to be added to or updated in the presence topic. However, it does not seem to limit results to only sensors. OMG_01 and OMG_Test are about 1 meter apart for testing. Looking at the image below both have “Publish Only Sensors” enabled but OMG_Test has a growing list of results and OMG_01 only shows the one or two beacons and the list does not appear to be expanding. So it does not feel like it is limiting the results to only sensors???

I will leave things set up with these to OMG Gateways in case you want me to do more testing. Thanks again for all your help!

Have you cleared the BTtoMQTT list for OMG_TEST for all the received messages from when “Publish Only Sensors” was still off?
MQTT Explorer keeps all the past messages stored there until a restart of MQTT Explorer, so if you clear them out, or the whole BTtoMQTT topic you should really only see the decoded sensors then.

Do they still appear after you have deleted them and have “Publish Only Sensors” on?

The first one also seems to be an iBeacon, with the long uuid instead of the MAC/id as the sub-topic?!?

Yes, I Cleared the list. I did so again and immediately I am up to 5 items. The first one which appears to be an iBeacon with the long uuid instead of the MAC/id as the sub-topic is that way because I updated the “presuseuuid” to true as whatever this device is it was generating Random MAC’s and also added to the Growing list.

After we get past this I hve a few more questions / suggestions for this but I will keep them out of the mix for now. Thanks again!

Yes, I Cleared the list. I did so again and immediately I am up to 5 items.

Are any of them non-sensors?

The iBeacon is a valid decoded sensor, so I am assuming that all the other ones coming up are also valid decoded sensors, but cannot tell from the cut off screenshot above.

Could it be that your other gateways do have a white-list applied, so that they only receive these white-listed devices, instead of all available sensors?

Actually, I seem to have confirmed the “Publish Only Sensors” to also publish non-sensors.

I have not set up any white lists so I do not think that is an issue. I used MQTT Explorer to copy the config items/settings from OMG_01 to the newly installed OMG_TEST so they should be configured the same. But they return significantly different results when “Publish Only Sensors” is enabled.

I am not sure (I am far from an expert…) what differentiates sensors from non sensors but I assumed the BlueCharm, Tile and Long uuid Item were possibly recognized as sensors and things like my Nvidia Shields are not Sensors. They drop from the list when I click “Publish Only Sensors” on 1.7.0 but do not on the OMG_TEST Gateway. I am getting different results from two what I believe are identically configured Gateways???

New test build is ready.

It has SHA:c21e2e

This should fix the Publish Only Sensors issue, and then you might be ready for your

few more questions / suggestions

:wink:

After loading and a quick look this does seem to have fixed the Publish Only Sensors issue. I will let this run for a while to confirm later today that I don’t see any issues. This image is after a few minutes running which would previously had 5-10 items in the list.

image

I will think through my other questions and suggestions to make sure they are clear (as clear as I can make them) before I add them so as not to consume a bunch of your time. Thanks again for all your assistance!

1 Like

So I updated one more of my OMG Gateways and everything continues to work on those updated to the DEV firmware. So on to a few questions….

First, and not in my original list. Can I update my devices wirelessly or do I need to bring them back to my computer to add the DEV firmware? And should I even install this on all of them or just wait for the next official release??? If wait, when might that be???

I seem to see a significant number of devices that do not remain on one MAC address but instead seem to increment their value over time. See image below. Not really random but not a consistent value. I had set the “pubuuid4topic” to true to fix the random MAC issue which clearly worked for my one Sensor “1ca92e23f0874df7b9a2fd4b716a4bf6” but not these other devices. So not sure if there is a way to fix this??? Just curious….

What does the "presuseuuid"setting actually do? I changed it from false to true to see if it would help with the incrementing MAC addresses in the 1.7.0 version but could not see any change. Just curious…….

In working with the Presence capability in Home Assistant I do see my BlueCharm device move from area to area and Home Assistant displays different OMG_## gateways as I move through my house. I have set up 10 devices as you can see from my earlier screenshots. However the range of the Bluetooth gateways appear to be pretty significant and receives data at longer distances than I would have expected. This is not much of an issue with the BlueCharm Tracker, but with the Tile tracker being with my wife’s keys on the kitchen table the presence seems to jump around quite a bit though the tile is not moving. I am curious if a sensitivity slider could be added which could be adjusted to lower the sensitivity or range to trackers. Might this make the Tile less susceptible to jumping around? For example, don’t publish devices with a distance greater than 15 feet might help reduce false changes in location??? Not sure if this is a good idea or not. Just trying to improve the location accuracy when multiple OMG gateways are available.

I also own a Samsung Galaxy Watch 4. I am not sure what it sends out as BLE messages but I did note that it periodically shows up in MQTT Explorer with Data like this:

{

“id”: “6D:1E:8A:61:D2:60”,

“name”: “Galaxy Watch4 (42RX)”,

“rssi”: -91,

“distance”: 25.51913

}

I would love to be able to track its location as a tracker, but it does not appear unless I disable the “Publish Only Sensors” so I guess its not a sensor or a tracker. I also believe that it changes its MAC address over time. Not sure if this changes depending on whether or not the watch is actively attached to my Samsung phone? I realize this is not really a OMG issue but was just curious what might be possible as the Watch is clearly visible at least at some times?

Hope you have a great weekend ahead!

Please don’t update any more for the time being, as the previous test build SHA:c21e2e has been overwritten by the nightly development build, where the above changes haven’t been merged into yet.

Not necessarily until the next official release, but once the above changes have been merged into development, then you can install any of the nightly test builds at the above URL.

Can I update my devices wirelessly or do I need to bring them back to my computer to add the DEV firmware?

There are two ways to update your gateways without having to fetch each one to hook up to your comoputer
• By accessing the WebUI’s Firmware Upgrade section of your gateways.
• By using Visual Studio Code with the PlarformIO extension installed, so you can create your own builds, which would also make it easier to set all relevant settings in advance of each install, like WiFi and MQTT credentials, but also BLE scan intervals etc.

The “pubuuid4topic” is only applicable to recognised iBeacon protocol devices, like the one you seem to have. For all other unknown devices which might randomly change their MAC address there is currently only a possibility to uniquely identify them with Theengs Gateway’s Resolving random private addresses functionality, if you know its Bluetooth Identity Address and corresponding Identity Resolving Key. As this is currently only implemented in Theengs Gateway it will only be practically available for Home/Away Device Tracker utilisation.

You best bet to not have your other undecodable unwanted Bluetooth devices not being published to your broker is to apply a white- or black-list to your OMG gateways.

May I ask what these Room Shields actually are? Do they possibly broadcast BLE advertisement data which might hold decodable information? To see that you can set Advertisement and Advanced Data to true on one or more, or your test gateway.

However the range of the Bluetooth gateways appear to be pretty significant and receives data at longer distances than I would have expected. This is not much of an issue with the BlueCharm Tracker, but with the Tile tracker being with my wife’s keys on the kitchen table the presence seems to jump around quite a bit though the tile is not moving.
…
I am curious if a sensitivity slider could be added which could be adjusted to lower the sensitivity or range to trackers. Might this make the Tile less susceptible to jumping around?

There are two issues here. First the gateway’s sensitivity, which cannot be easily adjusted by a slider, as this option is so rarely used by users that it is not included in the HA UI section of the gateways as not to produce too much clutter. There is the option of setting the minimum RSSI accepted to publish device data, which does exactly what you want to do, reduce or increase the sensitivity of each individual gateway as to with which single strength received it will publish a trackers data or not. This is best set individually for each gateway, as their individual ranges very likely vary.

For the Tile there is also a difference that it requires active scanning, which, if not set to the same interval as passive scanning (intervalacts vs. interval in the BTtoMQTT data), could produce this kind of jumping you are seeing. Also keep in mind that active scanning has a slightly detrimental effect on the device’s battery life, whereas passive scanning doesn’t affect it at all.

Generally though, the best trackers for the most precise room tracking are BLE trackers which allow you to set their txpower @ 1 m, like some generic iBeacons for example. By testing these devices’ reception with ones individual gateways exactly at 1 metre distance, and then setting this value accordingly, the calculated RSSI based distance calculation is more accurate and the minimum RSSI mentioned above can also be more precisely set.

I also own a Samsung Galaxy Watch 4. I am not sure what it sends out as BLE messages but I did note that it periodically shows up in MQTT Explorer with Data like this:

Once you set the Advertisement and Advanced Data to true you will see that your Galaxy watch actually sends a lot more data. From the little experience I have with these watches I would say that its manufacturerdata likely starts with "750001000200010…". If this is restricted to the Galaxy watches, or if Galaxy phones might have the same manufacturerdata, and if the remaining data does contain any useful decodable information or not I also do not know. Let’s see what yours broadcasts :wink:

At least the very obvious part of the manufacturerdata would likely allow for creating a decoder to also make Galaxy watches a sensor, with automatic device tracker discovery. If it does change it’s MAC address however, again this would currently only be possible with Theengs Gateway and its Identity MAC/IRK functionality - if you can get information on how to find these out for your Galaxy watch.

While all the above linked to setting possibilities can be done in MQTT Explorer’s Publish section, which you might have used for this so far, keep in mind that this is also possible in the Console section of the WebUI.

Hope you have a great weekend ahead!

Thank you, and have a great weekend, too!

I will take a look at this PlatformIO Extension if I can find it…

This is why I was using the “Publish Only Sensors” I will look at the White list feature and give it a try.

The Room Shields are Android TV Boxes. Details can be seen here:
Nvidia Shield TV - Wikipedia

This is exactly what I was suggesting. Must have missed it in the Docs. Sorry!

I will look into the Watch Question a bit on my own and circle back if I have any questions or findings I think may interest you!