Hi all,
I’m looking to add presence detection to my Wyoming satellites and this seems to fit the bill. Installation was straight forward (good instructions on the site). I can see the MQTT messages via MQTT explorer under the “/home” topic. However, most of my bluetooth devices aren’t on the supported list - so they aren’t being fully recognized. Nor is my phone (android) which is pushing out IBeacon messages (using the companion app). On my Home Assistant side I do have mqtt_room enabled working with a couple of esp32 running espresense. Which gets me to my question: how do I get presence info from OMG? I did turn on the presence option within the theengsgw.conf file - which adds a “presence” subtopic under “/home/TheengsGateway” but not sure what to do with that. Love some advice on where to look for the next steps. Thanks in advance,
As I’ve been researching, I’ve come across something I would appreciate some clarification on… IBeacon is listed as a supported device here: iBeacon | Theengs Decoder: BLE message decoder
but here, it implies it’s not supported: Use | Theengs Gateway: BLE to MQTT bridge
In the 2nd link it notes iBeacon continually changing - but that’s not the case (especially with the HA Companion App).
That said, looking at the topic for theegns I don’t see any devices being identified as IBeacon, even though I can see those same devices (android phone/watch) under the espresence thread (from esp32(s)).
So am I missing something? Would love some clarification. Thanks in advance,
Welcome @tsx-5
iBeacons are definitely received and published by MQTT messages with OpenMQTTGateway and Theengs Gateway, with or without random MAC addresses, depending on which device sends out the these iBeacon broadcasts. Unless, of course, there is a white-list or some other restrictive setting applied preventing their publishing.
For both the auto-discovery of such iBeacon is disabled however, to prevent large amounts of iBeacon discoveries, as iBeacon broadcasts are send out by a large number of devices for pairing purposes or such.
I hope this helps in clarifying things a bit.
Thanks @DigiH,
Can I bother you for a bit more guidance? So, I made sure options publishing IBeacon is enabled and I’m pretty sure I have options for discovery (which is supposed to be on by default anyway) on. I do see IBeacon messages come across MQTTExplorer, but nothing is showing up in Home Assistant. The ID’s in MQTTExplorer are the standard ID’s not the unique ID that i’d expect from an IBeacon - however there is a UUID, which looks much closer to what I would expect. From there, I’m kind of at a lost… Any idea what I should be looking at/for?
iBeacons are generally not being auto-discovered, as about 99% of iBeacon discoveries would be unwanted ones, and therefore do not show up in the MQTT Integration in HA. But you can take out IBEACON from
"discovery_filter": [
"IBEACON"
],
in your theengsgw.conf settings file. Assuming here that you are using the pip installation of Theengs Gateway.
Just be careful to only let Theengs Gateway run for a little while for your iBeacons to be discovered, then turn the filter on or discovery off again to avoid lots and lots of such unwanted iBeacons discoveries.
And are you certain your Android phone and watch do not have random MAC addresses? Unless they both are quite old, with the phone running an older version of Android, they will likely have random MAC addresses.
I don’t have any personal experience with ESPresence, but AFAIK it connects to phones and watches initially to fetch any Identity MAC and IRK to be able to resolve random MACs. If that is the case you could copy both from wherever ESPresence stores them and enter them into the Theengs Gateway settings for the same random MAC resolving.
One easy way to find out if your phone has random MACs or not, is to install the freeware
Beacon Scope
on your phone, and set up and iBeacon beacon in it, with the special UUID
54686565-6E67-732D-6942-6561636F6E31
This will then be recognised and decoded as a dedicated Theengs iBeacon Tracker and also auto-discoverd in HA if discovery is on.
This also doesn’t require any changes to the above mentioned discovery filter.
If you monitor this Theengs iBeacon Tracker in MQTT Explorer you will see if its MAC/id always stays the same, or if it will change every 15 minutes or so, making it look as if a new Theengs iBeacon Tracker with new MAC/id appears every 15 minutes.
Once you know for sure if the phone has a static or random MAC address you can proceed either with copying the required Identity MAC/IRK into Theengs Gateway or just disabling the iBeacon discovery filter for a static MAC/id.
Really stocked to have your assistance 
I did indeed remove the “IBEACON” filter from my configuration file - as you correctly deduced that I am using the pip installation method.
So, while my phone may have random MAC address - the way around this is using the Home Assistant Companion App ( Sensors | Home Assistant Companion Docs ). Using the App removes any concern about MAC and IRK.
And watching MQTT Explorer - does verify that the MAC changes, but the UUID stays the same. Assuming the IBeacon Tracker under Home Assistant is the integration that would create devices ( iBeacon Tracker - Home Assistant ) this shouldn’t be a problem. However, I’m not see devices being added… But, maybe I haven’t waited long enough?
Another question that reading the IBeacon Tracker integration brings up - there’s no node device created for the actual gateway. So, how do I know what room it a detected device is in?
Again, really appreciate your help… I’m thinking I’m getting closer.
As I was reviewing other threads, I noted that there’s a diagnostic command (didn’t see this option with the “-h” option). What I learned from running that is the discovery_filter is still on. Even though my config file removed that switch all together. Nor did it work when I just removed the “IBEACON” keyword from the option and left the keyword there. That said, IBEACON devices show up in MQTT Explorer regardless. So, what is that option supposed to do? And how should I edit it within the config file?
Here’s the output of the diagnostic command, just in case:
# Theengs Gateway Diagnostics
## Package Versions
| Name | Value |
|--------------------|--------|
| Theengs Gateway | 1.5.0 |
| Theengs Decoder | 1.7.8 |
| Bleak | 0.22.2 |
| Bluetooth Clocks | 0.2.0 |
| Bluetooth Numbers | 1.1.1 |
| Paho MQTT | 2.1.0 |
| Bluetooth Adapters | 0.19.3 |
## Python
| Name | Value |
|----------------|----------------------------------|
| Version | 3.11.2 |
| Implementation | CPython |
| Compiler | GCC 12.2.0 |
| Executable | /home/bat-sat/theengs/bin/python |
## Operating System
| Name | Value |
|--------------|----------------------------------------------------|
| System | Linux |
| Release | 6.6.20+rpt-rpi-v8 |
| Version | #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) |
| Machine type | aarch64 |
| Distribution | Debian GNU/Linux 12 (bookworm) |
## Configuration
File: /home/bat-sat/theengsgw.conf
{
"adapter": "",
"bindkeys": {},
"blacklist": [
"D7:36:36:37:55:6C",
"F8:35:0B:57:0F:C6",
"C4:36:34:39:1B:43",
"D3:38:32:34:75:2E",
"CE:38:32:34:48:05",
"CC:6A:10:2C:3A:7C",
"C7:36:35:30:21:1C",
"15:C4:23:04:07:9D"
],
"ble": 1,
"ble_scan_time": 5,
"ble_time_between_scans": 5,
"discovery": 1,
"discovery_device_name": "TheengsGateway",
"discovery_filter": [
"IBEACON"
],
"discovery_topic": "homeassistant",
"enable_tls": 0,
"enable_websocket": 0,
"general_presence": 1,
"hass_discovery": 1,
"host": "192.168.0.22",
"identities": {},
"log_level": "INFO",
"lwt_topic": "home/TheengsGateway/LWT",
"pass": "***",
"port": 1883,
"presence": 1,
"presence_topic": "home/TheengsGateway/presence",
"publish_advdata": 0,
"publish_all": 1,
"publish_topic": "home/TheengsGateway/BTtoMQTT",
"pubuuid4topic": 1,
"scanning_mode": "active",
"subscribe_topic": "home/+/BTtoMQTT/undecoded",
"time_format": 0,
"time_sync": [],
"tls_insecure": 0,
"tracker_timeout": 120,
"user": "***",
"whitelist": []
}
## Bluetooth adapters
Default adapter: hci0
### hci0
| Name | Value |
|--------------|-------------------------|
| address | B8:27:EB:XX:XX:XX |
| sw_version | bat-sat |
| hw_version | usb:v1D6Bp0246d0542 |
| passive_scan | False |
| manufacturer | Raspberry Pi Foundation |
| product | bcm43438-bt |
| vendor_id | None |
| product_id | None |
Only if you want to stick with ESPresence, which likely connects to the phone initially to fetch the Identity MAC and IRK. For Theengs Gateway a random MAC address is definitely an issue, which can also only be resolved with an existing Identity MAC and IRK.
This is usually the case with iBeacons, the UUID being an identifiable part of the BLE broadcast, which some hardware devices actually use to identify themselves as certain models, like the BM2 battery monitor, for example. But all of them having the same UUID, as most other hardware devices do, it is the actual MAC address which is the vital part for Theengs Gateway for correct recognition and publishing of an iBeacon, be it with a static MAC or random MAC, but the latter only if the Identity MAC and IRK are supplied.
I’m certain you can read up on the ESPresence side or in the HA forum on where it stores these details for your phone and watch.
That is correct, only OpenMQTTGateway auto-discovers its own settings UI for the gateway, Theengs Gateway has its theengsgw.conf settings file. But you can differentiate the rooms by the name you give your different gateways in
…
"discovery_device_name": "TGWRoom",
…
"lwt_topic": "home/TGWRoom/LWT",
…
"presence_topic": "home/TGWRoom/presence",
…
"publish_topic": "home/TGWRoom/BTtoMQTT",
…
I honestly do not know, as I haven’t implemented the discovery_filter, nor have I ever tried to remove IBEACON from it to auto-discover iBeacons 
As I’m currently travelling I also cannot test this out with my set-up at home. Someone else might have to jump in here.
Safe travels!
Thanks for the changing name suggestion, will get right on that!
As I’ve looked at my ESPrence setup and compared it to what Theengs is doing - there an important difference. ESPrence isn’t using IRK or anything like that… What it’s doing identifying the device_id differently. It has a MAC tag, where the MAC address is located and a UUID tag where the UUID tag is located. But for the device_id it is defined by the type of device it is - so you get device IDs like:
“id”: “iBeacon:xxxxxxxx-xxxx-xxxx-be0a-xxxxxxxxxxxx-100-40004”
“id”: "sonos:xxxxxxxxxxxx
Then within the configuration.yaml file you define a device looking for that id
platform: mqtt_room name: "Pixel Room Tracker" state_topic: "espresense/rooms" device_id: "iBeacon:xxxxxxxx-xxxx-xxxx-be0a-xxxxxxxxxxxx-100-40004" timeout: 10 away_timeout: 120
OpenMQTTGateway does have a pubuuid4topic settings option, which likely doesn’t help you, as it only runs on an ESP32.
Theengs Gateway, however, does not have this setting option.
And while it is always great to have new users of Theengs Gateway, may I frankly ask what keeps you from using ESPresence with the HA Companion app’s iBeacon, as it seems to be working perfectly exactly how you want it?
Well,
I’ve setup a couple Wyoming Satellites ( GitHub - rhasspy/wyoming-satellite: Remote voice satellite using Wyoming protocol ) which are running on Raspberry PIs. While the esp32 are really cheap - they are another device I have to find places to hide around the house. I’d like to minimize that. So, I was looking for an option to do that function on the PIs. There is room-assistant, but it’s not being maintained and comes with a whole list of caveats. Which brought me here.
So I take it ESPresence doesn’t run on RPis. Just shows how little I know about it 
The only way it will work with Theengs Gateway however is by knowing the Identity MAC and IRk, to be able to resolve the random MAC addre4sses.The same for the watch, if it also has a random MAC.
Nope,
espresence doesn’t run on RPIs :-/ And it doesn’t rely on Identity MAC and IRK to do presence detection… However, it turns out you can “enroll” a device - which will tell you the IRK (yeah!). That said, I’m a bit confused on what is meant by “identity MAC” - aren’t all the MAC address’s being generated random?
There you will have one part required for Theengs Gateway, the IRK.
All the broadcast randomised MAC address are different, that is correct, but with the appropriate IRK - as the acronym for Identity Resolving Key suggests - they all resolve to the one true Identity MAC address. On my iPhone this Bluetooth MAC address can be easily found in the settings. I would assume on Android it is displayed somewhere similar.
I assumed the same thing, so I looked up under settings/about on my android phone and found a bluetooth address - which I added to the theengsgw.conf file, as such:
"identities": { "xx:xx:xx:xx:xx:xx": "<long string of text here>"},
When I run python -m TheengsGateway.diagnose, I get (for that section)
"host": "192.168.0.22",
"identities": {
"INVALID ADDRESS": "***"
},
Am I not able to add this to the configuration file?
You should be, and it looks similar to the identities I have here.
Maybe some unwanted invisible character or line break?
What happens when you clear the identities section completely from theengsgw.conf, and just start up once with the additional --identities argument, as described in the documentation.
TheengsGateway --identities 00:11:22:33:44:55:66 0dc540f3025b474b9ef1085e051b1add
What happens then, and does it look differently in theengsgw.conf when being assigned through the start-up argument?
AND without wanting you to divulge your Identity MAC, are the letters upper or lower case in the MAC?
First off - continued thanks for helping me out!
When I ran the command from the terminal directly, I was met with lots of text…
TheengsGateway --identities xx:xx:xx:x:xx:x 2b3e386c97beb8998d81e926f6f6cca8
INFO:BLEGateway:Starting BLE scan^M
INFO:BLEGateway:Connected to MQTT broker^M
INFO:BLEGateway:Subscribed to home/+/BTtoMQTT/undecoded^M
ERROR:root:A message handler raised an exception: Invalid base64-encoded string: number of data characters (9) cannot be 1 more than a multiple of 4
Traceback (most recent call last):
File “/home/bat-sat/theengs/lib/python3.11/site-packages/TheengsGateway/privacy.py”, line 31, in resolve_private_address
key = bytes.fromhex(irk)[::-1]
^^^^^^^^^^^^^^^^^^^
ValueError: non-hexadecimal number found in fromhex() arg at position 3During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “src/dbus_fast/message_bus.py”, line 805, in dbus_fast.message_bus.BaseMessageBus._process_message
File “/home/bat-sat/theengs/lib/python3.11/site-packages/bleak/backends/bluezdbus/manager.py”, line 928, in _parse_msg
self._run_advertisement_callbacks(
File “/home/bat-sat/theengs/lib/python3.11/site-packages/bleak/backends/bluezdbus/manager.py”, line 1054, in _run_advertisement_callbacks
callback(device_path, device.copy())
File “/home/bat-sat/theengs/lib/python3.11/site-packages/bleak/backends/bluezdbus/scanner.py”, line 273, in _handle_advertising_data
self.call_detection_callbacks(device, advertisement_data)
File “/home/bat-sat/theengs/lib/python3.11/site-packages/bleak/backends/scanner.py”, line 247, in call_detection_callbacks
callback(device, advertisement_data)
File “/home/bat-sat/theengs/lib/python3.11/site-packages/TheengsGateway/ble_gateway.py”, line 462, in detection_callback
address = self.rpa2id(device.address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/bat-sat/theengs/lib/python3.11/site-packages/TheengsGateway/ble_gateway.py”, line 567, in rpa2id
if resolve_private_address(address, irk):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/bat-sat/theengs/lib/python3.11/site-packages/TheengsGateway/privacy.py”, line 34, in resolve_private_address
key = b64decode(irk)[::-1]
^^^^^^^^^^^^^^
File “/usr/lib/python3.11/base64.py”, line 88, in b64decode
return binascii.a2b_base64(s, strict_mode=validate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
binascii.Error: Invalid base64-encoded string: number of data characters (9) cannot be 1 more than a multiple of 4
ERROR:root:A message handler raised an exception: Invalid base64-encoded string: number of data characters (9) cannot be 1 more than a multiple of 4
Traceback (most recent call last):
File “/home/bat-sat/theengs/lib/python3.11/site-packages/TheengsGateway/privacy.py”, line 31, in resolve_private_address
key = bytes.fromhex(irk)[::-1]
What I take away from it - is that it’s not liking the IRK :-/
I assume your IRK is in hec, correct? How long is it?
If I remember correctly it might be required to reverse the IRK retrieved from ESPresence , so
aa11bb2244…dd66ee77ff
to
ff77ee66dd…4422bb11aa
or possibly required as base64 from the above, although both hex and base64 should work fine.
All I can suggest now is to try the hex reversed, if that fails convert the retrieved IRK to base64, if that fails convert the reversed hex to base64
Yeah, its in 32 hex. Will run through those iterations (we’ll see how long these old eyes hold up )
Same here bedtime here on my end.
So upper case MAC entered, and 32 hex IRK.
The above is really all I can think of at the moment. If either of these iterations still don’t work, some other team member with an Android phone might have to look into it, to see if it is a code issue or if the retrieved IRK might have to be wrangled in some different way, although 32 hex sounds correct, at least from the examples given in the docs.
And thank you for being so proactively persistent in finding out how to retrieve the IRK and all the info, and not giving up halfway through