Custom BLE beacon: Demultiplexing and publishing of multiple sensors in "manufacturerdata" parameter

Theengs OpenMQTTGateway Bluetooth gateways
1

Hello,

A custom BLE beacon encapsulates the values ​​of some sensors (3 temperatures, 2 atmospheric pressure, moisture) in the “manufacturerdata” parameter.
I know how the values ​​have been multiplexed and their position in the “manufacturerdata” parameter.
My goal is to separate the individual values ​​and publish them with as many topics towards Home Assistant.

This is what I get in the “manufacturerdata” topic: home/OpenMQTTGateway_ESP32_OLM_POE/BTtoMQTT/AABBCCDDEEFF/manufacturerdata {“manufacturerdata”: “30000215dc029e04cf240000000000000000000000008102c8”}
(with “#define simplePublishing true” in User_config.h)

My attempt was this:
I integrated the BLE in Home Assistant and by modifying the procBLETask function in ZgatewayBT.ino I was able to demultiplex the first temperature value and publish it in Home Assistant.
Now I have no idea how to publish the values ​​of the other sensors; theoretically I should have a parameter available for each sensor in order to publish them separately with as many topics, but I only have “manufacturerdata”.
Basically, what I would need is to receive something like this:
{
“id”: “AA: BB: CC: DD: EE: FF”,
“mac_type”: 0,
“sensor_1”,
“sensor_2”,
“…”,
“sensor_N”,
“rssi”:xx,
“distance”: xx
}
It’s possible?
I hope my meaning is clear.

Greetings

2

Hi @tommaso,

since it is a custom BLE beacon, likely not useful to other users unless it might be an open-hardware project, you could instead create a personal BLE decoder for your own beacon, similar to all the other BLE decoders of the Theengs Decoder library which OpenMQTTGatway uses for BLE decoding.

Best to leave the procBLETask function in ZgatewayBT.ino in OMG alone for this task :wink:

That is if you are familiar with GitHub and forking projects there.

Other than that I’d only be able to help you achieve this in OpenHAB, but I’m sure others will be able to assist for direct decoding in HA as well. In either case I would set simplePublishing to false though, as this would include the manufacturerdata within the MQTT message, along with the MAC address etc. for easier white/black-listing of the device(s), while the manufacturerdata still being accessible for decoding.

1 Like
3

Hi DigiH

Thank you very much for the suggestion.

It is a HW/FW platform made by me based on STM MCU and BLE. Currently the prototype uses 16 sensors for the detection of environmental parameters, air quality and environmental pollution. It works fine locally, but I would like to access data remotely too and I think using openMQTTGateway is a good solution.

I have never delved into the Theengs Decoder library code but I will try.
It will be a great challenge for me! :slightly_smiling_face:

Greetings

4

This sounds very interesting! May I ask which 16 sensor types are included? Are you by any chance planning on making the hardware schematics and firmware public (on GitHub)? In that case a decoder for this could also be included in the official release of Theengs Decoder eventually.

Let us know how you get on :slight_smile: I recently also submitted a new decoder for a similar air quality device, which you could use as some guidance along the way

As you can see, there isn’t that much to it, and in your personal only decoder case you wouldn’t even have to worry about entries in config.js, a device.md for documentation or likely even no test_ble.cpp test cases, although the latter helps in making sure everything is working fine with the new decoder :wink:

To start fork Theengs Decoder on GitHub, create a new branch for your decoder in your fork … and off you go :slight_smile:

1 Like
5

Hi DigiH

At the moment the board supports the following sensors:
temperature (with calculation of dew point and heat index), RH, barometer (absolute and relative pressure),
VOC, CO2, CO, NO2, NH3, CH2O, O3, SO2, C6H6,
PM (x) (PM1.0, PM2.5, PM4.0, PM10).
Set-ups with any combination of these sensors are possible.
The FW is complete, but the HW is at the pre-prototype level. I’m working on a more industrialized version, and I don’t rule out uploading it to a GitHub fork when it is completed.
The reference to the decoder for the Qingping device is very useful and is an excellent starting point for me.
Thank you very much.

Greetings

6

Hi Tommaso,

wow, that really is a very extensive air quality setup. Really hope to see some details whenever you are ready.

All the best.

7

Hi DigiH.

I spent some time trying to create my own BLE decoder in theengs_decoder. I’ll list what I’ve done:

  1. I forked Theengs Decoder at “GitHub - Algol655/decoder: Efficient, portable and lightweight library for Internet of Things payload decoding.”.
  2. Here I edited/added the files as you suggested in the “Qingping Air Monitor Lite CGDN1” example.
  3. The added files are “src/devices/SENSUS_191_json.h” and “docs/devices/SENSUS_191.md”.
    I have also added the file “docs/ Sensus191_BLE_Beacon_Payload.pdf”, which shows the format of my Beacon Advertising packets.
  4. Also in “platform.ini” I changed the URL of the decoder with my own.

At the moment only the Beacon Advertisement with AdvID = 0x01 should be decoded,but it is not working.

I attach the MQTT Explorer screen, which shows that my sensor is not recognized and consequently the values, encapsulated in manufacturerdata, are not decoded.

MQTT_Explorer
MQTT_Explorer1383×920 114 KB

Where did I go wrong?

Thank you
Greetings

8

Hi @tommaso

Just after a quick look, please amend the following:
• In decoder.h put SENSUS_191 before BLE_ID_MAX, and also put a comma after it
• the same with the comma in config.js, only for the doc to work, not vital for the actual decoder working
• In your decoder file, you have an extra trailing comma in the last property definition

  }"forecast":{
     "condition":["manufacturerdata", 25, "01"],
     "decoder":["value_from_hex_data", "manufacturerdata", 19, 1, true, false]**,**
  }

which needs to be deleted.

So all really only small comma issues from what I can see at a first glance :slight_smile:, other than that it looks really good!

Let me know how you get on with the above changes.

P.S. for the temperature properties you might want to stick to the Decoder tempc formatting, e.g. tempc, tempc2, tempc3 … this allows for the Decoder to automatically also create and convert the temperatures to Fahrenheit. Possibly not important if you just use the decoder for yourself with Celsius, but good for a general release and other people wanting to use it with Fahrenheit.

Also not sure what happened there

which caused an error when I tried to build your version - the ArduinoJson submodule is definitely needed. So you want to revert this commit.

Also would you mind pasting the raw manufacturerdata here as text, so I don’t have to type it off your screenshot above :wink: I will then create a test case here for it to see how the decoder is working.

Also - having looked at your Sensus191_BLE_Beacon_Payload.pdf a bit more, the current property “value_from_hex_data” decoding statements need to be adjusted a bit.

The whole manufacturerdata is being used, so your above "ffff550801ae8…", and the counting is from index 0, with each indivudual index counted, not two-index bytes. So for the first temperature it should be
"decoder":["value_from_hex_data", "manufacturerdata", 6, 4, true, true],
to get 26.4ºC
This index counting is also applicable to the conditions.

9

Hi DigiH

With the changes suggested by you it works!

MQTT_Explorer1
MQTT_Explorer1590×638 40.2 KB

I can build even without the ArduinoJson submodule, it seems that in my version of platformIO it is not necessary … I don’t know why.

These are the 5 raw manufacturerdata transmitted in their respective advetising beacons:
{“id”:“02:80:E1:00:00:F5”,“mac_type”:0,“manufacturerdata”:“ffff5510010b8c0100a302cf002201d18b010082001213000001”,“rssi”:-74,“distance”:5.270232}
{“id”:“02:80:E1:00:00:F5”,“mac_type”:0,“manufacturerdata”:“ffff5521006f020a0009006036640e3f176e2c00001213000002”,“rssi”:-72,“distance”:4.287841}
{“id”:“02:80:E1:00:00:F5”,“mac_type”:0,“manufacturerdata”:“ffff550d00b70102000b00c7085002be03822100001213000003”,“rssi”:-72,“distance”:4.287841}
{“id”:“02:80:E1:00:00:F5”,“mac_type”:0,“manufacturerdata”:“ffff550200030007000e00060000000000010001001213000004”,“rssi”:-71,“distance”:3.860897}
{“id”:“02:80:E1:00:00:F5”,“mac_type”:0,“manufacturerdata”:“ffff553f003f0000000000060000000000010001001213000080”,“rssi”:-71,“distance”:3.860897}

Next steps: also publish the data contained in the other 4 packets and integration in Home Assistant.
What is the maximum size of the “const char * _SENSUS_191_json” structure? My doubt is that it cannot contain all data.

Thank you
Greetings

1 Like
10

Hi @tommaso,

I’ve left a comment on one of your commits, the new index counts look great on the “value_from_hex_data” are fine, but the conditions index counts should also change, they only work by chance at the moment :wink:

Good thinking :slight_smile: that’s what I wanted to comment on next. Have a look at the version which I did with a few changes

https://github.com/theengs/decoder/compare/development...DigiH:decoder:algol-sensus191

Since you don’t want to overload the const chars * I changed their names and the decoder’s file name with an extra _1, to indicate Adv ID 01. Since all the other info is always separately broadcast in their own Adv ID messages it’ll be best to duplicate the current decoder with the _2, _3 … variations.

The model_id doesn’t really need the _1, I just left it in there for clarification, also for the test case.

Also have a look at my extended model condition, also taking into account the …"manufacturerdata", "index", 50, "01" for the separate Adv ID.

If all properties are always sent with each Adv ID number, then with the above the individual property conditions could also be deleted.

Also regarding the integration into Home Assistant. While you have nicely changed the temperatures (also have a look at how I underscored the _dp and _hi for a bit clearer spotting of the temperature differences - dew point and heat index- in the raw MQTT messages) you should also change humidity to hum, pres etc. to allow for correct unified HA discovery. Also the pressure unit from mBar to hPa - all to automate the HA discovery inclusive the units.

With the above you can continue with the remaining packets decoders along the same way :slight_smile:

11

Hi DigiH

First of all, thank you very much for the quick, clear and very useful answers.
I entered the missing beacons and now all values are published correctly.

MQTT_Explorer2
MQTT_Explorer21184×952 61 KB

I thought of putting the 5 “const char * _SENSUS_191_x_json” structures in one file. I hope this has no contraindications.

Now I only have the integration with Home Assistant, I will work on it in the next few days

Thanks again
Greetings

12

Hi Tommaso,

you’re very welcome :slight_smile:

Just a few more additional pointers

• Both pressure properties have "decoder":["value_from_hex_data", "manufacturerdata", XX, 8, true, true], - with the first boolean correctly indicating true for little endianess, but the second boolean indicates if the result can be negative, which isn’t the case here and should be false. While you probably won’t send data which could be interpreted as negative it just clarifies the decoder definitions :wink: possibly also applicable for some other properties.

• As I posted before, with the model conditions for all the decoders checking for the advertising IDs at index 50, the individual additional condition checks for each property is superfluous, and only really takes up space at the moment :wink:
One better idea would be to have a condition check to see if the actual data is actually properly transmitted. This would possibly require some firmware changes on the actual device, as I don’t know the firmware workings. Several other sensors actually send an unrealistic highest value of ffff if a particular sensor is faulty, broken or had to be disconnected for some reason. Then with a property condition check like

      },"tempc":{
         "condition":["manufacturerdata", 6, "!", "ffff"],
         "decoder":["value_from_hex_data", "manufacturerdata", 6, 4, true, true],
         "post_proc":["/", 10]
      },

the temperature would only be decoded if it is not ffff when the temperature sensor might be out of order. This would also eliminate messed up charting etc. on your HA end, which could happen with otherwise unrealistic values, and it would be a better use of the property conditions as the current unnecessary repeated index 50 Adv ID check.

• not knowing exactly how your sensor firmware works I’m not sure about the units, but

  1. Isn’t CO2 usually always defined as ppm?
  2. any cubic meter units should have the superscript 3, as in μg/m³ - looks nicer :wink: but it’s also the way OpenMQTTGateway and Home Assistant define and expect these units for discovery. Easiest to copy from here or other decoders and paste.

Just two things I stumbled across when looking through all the decoders.

Should be fine if you still define each decoder for separate processing - quickly looking at your latest commit I think you might have deleted a bit too much in decoder.h :wink: have a look at the CGG1_V1 and CGG1_V2 decoders, which do something similar, but yours also having separate json_props definitions.

All the best

P.S. I really hope we see some information, schematics, parts list and possibly firmware for this device at some stage as I find it impressively interesting! :slight_smile:

13

Hi DigiH

thanks a lot for your always valuable advice!

Right. Fixed in the latest commit
It seems to me that when the first parameter of “decoder” is “string_from_hex_data”, endianness is fixed regardless of the value of the first boolean. Is that so?

Fixed in the last commit

Good idea! Done on the decoder, but the FW is to be updated.
Currently, the after reset built in test, reports the sensors configuration and their status in the SensorStatus register, published in ADV_ID 0x80

Right. Fixed in the latest commit

I did not understand this well. In any case I have restored the lines in decoder.h

In GitHub - Algol655/F3_BSP: FW package used in Sensus projects I put the FW and an image of the assembled device. While the FW is quite complete, as you can see the HW is still in the preprototypical phase

Greetings

14

That is correct, as no booleans should/can be give for “string_from_hex_data” - needs a better clarification in the docs. Thanks.

I’ll definitely have a look through that :slight_smile: thanks.