OpenMQTTGateway v1.4.0

1technophile · February 13, 2023, 6:58pm

What’s Changed

Adaptive scanning

This release brings a very interesting feature for Bluetooth Low Energy scanning.
You may have already asked yourself, what should I set for the interval between my scans? Do I use passive or active scanning? Or how do these parameters impact the battery life of my sensors?
To remove these questions, @DigiH and I have integrated an innovative function - Adaptive scanning

Adaptive scanning will decide automatically following your devices if you need to use passive/active/continuous scanning.

For example, a door or a PIR sensor will require continuous scanning, so if detected, the gateway will reduce its time between scans to the minimum. Or your devices may also require active scanning to retrieve data. The gateway will trigger active scans at regular intervals in this case.
Behind the scenes, a big work on device categorization and analysis has been done in Theengs Decoder to enable this feature.

Why are we doing this?
Increase your sensors’ battery life, decrease network traffic, and reduce gateway power consumption.

See below in Home Assistant how the interval is automatically adapted after the detection of a PIR sensor nearby.

This new feature is activated by default, you can deactivate it, and the gateway will go directly into continuous active scanning.
Consequently, the esp32ble-dev-cont environment has been merged into esp32ble-dev, and the “Active scan” command has been removed. You can remove the topic from your broker to delete it from the controller.

Theengs Plug

Also, this release brings support for the second RGB Led of the Theengs Plug alongside the capability to reset at the start. I encourage Theengs Plug users to update following this guide.

New boards

Added board AirM2M CORE ESP32C3 by @xe5700 in #1429

New BLE devices

Devices	Model	Measurements
Polar	H10 Chest strap	activity heart rate

Features

[HA Discovery] - Optimize low power mode discovery by @1technophile in #1443
[ONOFF] Set power indicator status at start by @1technophile in #1456
[LED RGB] Enable ON OFF Led power display depending on actuator status by @1technophile in #1451
[BT] Adaptive scan by @1technophile in #1440
[BT] Continuous active scanning when adaptive scanning is false by @1technophile in #1448
[BT] Improve conditions of adaptive scanning by @1technophile in #1450
[BT] Bump decoder to v1.2.0 by @1technophile in #1454
[BT] Active Scan Interval by @DigiH in #1433
[BT] BLE Tag properties visibility by @DigiH in #1432
[BT] SwitchBot Contact Sensor Counters by @DigiH in theengs/decoder#283
[BT] Polar H10 Heart Rate Sensor by @DigiH in theengs/decoder#274

Docs

[DOCS] Add reset time and low power mode indications by @1technophile in #1452
[DOCS] - Add nightly builds link by @1technophile in #1441
First Pass - Added availability_topic support by @NorthernMan54 in #1442

Miscellaneous

[LPM] Add capability to deactivate the function at build time by @1technophile in #1449
[LPM] - Fix reboot loop when low power is used by @1technophile in #1444
[LPM] - Update state by @1technophile in #1445
[WIFI] Increase delay between first wifi connection attempt by @1technophile in #1446
[TRIGGER] Enable reset at the start when InputGPIO is used by @1technophile in #1447
Add missing ThingPulse envs by @marcelstoer in #1455
Removed availability reporting for non-local sensors ie Bluetooth sensors by @NorthernMan54 in #1453
[CI] Only build doc and binaries on owner repository by @1technophile in #1434
Bump http-cache-semantics from 4.1.0 to 4.1.1 by @dependabot in #1435

New Contributors

@xe5700 made their first contribution in #1429
@marcelstoer made their first contribution in #1455

Full Changelog: v1.3.0…v1.4.0

esa66 · February 16, 2023, 1:59pm

Dear Florian,

Can you adapt the maximum time between active scan to 600 sec for Temp/Hr sensor ?
If yes, I would like to install it and set the active scan to 5 minutes and check if batteries will last a bit more.

Thanks
Eric

1technophile · February 16, 2023, 2:32pm

Hello,

You can adapt it with the following MQTT command:
home/OpenMQTTGateway/commands/MQTTtoBT/config -m '{"interval":600000}'

esa66 · February 16, 2023, 2:59pm

Hi Florian, DigiH,

A request about the new Adaptive scanning.

It mean if it detect a PIR, it will reduce the scan to +/- 1sec ?
In this case all other devices like Sensor will be scanned every same period of 1 Sec.
Is it correct ?
If yes, is it a good think ?

You say “a door or a PIR sensor will require continuous scanning”.
For this kind of sensor, it should itself send multiple time spontaneously an alarm.

If I install the last version, I suppose I have to enable Adaptive Scan to make it scan T° sensors.
I don’t want it detect something which will reduce the scan to 1 sec.
Is it possible to lock the scan period to 5 minutes.

Thanks

Eric

1technophile · February 16, 2023, 3:28pm

Hello,

You still can deactivate the adaptive scanning and adjust the interval as you want, like 5 minutes.

The interval will go to 100ms if it detects a PIR BLE device.

DigiH · February 16, 2023, 4:39pm

I think there was a bit of confusion sounds as if @esa66 wants it similar to how I have it set, only have active scanning every 5 minutes, but more frequent, or even continuous passive scanning.

So with adaptive scanning true or false, interval is or can be set to 100 ms if a PIR is present.

Then set the active scan interval with

mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoBT/config -m '{"intervalacts":300000}'

to only have one active scan every 5 minutes.

esa66 · February 16, 2023, 5:42pm

Hi,
If I set {“interval”:50000}
I can see it is accepted by OMG :

But I can’t see a value bigger than 100 s

I set to 600000, it stay at 50 s

It is like it refuses to set a value bigger than 100 s.

Can you confirm the value is accepted even if I see another value on the OMG interface ?

@DigiH

Correct, 5 minutes is enough for T° Sensor.
For other sensor like a PIR, listening can be every time (I suppose listening is passive scanning).
Because I suppose a PIR will transmit is status directly if there is a change. It seems logic.

Remark : I suppose ‘{“intervalacts”:300000}’ is for the last version V1.4.0
And {“interval”:50000} is for precedent version.

1technophile · February 16, 2023, 6:46pm

As long as you see it in the broker is is accepted

DigiH · February 16, 2023, 6:50pm

{"interval":50000} is for passive scanning, so you would want that to 100 ms if you have a PIR - assuming that your PIR works fine with passive scanning. Adaptive scan would do that automatically now.

{"intervalacts":300000} is the additional separate interval for active scanning, which allows such sensors to be actively scanned at a lower frequency.

How did you set it to 600000? I just set mine to 1200000 and it sticks fine. Is it only a display issue in HA, what does it say in the BTtoMQTT message in MQTT Explorer? I can see 600000 in your screenshot above.

esa66 · February 17, 2023, 10:20am

Hi Florian,

I confirm it works based on a period of 10 minutes (I exaggerate to see it on the plot).

Now, I’ll check the battery discharge for a period of +/- 1 week

Thanks

esa66 · February 18, 2023, 3:24pm

Hi,

I make the upgrade to 1.4.0

I create a beacon with my phone to test the difference between active and passive scan.

I can see it with MQTT Explorer and it is updated at same frequency as SB Meter at rate of 60 sec (for test).

It should be transmitted and discovered into HA. But I don’t see new entity or device in HA.
My configuration is as follow :

{
“bleconnect”: false,
“interval”: 50000,
“adaptivescan”: false,
“intervalacts”: 61000,
“intervalcnct”: 3600000,
“onlysensors”: false,
“hasspresence”: false,
“presenceTopic”: “presence/”,
“presenceUseBeaconUuid”: true,
“minrssi”: -100,
“extDecoderEnable”: false,
“extDecoderTopic”: “undecoded”,
“filterConnectable”: false,
“pubadvdata”: true,
“pubBeaconUuidForTopic”: true,
“ignoreWBlist”: false,
“btqblck”: 1,
“btqsum”: 419,
“btqsnd”: 391,
“btqavg”: 1.071611
}

So, I decided to activate “Adaptive scan” and set
{“interval”:60000} for the beacon and {“intervalacts”:600000} for the Meter Plus.

In this case, it seems the SB Meter are scanned each minute.
I suppose the battery of Meter plus will be more solicited and I don’t want this.

I read the documentation about the new adaptive scan.
I try to understand how it works and differences between all mode.
What I understand : correct me

Braodcast : is a message send by the gateway to all devices to transmit their data. It is used by passive mode : Correct ?

passive scan scanning is controlled by {“interval”:60000}. It is for Beacon, PIR, Door status which require high rate of scanning (using broadcast ?).

active scan : controlled by {“intervalacts”:600000} . It is for Meter with a low scan period.

BLE connect controlled by intervalcnct. What is the specific need of this type of connection ?

Adaptive Scan
I understand that if it is deactivated, it will not do “passive passive” scan.
Below information’s is a bit contradicting ?

I also noticed that if I disable adaptive scan it always scan at rate of the “passive scan”.

And I can also see it is scanned as “active scan”

Remark :

In scan BLE, it is related to interval and not ‘intervalacts’ which is now for active scan
When Active scan is disable, period for both scan is set to minimum 100mS which can dry battery very quick if you don’t see it.

I have the same issue after a power lost. Is it possible to load the last configuration.
I know how to restore, {"load":true}, but if I’m not there it can run a lot of time.
Presence :
I can see the Meter are also scanned in the presence Topic. I though it was only for beacon ?

For my purpose, I only use Meter at scanning rate of 10 minutes.
So, I set :

Passive scan to 5 minutes : for test purpose, so I can see it also scan the Meter but without measure.
Active scan to 10 minutes to retrieve Measure from Meter.

A few questions :

This is not important for me as it was only for a test, but why I don’t see the iBeacon in HA
If I want a simple ibeacon not in real time. Is it enough to use “active scan” at the same rate as the Meter (+/- 10 minutes).
As it is for slow rate, can you set “active scan” in minute like you do for “Connect BT” ?

Have a nice day
Best regards.

DigiH · February 19, 2023, 1:49pm

I have propsed some additions to the documentation, which you can view at

github.com

DigiH/OpenMQTTGateway/blob/fab474d5bd18f22c3975f430382bd0621ff67528/docs/use/ble.md#advanced-publishing-advertisement-and-advanced-data-default-false

# BLE gateway

::: warning
We strongly encourage the use of a white-list (see below) so as to collect data from your devices only and not from other MAC addresses.
By default the gateway scans the advertizing BLE devices nearby with their MAC addresses. Depending on your country, it may be illegal to monitor networks for MAC addresses, especially on networks that you do not own. Please check your country's laws (for US Section 18 U.S. Code § 2511) - [discussion here](https://github.com/schollz/howmanypeoplearearound/issues/4).
:::

## Receiving signals from BLE beacon devices for Presence detection

Subscribe to all the messages with mosquitto or open your MQTT client software:

`    sudo mosquitto_sub -t +/# -v`

_NOTE: HM-10 or HM-11 module needed if you are not using ESP32; configure in `User_config.h`_

The BT gateway module for OpenMQTTGateway enables the detection of BLE beacons and their signal strength.  Generally BLE devices will not broadcast if they are paired so you may need to ensure your beacons is unpaired before it will be seen by the gateway.

If beacons are detected the gateway will periodically publish messages to MQTT (beacons must not be paired, see above):

```

This file has been truncated. show original

Let us know if this clears up your above confusion about adaptive, passive, active and continuous scanning, or if it possibly needs further information.

esa66 · February 24, 2023, 5:05pm

Hi,

I noticed the battery discharge a bit quick and noticed that adaptive scan and interval come back to 100mS

I don’t know what happens, but :

When you toggle the “Adaptive scan”, it reset both period scan to 100mS. I don’t do it.
When you restart (Button SYS Restart Gateway) or lost of current. Default period is 100mS

After setting again both scan period, I do a save : {“save”:true}
After a restart, I can restore with the command reload {“load”:true}.
How can I force to reload the saved value at restart ?
I want to be sure it stay at 10 minutes scan because 100mS is too quick and will discharge the battery.

Best Regard

DigiH · February 24, 2023, 5:10pm

Hi @esa66,

what happens if you don’t toggle adaptive scanning, leave "adaprivescan":false, set both interval and intervalacts to your liking, then {“save”:true}, and it should stick over any restart without having to load.

Do you, by any chance, also have a SwitchBot Moteion Sensor or SwitchBot Contact Sensor? As these do actually require constant active scanning (“intervalacts”:100).

esa66 · February 25, 2023, 10:22am

Hi,

I make a short video to show the issue.
Maybe it will show something I do wrongly.

Remark : After the SYS restart, I wrongly set the value again + save, but it is enough to make a “load”
This is what I make after the power restart.

Video

Best Regards

DigiH · February 25, 2023, 10:29am

Hi @esa66,

The video is not showing, only a still image - I think due to the restrictions of the forum software.

Could you try the latest development build, to see if you issue has been remedied with the latest changes, some made after your initial report.

Please let us know how it is working for you with this version.

For any videos to show properly they world need to be uploaded to YouTube or similar else and then linked to here.

esa66 · February 25, 2023, 10:41am

Hi,
I was still busy to upload a video, after I see the simple picture
Now, there is a link to download the video.
Can you check the same I do in your LAB ?
You are too quick

DigiH · February 25, 2023, 11:13am

Thanks for the video, and one thing which immediately strikes me as odd is the fact that you still have BT Active Scan present in your HA OMG gateway interface. A legacy property which should not be present any longer with adaptive scanning, and which I suspect might cause the reset to the 100ms for both interval and intervalacts after a restart.

I’m not that knowledgeable about the HA discovery and therefore HA interface options, so someone else will have to have a look at this oddity.

Just to see what happens, could you set your Active Scan switch to OFF the whole time, and see if it makes any difference?

Did you also try the above mentioned test development buidl, to see if it makes any changes for you?

esa66 · March 3, 2023, 8:06am

Hi digiH,
I have understood the shape for the gateway isn’t correctly updated.
I have disabled the “Active Scan”. I didn’t understand that it is no more active. It continue to scan with periodic request of 10 minutes

I make a sysrestart and noticed that default scan of 100mS is restored.

To reproduce the issue, you can do the following :
Topic : home/OpenMQTTGateway_ESP32_BLE/commands/MQTTtoBT/config

Step 1 : Save a new value : {“interval”:600001,“intervalacts”:600002,“save”:true}
Step 2 : Set a new value without save : {“interval”:600001,“intervalacts”:600003}
Step 3 : Check saved value : {“load”:true} ==> it restores “intervalacts”: 600002,
Step 4 : SYSRESTART ==> Default value of 100mS is activated : “intervalacts”: 100
Step 5 : Load saved value : {“load”:true}

Conclusion, After a restart, saved values are not restored.
Hope it help.

Next step :
I have decided to build a LAB environment for testing latest development build.
I’ll reinstall everything on Debian but it will take +/- 1 week.

Have a nice week-end

DigiH · March 3, 2023, 1:16pm

My problem is, that after step 4 (even without step 3) I get
…
“interval”:600001,
“adaptivescan”:false,
“intervalacts”:600002
…

as expected, without having to do step 5 to load anything. Admittedly with the development branch, as suggested above, which you can also install at

The issue remaining is, do you also have "adaptivescan":false, as otherwise you might well have a device being scanned which does require continuous active scanning, in which case adaptive scanning would correctly set the values back to "interval":100 and "intervalacts": 100.