Formatting/transformation integration

Hi Folks,

New to OpenMQTT. Plan on using it to send from radio frequency oil tank sensor (RTL 433) and forward it to my solar/tank monitoring software running on victron’s venus os.

At the moment I use a pi to capture the sensor data via rtl433 and use python to extract the depth data from the sensor. And then calculate the level in litres and remaining % of the tank before forwarding the raw values to the victron MQTT server.

I planned on ditching the pi and going for a lilygo instead. But that would mean i’d need to have some sort of middle man MQTT topic for OpenMQTT to use and then node red on the victron device to take the data from OpenMQTT topic, then perform the formatting and maths and foward to the tank topics on victron device.

My plan was implementing the formatting and multiple topic posting on the lilygo itself. And wondered if there was already a generalised implementation of this?

If not I was tempted to try generalise it myself.
ie have a settings.json file. Where i would store a list of devices that i want to format/calculations to be done before sending to MQTT channels. Something like

{[sensor1: { 
              "identification element": "data.id"
              "identification_method": "equals"
              "identification_match": "123456"
              "dataExtractKeys": ["data.depth","data.depth","data.battery"]
              "dataFormula": [var*4*5,var*4*5/1200,1]
              "mqttTopics": ["/W/VictronMQTT/tank/level","/W/VictronMQTT/tank/remaining","/W/VictronMQTT/tank/battery"]



]
}

The logic here when sensor data is received. The program loops through settings.json items. Extracts the identification_element using the keys data.id and if identification method is equals. it’ll say is the contents of data.id equal to 123456 ie the identification_match var. You could then implement other identification methods like contains.

if above identification is true for sensor then we continue
the dataExtract keys, Formula and mqttTopics are all linked. First data is extract from sensor dataExtractKeys the corresponding array formula is appled where var is replaced by the content of dataExtractKeys. Then finally the message is send to
BASETOPIC + /W/VictronMQTT/tank/level

Then the process is looped for the other dataExtractKeys, forumula and topics.

This is my best guess at implementing a generalised format/transformation function into openMQTT. Looking for guidance in terms of does this functionality already exist. And if there’s any flaws/issue with my logic other than thtis adding delay in sensor send times as the code would loop settings.json to check if the sensor data received is a device contained in settings.json

Hello,

There is no transformation system at the moment.
From my understanding, what is preventing you from doing the transformations on the Node-Red side instead of OMG?

Hi @1technophile

Thanks for the response. You’re right there isn’t anything stopping us doing that. A friend of mine has the setup you describe.

It’s just adding a middle man to the infrastructure (both the extra MQTT topic for OMG to post raw sensor data to and the need for node red to be running somewhere) and we were considering bypassing that and getting OMG to do the transformations/calcs.

Given the lack of interest I’ll probably just pull something together offline for myself and my friend but thought it was worth me asking. Just in case it was already a thing, and if not if there was any interest for such a feature even if in a different form.

Appreciate the response though, really an amazing bit of a kit you’ve made here!!!

Thanks, we have had the chance to have great contributions.
The transformation could be interesting and be part of the following types of functions but more towards the end I think (trying to sort them by value for most of the users)

  • filtering, general white list, black list
  • redirection, being able to direct messages to different communication module, DHTtoLORA, RTLtoLORA…
  • uC hosted rules

All the syntaxes and capabilities of this would need to be thought consistently.

Thanks,

I’ll keep an eye on how filtering is implemented. And I’ll follow the same layout for my settings file and share snips in case they’re ever useful.

I agree this would be a nice addition. I also use a Watchman with Home Assistant doing the calculation. The problem with this method for me is when Home Assistant is restarted it loses the previous calculation, the sensor is unavailable for a while and so until the next message is received the value is garbage. Watchman only transmits once every 30 minutes (it has very good battery life)

I’ve got a lilygo on order. Once its arrived I’ll post a fork and try align with the layout of filter branch to keep as consistent as possible.