Unable upgrade firmware by web server

gitacc · November 6, 2023, 12:33pm

Hello,

I attempted to upgrade the firmware through my web server but encountered a failure. According to the log message, my device was able to download the firmware.bin file from the web server. However, the firmware upgrade process could not be completed due to the error message “Magic header does not start with 0xE9”.

What should I do to handle this issue?

Many thanks.

Log message
[ 97368][D][HTTPClient.cpp:303] beginInternal(): protocol: http, host: www.myhost.com port: 80 url: /firmware.bin
[ 97919][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0
[ 98201][D][HTTPClient.cpp:1156] connect(): connected to www.myhost.com:80
[ 98516][D][HTTPClient.cpp:1307] handleHeaderResponse(): code: 200
[ 98516][D][HTTPClient.cpp:1310] handleHeaderResponse(): size: 850
[ 98516][D][HTTPClient.cpp:628] sendRequest(): sendRequest code=200

[ 98523][D][zzHTTPUpdate.cpp:218] handleUpdate(): Header read fin.
[ 98529][D][zzHTTPUpdate.cpp:219] handleUpdate(): Server header:
[ 98535][D][zzHTTPUpdate.cpp:220] handleUpdate(): - code: 200
[ 98540][D][zzHTTPUpdate.cpp:221] handleUpdate(): - len: 850
[ 98546][D][zzHTTPUpdate.cpp:227] handleUpdate(): ESP32 info:
[ 98552][D][zzHTTPUpdate.cpp:228] handleUpdate(): - free Space: 3342336
[ 98558][D][zzHTTPUpdate.cpp:229] handleUpdate(): - current Sketch Size: 1476736
[ 98783][D][zzHTTPUpdate.cpp:281] handleUpdate(): runUpdate flash…
[ 98783][E][zzHTTPUpdate.cpp:298] handleUpdate(): Magic header does not start with 0xE9
[ 98786][D][HTTPClient.cpp:388] disconnect(): still data in buffer (850), clean up.

1technophile · November 6, 2023, 6:15pm

Hello,

Could you give more details:

environment
version used
target version
command used to update

gitacc · November 8, 2023, 7:59am

Hello,

Thanks for reply.

My objective is to enable firmware upgrades for my device through my web server in the future. To test this functionality, I initially flashed my ESP32 device. Subsequently, I attempted to upgrade the firmware using the same firmware.bin file.

For the firmware upgrade process, I changes OTA URL in my device’s WebUI. Specifically, I modified them to point to the address of my web server: http://www.myhost.com/firmware.bin.

Here’s the environment of my device.

[env:esp32s3-dev-c1-ble]
platform = ${com.esp32_platform}
board = esp32-s3-devkitc-1
lib_deps =
${com-esp32.lib_deps}
${libraries.ble}
${libraries.decoder}
${libraries.adafruit_neopixel}
build_flags =
${com-esp32.build_flags}
‘-DZgatewayBT=“BT”’
‘-DANEOPIX_IND_DATA_GPIO=15’
‘-DANEOPIX_BRIGHTNESS=5’
‘-DANEOPIX_IND_NUM_LEDS=1’
‘-DRGB_INDICATORS=true’
‘-DNO_INT_TEMP_READING=true’
'-DGateway_Short_Name=“BLE_GATEWAY
‘-DWifiManager_password=“admin12345”’
'-DWEBUI_LOGIN=“login”’
‘-Dota_password=“password”’
'-DBase_Topic=“ble-gw/”’
‘-DMQTTDecodeTopic=“undecoded”’
‘-DpubBLEAdvData=true’
custom_description = BLE gateway on the S3

1technophile · November 8, 2023, 6:53pm

Hello,

Could you try with the MQTT command instead:

gitacc · November 10, 2023, 10:40am

Hello,

Thank you for your help.

I referred to “Firmware update from MQTT (ESP only)” and conducted testing using the command provided. The outcome was similar to performing a firmware upgrade through the WebUI.

Firmware upgrade command
mosquitto_pub -t “BLE_GATEWAY/84FCE6644420/commands/MQTTtoSYS/firmware_update” -m ‘{“password”: “password”, “url”: “http:/ /www.myhost.com/firmware.bin”}’

Log message
[444066][D][HTTPClient.cpp:303] beginInternal(): protocol: http, host: www. myhost.com port: 80 url: /firmware.bin
[444615][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0
[445687][D][HTTPClient.cpp:1156] connect(): connected to www .myhost.com:80
[446001][D][HTTPClient.cpp:1307] handleHeaderResponse(): code: 200
[446001][D][HTTPClient.cpp:1310] handleHeaderResponse(): size: 850
[446001][D][HTTPClient.cpp:628] sendRequest(): sendRequest code=200

[446008][D][zzHTTPUpdate.cpp:218] handleUpdate(): Header read fin.
[446014][D][zzHTTPUpdate.cpp:219] handleUpdate(): Server header:
[446020][D][zzHTTPUpdate.cpp:220] handleUpdate(): - code: 200
[446025][D][zzHTTPUpdate.cpp:221] handleUpdate(): - len: 850
[446031][D][zzHTTPUpdate.cpp:227] handleUpdate(): ESP32 info:
[446037][D][zzHTTPUpdate.cpp:228] handleUpdate(): - free Space: 3342336
[446043][D][zzHTTPUpdate.cpp:229] handleUpdate(): - current Sketch Size: 1476736
[446268][D][zzHTTPUpdate.cpp:281] handleUpdate(): runUpdate flash…
[446268][E][zzHTTPUpdate.cpp:298] handleUpdate(): Magic header does not start with 0xE9
[446271][D][HTTPClient.cpp:388] disconnect(): still data in buffer (850), clean up.

1technophile · November 10, 2023, 2:11pm

If you go to the wifi manager portal and update your device with it (and the same firmware.bin) does it work ?

gitacc · November 11, 2023, 11:33am

Hello,

Thanks.

I built the project and successfully flashed it onto my device. The firmware is functioning normally on my device.

Subsequently, I attempted to upgrade the firmware using the same firmware.bin file through both the WebUI and firmware update from MQTT. However, during the upgrade process, I encountered an error message stating “Magic header does not start with 0xE9” in both cases.

Log message - Upgrade firmware through WebUI
[103833][D][HTTPClient.cpp:303] beginInternal(): protocol: http, host: www.myhost .com port: 80 url: /firmware.bin
[104383][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0
[104669][D][HTTPClient.cpp:1156] connect(): connected to www.myhost .com:80
[104983][D][HTTPClient.cpp:1307] handleHeaderResponse(): code: 200
[104983][D][HTTPClient.cpp:1310] handleHeaderResponse(): size: 844
[104983][D][HTTPClient.cpp:628] sendRequest(): sendRequest code=200

[104990][D][zzHTTPUpdate.cpp:218] handleUpdate(): Header read fin.
[104996][D][zzHTTPUpdate.cpp:219] handleUpdate(): Server header:
[105002][D][zzHTTPUpdate.cpp:220] handleUpdate(): - code: 200
[105007][D][zzHTTPUpdate.cpp:221] handleUpdate(): - len: 844
[105013][D][zzHTTPUpdate.cpp:227] handleUpdate(): ESP32 info:
[105019][D][zzHTTPUpdate.cpp:228] handleUpdate(): - free Space: 3342336
[105025][D][zzHTTPUpdate.cpp:229] handleUpdate(): - current Sketch Size: 1476736
[105250][D][zzHTTPUpdate.cpp:281] handleUpdate(): runUpdate flash…
[105250][E][zzHTTPUpdate.cpp:298] handleUpdate(): Magic header does not start with 0xE9
[105253][D][HTTPClient.cpp:388] disconnect(): still data in buffer (844), clean up.

Firmware upgrade command
mosquitto_pub -t “BLE_GATEWAY/84FCE6644420/commands/MQTTtoSYS/firmware_update” -m ‘{“password”: “password”, “url”: “http:/ /www.myhost.com/firmware.bin”}’

Log message - Upgrade firmware from MQTT
[831146][D][HTTPClient.cpp:303] beginInternal(): protocol: http, host: www.myhost .com port: 80 url: /firmware.bin
[831693][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0
[832048][D][HTTPClient.cpp:1156] connect(): connected to www. myhost .com:80
[832362][D][HTTPClient.cpp:1307] handleHeaderResponse(): code: 200
[832362][D][HTTPClient.cpp:1310] handleHeaderResponse(): size: 844
[832362][D][HTTPClient.cpp:628] sendRequest(): sendRequest code=200

[832369][D][zzHTTPUpdate.cpp:218] handleUpdate(): Header read fin.
[832375][D][zzHTTPUpdate.cpp:219] handleUpdate(): Server header:
[832381][D][zzHTTPUpdate.cpp:220] handleUpdate(): - code: 200
[832386][D][zzHTTPUpdate.cpp:221] handleUpdate(): - len: 844
[832392][D][zzHTTPUpdate.cpp:227] handleUpdate(): ESP32 info:
[832398][D][zzHTTPUpdate.cpp:228] handleUpdate(): - free Space: 3342336
[832404][D][zzHTTPUpdate.cpp:229] handleUpdate(): - current Sketch Size: 1476736
[832629][D][zzHTTPUpdate.cpp:281] handleUpdate(): runUpdate flash…
[832629][E][zzHTTPUpdate.cpp:298] handleUpdate(): Magic header does not start with 0xE9
[832632][D][HTTPClient.cpp:388] disconnect(): still data in buffer (844), clean up.

1technophile · November 11, 2023, 12:41pm

Hello,
Wifi Manager is the configuration portal that you get at the start, it has an option on the first page to “update”. This is a different mechanism than the 2 above.
Worth a try to check if it comes from a firmware issue.

gitacc · November 11, 2023, 1:45pm

Hello,

Sorry for my misunderstanding.

I attempted to upgrade my device firmware from two different sources

Original build file (firmware file that was compiled from the project)
Firmware file downloaded from my web server

I was able to successfully upgrade the firmware using both sources through the WiFi Manager.

1technophile · November 11, 2023, 2:12pm

When you put a GitHub url of a firmware file like
https://github.com/1technophile/OpenMQTTGateway/releases/download/v1.6.0/esp32s3-dev-c1-ble-firmware.bin

Does it work ?

gitacc · November 11, 2023, 6:00pm

Hello,

I tried to upgrade the firmware using both the WebUI and WiFi Manager, utilizing the esp32s3-dev-c1-ble-firmware.bin file.

WebUI was unable to upgrade the firmware using the provided URL (https://github.com/1technophile/OpenMQTTGateway/releases/download/v1.6.0/esp32s3-dev-c1-ble-firmware.bin).

WiFi Manager was able to successfully upgrade the firmware.

Log message - Firmware upgrade by WebUI
[ 96888][D][HTTPClient.cpp:303] beginInternal(): protocol: https, host: github.com port: 443 url: /1technophile/OpenMQTTGateway/releases/download/v1.6.0/esp32s3-dev-c1-ble-firmware.bin
[ 97446][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0

Guru Meditation Error: Core 1 panic’ed (Unhandled debug exception).
Debug exception reason: Stack canary watchpoint triggered (loopTask)
Core 1 register dump:
PC : 0x40382b87 PS : 0x00060036 A0 : 0x80381d03 A1 : 0x3fce9fc0
A2 : 0x3fc99b30 A3 : 0xb33fffff A4 : 0x0000cdcd A5 : 0x00060023
A6 : 0x00060023 A7 : 0x0000abab A8 : 0xb33fffff A9 : 0xffffffff
A10 : 0x3fcb4f1c A11 : 0x00000100 A12 : 0x3fca8d98 A13 : 0x3fcea070
A14 : 0x02c99b30 A15 : 0x00ffffff SAR : 0x00000006 EXCCAUSE: 0x00000001
EXCVADDR: 0x00000000 LBEG : 0x400570e8 LEND : 0x400570f3 LCOUNT : 0x00000000

Backtrace: 0x40382b84:0x3fce9fc0 0x40381d00:0x3fcea000 0x403803bf:0x3fcea020 0x40380460:0x3fcea060 0x40379249:0x3fcea080 0x40379321:0x3fcea0b0 0x420cb70d:0x3fcea0d0 0x420cb727:0x3fcea0f0 0x420a6d7f:0x3fcea110 0x420a51d5:0x3fcea130 0x420a5295:0x3fcea160 0x420a00bc:0x3fcea180 0x420a021d:0x3fcea1a0 0x4209fd3f:0x3fcea210 0x4209b80f:0x3fcea230 0x4209c390:0x3fcea2a0 0x4209cde7:0x3fcea490 0x4209d131:0x3fcea5f0 0x4209d158:0x3fcea620 0x4209b407:0x3fcea680 0x4209b746:0x3fcea720 0x4209b76e:0x3fcea760 0x420a0e71:0x3fcea780 0x420a0ea3:0x3fcea7a0 0x420a0abd:0x3fcea870 0x420a0ada:0x3fcea8a0 0x420a0c45:0x3fcea8d0 0x420a7929:0x3fcea920 0x420a7b46:0x3fceaa40 0x420947bf:0x3fceaa70 0x42095c56:0x3fceaac0 0x42093ddd:0x3fceab60 0x42093e17:0x3fceab80 0x42045b4d:0x3fceaba0 0x42044bd2:0x3fceae30 0x42044c87:0x3fceae80 0x42044d56:0x3fceaec0 0x420f5499:0x3fceaee0 0x4204771d:0x3fceaf00 0x42048d29:0x3fceaf50 0x4204903f:0x3fceafd0 0x4201846f:0x3fceaff0 0x42018b57:0x3fceb080 0x4201389a:0x3fceb180 0x42013dad:0x3fceb570 0x420f3d11:0x3fcec220 0x4202061a:0x3fcec240 0x420206c1:0x3fcec260 0x4202079e:0x3fcec2a0 0x42020951:0x3fcec310 0x420149ee:0x3fcec360 0x42017b5b:0x3fcec390 0x4204d1d1:0x3fcec3d0

ELF file SHA256: c8942dbbf5904be8

gitacc · November 13, 2023, 6:38am

I have conducted some tests for my case and my device can upgrade its firmware using my web server.

Thanks for helping.

Log message
[ 44702][D][HTTPClient.cpp:303] beginInternal(): protocol: http, host: www.myhost.com port: 80 url: /firmware.bin
[ 45253][D][HTTPClient.cpp:598] sendRequest(): request type: ‘GET’ redirCount: 0
[ 45539][D][HTTPClient.cpp:1156] connect(): connected to www.myhost.com:80
[ 45854][D][HTTPClient.cpp:1307] handleHeaderResponse(): code: 200
[ 45854][D][HTTPClient.cpp:1310] handleHeaderResponse(): size: 1477808
[ 45855][D][HTTPClient.cpp:628] sendRequest(): sendRequest code=200

[ 45861][D][zzHTTPUpdate.cpp:218] handleUpdate(): Header read fin.
[ 45867][D][zzHTTPUpdate.cpp:219] handleUpdate(): Server header:
[ 45873][D][zzHTTPUpdate.cpp:220] handleUpdate(): - code: 200
[ 45879][D][zzHTTPUpdate.cpp:221] handleUpdate(): - len: 1477808
[ 45885][D][zzHTTPUpdate.cpp:227] handleUpdate(): ESP32 info:
[ 45891][D][zzHTTPUpdate.cpp:228] handleUpdate(): - free Space: 3342336
[ 45897][D][zzHTTPUpdate.cpp:229] handleUpdate(): - current Sketch Size: 1476720
[ 46122][D][zzHTTPUpdate.cpp:281] handleUpdate(): runUpdate flash…
[ 46122][D][Updater.cpp:133] begin(): OTA Partition: app1
[126525][D][zzHTTPUpdate.cpp:325] handleUpdate(): Update ok

1technophile · November 13, 2023, 11:54am

Did you identified what was not working?

gitacc · November 13, 2023, 1:22pm

Not entirely sure but it seems track to web server.

I made modifications to the “zzHTTPUpdate.cpp” file to dump TCP data, approximately 1KB in size.

Upon comparing this data with my firmware.bin file, it was differences. I then converted the TCP data to ASCII. The web server’s response that “Requires JavaScript to work” when I attempted to upgrade firmware.

After the web server was changed, my device was able to upgrade its firmware through web server.