FastLED seems broken for ESP32C3 boards

mrickma · March 8, 2023, 9:16am

Let me report my first steps with OpenMQTTGateway on an ESP32-C3-DevKitC-02 board from Espressiv. The board has an ESP32-C3-WROOM-02 module and a WS2812 RGB LED driven by GPIO8. I have chosen OMG’s esp32c3-dev-m1-ble environment to start with. I added TRIGGER_GPIO=7 for a pushbutton which I like to have for resetting WIFI and MQTT server data, and changed the Gateway_Name. Seemingly everything went well and the on board LED changed between off, green and sometimes blue. However details looked bad.
The MQTT log cycled along, first LWT online. Then there were many status messages followed by several messages about devices, only recognized or well decoded. Then the C3 went offline and the sequence started again. Time between offline events was 15 seconds on average. Maximum SYStoMQTT uptime was 3. The serial log showed the same frequency of:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)

followed by register dump, stack dump, Rebooting… and finally “WELCOME TO OpenMQTTGateway”. The details in the log of the core dump were really nice and reminded me of old gdb times.
OpenMQTTGateway got going on other ESP32C3 boards when omiting the FastLED library. So I tried to get its Blink example going. The same "Guru Meditation Error"s, dumps and reboots now happened about every second. I finally got my LED going in a small rainbow cycle program lib-depending on adafruit/Adafruit NeoPixel@^1.11.0 library. Overall I assume that FastLED seems broken for ESP32C3 boards.

Next step was removing the FastLED library and related directives from my OMG _env.ini. This resulted in a flood of error messages from the firmware interspersed with few log messages from OMG. Fortunately during WIFI setup it happend slowly, e.g.:

[21:12:43.806163 0.004583] E (97236) gpio: gpio_set_level(226): GPIO output gpio_num error
[21:12:43.810240 0.004078] W: MQTT connection...
[21:12:46.813463 3.003218] W: failure_number_mqtt: 8

This led me to User_config.h where I recognized that OpenMQTTGateway requires setting indicator LED declararations. If not given it defaults to GPIO 40 for ESP32. This pin is not present on ESP32C3. So I added dummy but valid GPIO declararations for LED_INFO, LED_SEND_RECEIVE and LED_ERROR to my build environment. The firmware errors are gone now. In the MQTT log SYStoMQTT uptime is > 42000 at present.

mrickma · March 8, 2023, 8:09pm

The ESP32-C3-DevKitC-02 got unresposive after the well known “assert rwble.c 261, param 00020000 00000000” error, discovered by @argafal. Maximum SYStoMQTT uptime was 77524 (> 21h), no reboots inbetween. Next step is upgrading the platform to espressif32@5.3.0 .

mrickma · March 10, 2023, 7:29pm

I switch the ESP32-C3-DevKitC-02 off now. Its latest uptime was 119884 (>33h), no reboots or errors. I regard the board stable for OMG without FastLED but dummy signalling LED directives included in the build configuration.
Back to the topic, my next step is to get the onboard RGB LED going for OMG signalling without FastLED using the “adafruit/Adafruit NeoPixel” library. For that User_config.h seems to be the right place. I have to write a variable definition and several preprocessor directives. I would start in line 460 of my User_config.h with an “#elif RGB_INDICATORS == 2”.
Is there a preferable way for enabling the RGB LED?

argafal · March 10, 2023, 7:49pm

@mrickma Fun to read about your NeoPixel idea. I’ll be curious to hear how it goes! The Lolin C3 mini has a NeoPixel on-board too.

mrickma · March 12, 2023, 9:21am

I added the “adafruit/Adafruit NeoPixel” library to User_config.h for OMG signalling . At first the result was a bit disappointing. The LED on the ESP32-C3-DevKitC-02 showed only green light (INFO LED). After stopping mosquitto it looked better, a slow red pulse (ERROR LED). With green dimmed I can see intermittent blue flashes (SEND RECEIVCE LED) now.
@argafal Can you test my code on your Lolin board? You can access my changes at GitHub. I also pushed a copy of my OMG build environments. At the very bottom of env_prod.ini you find how I added library and build flags. See you.

argafal · March 12, 2023, 2:01pm

@mrickma What a nice addition. I am testing it right now.

The first impression is good. I renamed your env_prod.ini to prod_env.ini so that is found by pio. I updated the pin definition for the NeoPixel to match the board: '-DANEOPIX_IND_DATA_GPIO=7'. I also changed the trigger pin to 9. The LED shows a status depending on MQTT connection (blue during boot, red pulse if unconnected, green if connected). Very nice!
I cannot really see the blue flashes at all. Maybe they would need to be a little bit longer, or the green a bit dimmer while keeping blue at the same level. maybe green overrides blue too quickly by accident, I don’t know.
If I stop the MQTT server, the NeoPixel blinks red. Good. However, if I restart the MQTT server, OMG does not always reconnect until I manually trigger a reboot by pressing the RST button. Odd.
OMG died randomly. This may or may not be related to your changes, I don’t know. The error message on the serial output was [185110][E][WiFiClient.cpp:268] connect(): socket error on fd 49, errno: 113, "Software caused connection abort". OMG tried to reconnect, but it never came back, full log of the serial output at the end of this post. A manual reboot (RST button) brought it back to life.

Reconnecting to /dev/ttyACM3     Connected!
E (1402) wifi:sta is connecting, return error
[  1395][E][WiFiSTA.cpp:317] begin(): connect failed! 0x3007
E (1443) wifi:Set status to INIT
[  3908][E][Preferences.cpp:483] getString(): nvs_get_str len fail: BTConfig NOT_FOUND
[  4996][E][esp32-hal-misc.c:128] disableCore0WDT(): Failed to remove Core 0 IDLE task from WDT
[185110][E][WiFiClient.cpp:268] connect(): socket error on fd 49, errno: 113, "Software caused connection abort"
E (207178) wifi:Association refused temporarily, comeback time 1024 mSec
E (208209) wifi:Set status to INIT
E (251264) wifi:Association refused temporarily, comeback time 1024 mSec
E (252296) wifi:Set status to INIT
E (263474) wifi:Association refused temporarily, comeback time 1024 mSec
E (264508) wifi:Set status to INIT
E (331435) wifi:Association refused temporarily, comeback time 1024 mSec
E (335449) wifi:Association refused temporarily, comeback time 1024 mSec
E (336479) wifi:Set status to INIT

argafal · March 12, 2023, 2:02pm

P.S. Maybe instead of blinking blue upon receiving a message a brief “turning off the green” status would be more visible?

argafal · March 12, 2023, 3:35pm

I observed the C3 board and an MQTT listener at the same time for a while. The blue flash is a hit and miss.

Sometimes I can see it really clearly. That is often when there are multiple messages at the same time.
Sometimes there is a message but no blue flash at all. That is often when it is a single message.

mrickma · March 12, 2023, 9:14pm

@argafal Thanks a lot for testing !!! The principle errors in your log are from “framework-arduinoespressif32”, i.e. very far down. I have no idea how to avoid them.
What I have done is writing some definitions in C/C++ code. They are not very elaborate and positioned in a header file. When OMG is up and wants to signal the work is done by two lines of functions: the color is xyz, show it. These functions just tell the LED library what OMG wants.
We both realized that the blue LED is only shortly seen or not at all. I think that the send / receive times are short and sometimes too short for us to perceive in between the green (human color vision). A blue island in the green ocean? Perhaps I should change colors.
I think that the errors which you reported are really decisive. Bit-banging an RGB LED adds another task which the framework may not reliably handle on ESP32C3 socs. Anyhow, I will start a long time test with my setup and see wether I get your errors. If yes, then it was a nice try. See you.

mrickma · March 14, 2023, 8:47am

@argafal As I am writing this, the ESP32-C3-DevKitC-02 got beyond 125000 uptime (>34h). The RGB led continues to show green with short intermittent blue send/receive flashes. There is no sign of reboot in the logs. The serial log shows just two errors within its first second of recording. They seem setup related and harmless. I really wonder about the difference in errors between your and my test. Of course we have different boards. But the ESP32C3 and build environments should be about the same. There may be a Good Witch around here.
My next steps are: testing visibility of color changes, need of serial output to be read and of course a long term run without serial grabbing. See you.

mrickma · March 19, 2023, 6:53pm

I have changed colors for better visibility: gold (send/receive) on a dimmed blue background (info) suits me best.
With a short test program I found that the ESP32-C3-DevKitC-02 did not depend on reading serial output. It has a CP210x UART Bridge The Lolin and Seed ESP21C3 boards recently tested here do USB/serial conversion internal to the soc. That my explain the difference.
In my next long time test the DevKitC was just connected to power supply with serial logging intact. I stopped the board after >42h. There were two reboots, no crash. For me that is tolerable but open to improvement.
@1technophile @argafal: Now the DevKitC was connected to power supply, had logging disabled and was using the espressif32@6.1.0 platform. Uptime increased steadily beyond 43h, no reboots and alike. I am really satisfied.
You can find my changes in the wroom32c3 branch of my OMG fork at github. I wonder whether changes like that adapted for the “Bump ESP32 platform” would be acceptable for OMG.
I have also tested OMG’s esp32c3-dev-m1-ble environment (uses FastLED) again, but now using the espressif32@6.1.0 platform. I still get the “Guru Meditation Error: Core 0 panic’ed (Interrupt wdt timeout on CPU0)”, followed by core dump and reboot every 16 sec. So my verdict remains that OMG currently can not use FastLED for the Espressif ESP32-C3-DevKitC-02. See you.

argafal · March 19, 2023, 7:05pm

@mrickma: I ~~want~~ am going to test your branch/new colours. I’m excited to see it!

~~Two questions:~~

~~Which environment contains the correct definitions?~~
Found it: esp32c3-dev-c2-ble
~~Is there still a parameter to dim the LED? You had it in the previous and I quite liked it. I cannot see it right now, but I might be lost in the many different environment definitions.~~
Time for testing it first

mrickma · March 19, 2023, 7:39pm

They are the esp32c3-dev-c2-ble and env:esp32c3-dev-c2-ble-no-serial environments. I have just fixed a naming bug in the latter. For simplicity:

[env:esp32c3-dev-c2-ble]
platform = espressif32@6.1.0  ; override
board = esp32-c3-devkitm-1
board_build.partitions = min_spiffs.csv
lib_deps =
  ${com-esp.lib_deps}
  ${libraries.wifimanager32}
  ${libraries.ble}
  ${libraries.decoder}
  adafruit/Adafruit NeoPixel@^1.11.0
build_flags =
  ${com-esp.build_flags}
  '-DZgatewayBT="BT"'
  '-DANEOPIX_IND_DATA_GPIO=8'
  '-DANEOPIX_IND_NUM_LEDS=1'
 ; '-DANEOPIX_BRIGHTNESS=10'  ; 0 - 255, default 20
  '-DRGB_INDICATORS=2'
  '-DNO_INT_TEMP_READING=true' ; No internal temperature reading on ESP32 C3 or S3
  '-DTRIGGER_GPIO=7'
  '-DGateway_Name="OpenMQTTGateway_ESP32C3_DKC02"'
custom_description = BLE gateway on Espressiv ESP32-C3-DevKitC-02

[env:esp32c3-dev-c2-ble-no-serial]
extends = env:esp32c3-dev-c2-ble
build_flags =
  ${env:esp32c3-dev-c2-ble.build_flags}
  '-DLOG_LEVEL=LOG_LEVEL_SILENT'
  '-DWM_DEBUG_LEVEL=0'
custom_description = BLE gateway on Espressiv ESP32-C3-DevKitC-02 without logging

Thank you for testing! See you.

argafal · March 19, 2023, 7:43pm

@mrickma
First of all, I really like your branch. How neat to have the little LED working, makes me happy to see it blink. Nicely done!

Could you integrate the following changes in environment.ini? With them the NeoPixel on Lolin C3 mini works out of the box:

diff --git a/environments.ini b/environments.ini
index 74b2416..9979122 100644
--- a/environments.ini
+++ b/environments.ini
@@ -1582,16 +1582,15 @@ lib_deps =
   ${libraries.wifimanager32}
   ${libraries.ble}
   ${libraries.decoder}
+; 07 Mar 2023: usage FastLED results in nearly immediate reboots
+; therefore the Adafruit NeoPixel library is used for the RGB LED
+  adafruit/Adafruit NeoPixel@^1.11.0
 build_flags =
   ${com-esp.build_flags}
   '-DZgatewayBT="BT"'
-; FastLED must be disabled. FastLED is not compatible with this board at the current moment.
-; The LED pins must still be defined, to avoid an error loop on pin definitions.
-; These are dummy definitions and are not actually used. Any choice 2 - 10, 20, 21 should be fine.
-; See https://community.openmqttgateway.com/t/watchdog-kills-omg-on-esp32-c3-within-seconds/2192/15
-  '-DLED_SEND_RECEIVE=10'
-  '-DLED_INFO=6'
-  '-DLED_ERROR=7'
+  '-DANEOPIX_IND_DATA_GPIO=7'
+  '-DANEOPIX_IND_NUM_LEDS=1'
+ ; '-DANEOPIX_BRIGHTNESS=10'  ; 0 - 255, default 20
+  '-DRGB_INDICATORS=2'        ; 2 = use Adafruit NeoPixel library
 ; The momentary switch on the board is connected to GPIO9
   '-DTRIGGER_GPIO=9'
   '-DNO_INT_TEMP_READING=true' ; No internal temperature on ESP32 C3 or S3

For the colours: I tried a few other combinations to satisfy my curiosity. I appreciate the effort you put into finding those two colours (blue and gold) to work together. It’s really not easy at all to find a good pair!

Personally, however, I’m not a big fan of the blue as the base tone. To me the green said “success” so very clearly and felt easier to intuitively understand. I also find blue to be a bit intrusive. I tried the following: a dim green as the base to show that the device is active. And a brighter green flash when receiving/sending data. To me it looked quite convincing. But of course I can see how this is totally a matter of taste and how you might want to keep blue/gold!

One other small proposal: I noticed that the LED stays blue (green) while rebooting the device. I would have expected it to be red. So I added two lines to initialize the LED colour. Full diff below.

diff --git a/main/User_config.h b/main/User_config.h
index 8bdca76..3d8bc63 100644
--- a/main/User_config.h
+++ b/main/User_config.h
@@ -492,16 +492,17 @@ Adafruit_NeoPixel leds(ANEOPIX_IND_NUM_LEDS, ANEOPIX_IND_DATA_GPIO, ANEOPIX_LED_
 // no ANEOPIX_CRITICAL_LED and FASTLED_IND_DATA_GPIO2 with Adafruit_NeoPixel
 // preprocessor calculates color values
 #  define ANEOPIX_RED ((0x3F * ANEOPIX_BRIGHTNESS) >> 8) << 16          // dimmed /4
-#  define ANEOPIX_GOLD (((0xFF * ANEOPIX_BRIGHTNESS) >> 8) << 16) | \
-                         (((0xD7 * ANEOPIX_BRIGHTNESS) >> 8) << 8)
-#  define ANEOPIX_GREEN ((0xFF * ANEOPIX_BRIGHTNESS) >> 8) << 8
+#  define ANEOPIX_GREEN ((0xFF * ANEOPIX_BRIGHTNESS) >> 8) << 8         // bright green = sending
+#  define ANEOPIX_GREEN_DIM ((0x3F * ANEOPIX_BRIGHTNESS) >> 8) << 8         // dimmed /4
 #  define ANEOPIX_BLUE (0x3F * ANEOPIX_BRIGHTNESS) >> 8                 // dimmed /4
 #  define ANEOPIX_BLACK 0
 
 #  define SetupIndicators()                      \
     pinMode(RGB_LED_POWER, OUTPUT);              \
     digitalWrite(RGB_LED_POWER, HIGH);           \
-    leds.begin();
+    leds.begin();                                \
+    leds.setPixelColor(ANEOPIX_ERROR_LED, ANEOPIX_RED);    \
+    leds.show();
 #  define ErrorIndicatorON()                               \
     leds.setPixelColor(ANEOPIX_ERROR_LED, ANEOPIX_RED);    \
     leds.show();
@@ -509,13 +510,13 @@ Adafruit_NeoPixel leds(ANEOPIX_IND_NUM_LEDS, ANEOPIX_IND_DATA_GPIO, ANEOPIX_LED_
     leds.setPixelColor(ANEOPIX_ERROR_LED, ANEOPIX_BLACK);  \
     leds.show();
 #  define SendReceiveIndicatorON()                                \
-    leds.setPixelColor(ANEOPIX_SEND_RECEIVCE_LED, ANEOPIX_GOLD);  \
+    leds.setPixelColor(ANEOPIX_SEND_RECEIVCE_LED, ANEOPIX_GREEN);  \
     leds.show();
 #  define SendReceiveIndicatorOFF()                               \
     leds.setPixelColor(ANEOPIX_SEND_RECEIVCE_LED, ANEOPIX_BLACK); \
     leds.show();
 #  define InfoIndicatorON()                              \
-    leds.setPixelColor(ANEOPIX_INFO_LED, ANEOPIX_BLUE); \
+    leds.setPixelColor(ANEOPIX_INFO_LED, ANEOPIX_GREEN_DIM); \
     leds.show();
 #  define InfoIndicatorOFF()                             \
     leds.setPixelColor(ANEOPIX_INFO_LED, ANEOPIX_BLACK); \

argafal · March 19, 2023, 7:45pm

P.S. What does '-DRGB_INDICATORS=2' do, and why is it necessary to have?

mrickma · March 19, 2023, 7:59pm

In OMG DRGB_INDICATORS=true is used. In C/C++ definition “true” translates into “1”. So setting it to “n” offers other possibilities. “2” in this case means use adafruit/Adafruit NeoPixel library for the RGB LED. See you.
edit: C++ definitons: false = 0, true = 1
so your choice >1

argafal · March 19, 2023, 8:13pm

Got it. That might deserve a comment in the environment defintions:

  '-DRGB_INDICATORS=2' ; 2 = use Adafruit NeoPixel library

mrickma · March 19, 2023, 8:37pm

You are right. I have just added it. See you.

mrickma · March 20, 2023, 10:15am

@argafal I apologize for being a bit slow. When committing your first diff I would like to write “authored by <your email>” in the message. May I take the address which I found at github for the original Lolin commit?
In your next diff you are going for the color of hope, whereas my colors were dominated by trust. Additionally there may be an official OMG color scheme which I have not found yet. I would like to make color schemes adjustable. Your green scheme may become default. For the blue scheme of trust one would have to use a -DANEOPIX_COLOR_SCHEME=1 directive during build.
Your addition to the “SetupIndicators()” is really sensible. I would like to see reboots too. Give me some time. See you.

argafal · March 20, 2023, 10:29am

@mrickma: Yes. You can use the one from my lolin_c3_mini pull request. They way I see it, you do not need to mention me: I copied & pasted those lines directly form you, so all credit should go your direction! Do as you see fit. For the colours, please choose as you like them best. You know, I’m just happy to see the little NeoPixel shine. Awesome addition to the project! I really like that you made that effort. See you.