Hardware Flash Corruption Issue

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Sun Jan 23, 2022 9:36 am

Hello WiFive or ESP_Sprite,

Do you have any idea like from where can i get checksum errors? like is there any source from where i can get it? Because i have checked into boot loader source code but couldn't found any relevant code for the same

Let me know if you have any idea regarding the source code or it is coming from pre-compiled library.
Regards,
Ritesh Prajapati

desp32fun
Posts: 7
Joined: Sat Jan 22, 2022 10:44 am

Re: Hardware Flash Corruption Issue

Postby desp32fun » Sun Jan 23, 2022 10:50 am

This is how the problem looks at our failing modules.

Code: Select all

ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x0780000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46


... and so on ...

desp32fun
Posts: 7
Joined: Sat Jan 22, 2022 10:44 am

Re: Hardware Flash Corruption Issue

Postby desp32fun » Sun Jan 23, 2022 11:15 am

Ritesh, could you please scan the QR Code on your faulty WROVER modules?
Simply use any qr code scanner software. You will receive a 18-digit number.

The first 2 digits tell you the fab,
the next 4 digits are the manufacturing date YYWW
and the rest is just a MAC address.

Then when you have it, we might find out if it's a suspicious batch, or if the problem is completely random.
Since we both surely did not use WROVER Modules from the same reel, it might give some clue.
I see failed modules from fab 09, Year 21, week 20. But since we don't use the modules for very long, this may
as well be a coincidence. That's why it would be nice to counter-check that with your modules.

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Sun Jan 23, 2022 11:34 am

desp32fun wrote:
Sun Jan 23, 2022 10:50 am
This is how the problem looks at our failing modules.

Code: Select all

ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x0780000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378179
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:380
ho 0 tail 12 room 4
load:0x07800000,len:3378177
ets Jul 29 2019 12:21:46


... and so on ...
Thanks. It seems that there is no any error have been seen but you are not able to recover that module using flash erase.

So, Into your case that issue is little bit different in which module is not able to recovered even after erasing flash memory while into our case that module is recovered after erasing flash memory
Regards,
Ritesh Prajapati

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Sun Jan 23, 2022 11:34 am

desp32fun wrote:
Sun Jan 23, 2022 11:15 am
Ritesh, could you please scan the QR Code on your faulty WROVER modules?
Simply use any qr code scanner software. You will receive a 18-digit number.

The first 2 digits tell you the fab,
the next 4 digits are the manufacturing date YYWW
and the rest is just a MAC address.

Then when you have it, we might find out if it's a suspicious batch, or if the problem is completely random.
Since we both surely did not use WROVER Modules from the same reel, it might give some clue.
I see failed modules from fab 09, Year 21, week 20. But since we don't use the modules for very long, this may
as well be a coincidence. That's why it would be nice to counter-check that with your modules.
Let me check and will provide details for the same once i will get it tomorrow.
Regards,
Ritesh Prajapati

WiFive
Posts: 3529
Joined: Tue Dec 01, 2015 7:35 am

Re: Hardware Flash Corruption Issue

Postby WiFive » Sun Jan 23, 2022 8:06 pm

There is no source code for ROM bootloader. If you see it happen more than once then it would be worth investigating.

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Mon Jan 24, 2022 12:55 pm

WiFive wrote:
Sun Jan 23, 2022 8:06 pm
There is no source code for ROM bootloader. If you see it happen more than once then it would be worth investigating.
Ok. Thanks for providing confirmation for the same.

I will let you know if we face similar issue into other modules as well
Regards,
Ritesh Prajapati

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Thu Feb 24, 2022 7:15 am

@WiFive,

We have faced same issue again into another module in which facing similar kind of issues like boot loader is not able to start firmware due to checksum mismatch.
[2022-02-24 11:06:43.291] ets Jul 29 2019 12:21:46
[2022-02-24 11:06:43.291]
[2022-02-24 11:06:43.291] rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
[2022-02-24 11:06:43.291] configsip: 0, SPIWP:0xee
[2022-02-24 11:06:43.291] clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[2022-02-24 11:06:43.291] mode:DOUT, clock div:2
[2022-02-24 11:06:43.291] load:0x3fff0018,len:4
[2022-02-24 11:06:43.291] load:0x3fff001c,len:5796
[2022-02-24 11:06:43.291] load:0x40078000,len:7756
[2022-02-24 11:06:43.291] load:0x40080000,len:5876
[2022-02-24 11:06:43.291] csum err:0xaf!=0xab
[2022-02-24 11:06:43.291] ets_main.c 384
[2022-02-24 11:06:43.625] ets Jul 29 2019 12:21:46
[2022-02-24 11:06:43.669]
[2022-02-24 11:06:43.669] rst:0x10 (RTCWDT_RTC_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
[2022-02-24 11:06:43.669] configsip: 0, SPIWP:0xee
[2022-02-24 11:06:43.669] clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[2022-02-24 11:06:43.669] mode:DOUT, clock div:2
[2022-02-24 11:06:43.669] load:0x3fff0018,len:4
[2022-02-24 11:06:43.669] load:0x3fff001c,len:5796
[2022-02-24 11:06:43.669] load:0x40078000,len:7756
[2022-02-24 11:06:43.669] load:0x40080000,len:5876
[2022-02-24 11:06:43.669] csum err:0xaf!=0xab
[2022-02-24 11:06:43.669] ets_main.c 384
[2022-02-24 11:06:44.003] ets Jul 29 2019 12:21:46
[2022-02-24 11:06:44.045]
[2022-02-24 11:06:44.045] rst:0x10 (RTCWDT_RTC_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
[2022-02-24 11:06:44.045] configsip: 0, SPIWP:0xee
[2022-02-24 11:06:44.045] clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[2022-02-24 11:06:44.045] mode:DOUT, clock div:2
[2022-02-24 11:06:44.045] load:0x3fff0018,len:4
[2022-02-24 11:06:44.045] load:0x3fff001c,len:5796
[2022-02-24 11:06:44.045] load:0x40078000,len:7756
[2022-02-24 11:06:44.045] load:0x40080000,len:5876
[2022-02-24 11:06:44.045] csum err:0xaf!=0xab
[2022-02-24 11:06:44.045] ets_main.c 384
[2022-02-24 11:06:44.381] ets Jul 29 2019 12:21:46
[2022-02-24 11:06:44.423]
[2022-02-24 11:06:44.423] rst:0x10 (RTCWDT_RTC_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
[2022-02-24 11:06:44.423] configsip: 0, SPIWP:0xee
[2022-02-24 11:06:44.423] clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[2022-02-24 11:06:44.423] mode:DOUT, clock div:2
[2022-02-24 11:06:44.423] load:0x3fff0018,len:4
[2022-02-24 11:06:44.423] load:0x3fff001c,len:5796
[2022-02-24 11:06:44.423] load:0x40078000,len:7756
[2022-02-24 11:06:44.423] load:0x40080000,len:5876
[2022-02-24 11:06:44.423] csum err:0xaf!=0xab
[2022-02-24 11:06:44.423] ets_main.c 384
[2022-02-24 11:06:44.759] ets Jul 29 2019 12:21:46
[2022-02-24 11:06:44.802]
Scenario is like Device was running fine at customer end without any issue so far since last one month but suddenly facing issue. Also device is connected with Solar Inverter and getting power from Inverter.

So, Will Power Source affect this type of issues or any voltage fluctuation will create this type of issue?

Let me know if anyone has any hint for the same. Because we are facing this type of issues randomly at customer site due to which we are facing product quality issues and some negative feedbacks.
Regards,
Ritesh Prajapati

ESP_Sprite
Posts: 9766
Joined: Thu Nov 26, 2015 4:08 am

Re: Hardware Flash Corruption Issue

Postby ESP_Sprite » Fri Feb 25, 2022 3:06 am

Can you dump the flash, and compare it to what should be there (as in: the flash on a functioning device)?

Ritesh
Posts: 1383
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Hardware Flash Corruption Issue

Postby Ritesh » Fri Feb 25, 2022 4:14 am

ESP_Sprite wrote:
Fri Feb 25, 2022 3:06 am
Can you dump the flash, and compare it to what should be there (as in: the flash on a functioning device)?
Hello,

We have already checked that as well and didn't get any difference while comparing with working and not working board.

Also We have checked EFUSE data, Flash Data, Flash ID and other debugging data into both working and not working devices which seems to be fine without any issue.

Let me know if you have any idea regarding this issue or you have already faced this issue at your end.
Regards,
Ritesh Prajapati

Who is online

Users browsing this forum: No registered users and 30 guests