Hardware Flash Corruption Issue

[ESP_Sprite]

Hm, I'm scratching my head here then, not much that I know of that corrupts the boot process but leaves a perfectly valid flash in place.. Any chance there's extraordinary circumstances (high EMI, very high or low temperatures, power fluctuations) that could have affected the device?

[Ritesh]

Hm, I'm scratching my head here then, not much that I know of that corrupts the boot process but leaves a perfectly valid flash in place.. Any chance there's extraordinary circumstances (high EMI, very high or low temperatures, power fluctuations) that could have affected the device?

Thanks for your prompt response.

There may be chances of Power Fluctuation or Temperature impact as it is used under roof where Solar System is installed and also device is getting powered from Solar Inverter.

We have already started process to measure with Power Regulator and started to validate that circuit as it is creating random issues.

Also we have faced similar kind of issue into one another device in which device is getting boot from Boot loader and trying to boot from 0x10000 default location in which Checsum Error is getting then trying to boot from secondary OTA partition.

Below are logs for that device for reference

[2022-02-22 12:02:22.998] ets Jul 29 2019 12:21:46
[2022-02-22 12:02:22.998]
[2022-02-22 12:02:22.998] rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
[2022-02-22 12:02:22.998] configsip: 0, SPIWP:0xee
[2022-02-22 12:02:22.998] clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[2022-02-22 12:02:22.998] mode:DOUT, clock div:2
[2022-02-22 12:02:22.998] load:0x3fff0018,len:4
[2022-02-22 12:02:22.998] load:0x3fff001c,len:5800
[2022-02-22 12:02:22.998] load:0x40078000,len:7756
[2022-02-22 12:02:22.998] load:0x40080000,len:5876
[2022-02-22 12:02:22.998] entry 0x40080318
[2022-02-22 12:02:22.998] I (29) boot: ESP-IDF v3.1-beta1-216-gd91c425178 2nd stage bootloader
[2022-02-22 12:02:22.998] I (29) boot: compile time 11:10:31
[2022-02-22 12:02:22.998] I (30) boot: Enabling RNG early entropy source...
[2022-02-22 12:02:22.998] I (36) boot: SPI Speed : 40MHz
[2022-02-22 12:02:22.998] I (40) boot: SPI Mode : DOUT
[2022-02-22 12:02:22.998] I (44) boot: SPI Flash Size : 16MB
[2022-02-22 12:02:22.998] I (48) boot: Partition Table:
[2022-02-22 12:02:22.998] I (52) boot: ## Label Usage Type ST Offset Length
[2022-02-22 12:02:22.998] I (59) boot: 0 nvs WiFi data 01 02 00009000 00004000
[2022-02-22 12:02:22.998] I (66) boot: 1 otadata OTA data 01 00 0000d000 00002000
[2022-02-22 12:02:22.998] I (74) boot: 2 phy_init RF data 01 01 0000f000 00001000
[2022-02-22 12:02:22.999] I (81) boot: 3 factory factory app 00 00 00010000 00200000
[2022-02-22 12:02:22.999] I (89) boot: 4 ota_0 OTA app 00 10 00210000 00200000
[2022-02-22 12:02:22.999] I (96) boot: 5 storage Unknown data 01 82 0081c000 00200000
[2022-02-22 12:02:22.999] I (104) boot: End of partition table
[2022-02-22 12:02:22.999] I (108) boot: Defaulting to factory image
[2022-02-22 12:02:22.999] I (113) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x2d2ec (185068) map
[2022-02-22 12:02:23.071] I (189) esp_image: segment 1: paddr=0x0003d314 vaddr=0x3ffb0000 size=0x02cfc ( 11516) load
[2022-02-22 12:02:23.071] I (193) esp_image: segment 2: paddr=0x00040018 vaddr=0x400d0018 size=0x9fcd0 (654544) map
[2022-02-22 12:02:23.330] I (432) esp_image: segment 3: paddr=0x000dfcf0 vaddr=0x3ffb2cfc size=0x00ed8 ( 3800) load
[2022-02-22 12:02:23.330] I (434) esp_image: segment 4: paddr=0x000e0bd0 vaddr=0x3ffb3bd4 size=0x00000 ( 0) load
[2022-02-22 12:02:23.330] I (439) esp_image: segment 5: paddr=0x000e0bd8 vaddr=0x40080000 size=0x00400 ( 1024) load
[2022-02-22 12:02:23.330] I (449) esp_image: segment 6: paddr=0x000e0fe0 vaddr=0x40080400 size=0x12340 ( 74560) load
[2022-02-22 12:02:23.390] I (488) esp_image: segment 7: paddr=0x000f3328 vaddr=0x400c0000 size=0x00000 ( 0) load
[2022-02-22 12:02:23.390] I (488) esp_image: segment 8: paddr=0x000f3330 vaddr=0x50000000 size=0x00000 ( 0) load
[2022-02-22 12:02:23.390] E (495) esp_image: Checksum failed. Calculated 0x29 read 0x2d
[2022-02-22 12:02:23.390] E (501) boot: Factory app partition is not bootable

That device is also placed for Solar System and powered by Inverter.

Let me know if need anything else from my end.

[Ritesh]

Hello ESP_Sprite and WiFive,

Let me know if need anything else from my end as per details provided into last comment in which we have faced into one more device.

So, We are not sure like what can be the issue for the same but need your quick help to resolve that issue because we are getting back products from customer due to that critical issue or we need to get back their product and need to give new one in replacement of those faulty devices which costs us more than actual product costs.

So, We are now loosing our products and market values due to that critical help.

It would be great if you can assist to resolve that issue ASAP.

[ESP_Sprite]

Given the fact that you have a commercial product, have you tried contacting our FAEs yet? Mail sales @ espressif.com if you don't have any contact, they can probably do more for you in this particular scenario than we can.

[Ritesh]

Given the fact that you have a commercial product, have you tried contacting our FAEs yet? Mail sales @ espressif.com if you don't have any contact, they can probably do more for you in this particular scenario than we can.

Hello Team,

Yes. We have already raised query to Espressif Team regarding problem which we have faced earlier and they suggested to send that faulty board at their end.

So, Let me check from my end to send one or two faulty devices in which we have seen problems into field.

Just i am curious that device is recovered after erasing flash memory manually which is strange thing. Do you have any idea regarding the same like how it can be recovered after erasing flash memory manually using tool or through Firmware Update (OTA) in case if secondary partition has been started with stored firmware.

[helpi90]

Hello Ritesh,

were you able to solve the problem? We just have the same problem with a product after we switched to the 16 MB chip. The 8 MB version was in use for several years and never had any problems. The chip is from 2021 week 51.

Unfortunately, deleting the flash does nothing for us either. Thanks for your help.

[Ritesh]

Hello Ritesh,

were you able to solve the problem? We just have the same problem with a product after we switched to the 16 MB chip. The 8 MB version was in use for several years and never had any problems. The chip is from 2021 week 51.

Unfortunately, deleting the flash does nothing for us either. Thanks for your help.

Hello,

Still we didn't get any response from Espressif Team due to lockdown at Shanghai. Our Product is reached to their office but they are not able to check it yet as there is major lockdown into Shanghai City where we have sent our product.

But it seems like there is issue into flash memory which has been used as per your case.

We have faced another problem regarding flash memory completely stopped to work while operating into battery in which we found that Flash Memory which has been used into particular slot was not same as we were using 16 MB Module earlier.

So, We have did some research around one month and found exact root cause of that problem. But still remaining to find rootcause for current issue which I have raised for the same.

[WiFive]

So, We have did some research around one month and found exact root cause of that problem. But still remaining to find rootcause for current issue which I have raised for the same.

Can you share the root cause you found and solution?

[Ritesh]

So, We have did some research around one month and found exact root cause of that problem. But still remaining to find rootcause for current issue which I have raised for the same.

Can you share the root cause you found and solution?

Hello Wi-Fi,

Sorry for late response.

We have checked and found that into 600 boards, ESP32-WROVER-IE modules have been used in which Flash is working only till 3.3. V while we were operating product over battery which will go beyond 3.3V due to flash was corrupted.

Then we have checked FCCID and found that it is ESP32-WROVER-IE instead of ESP32-WROVER due to which ESP32 Module Flash Memory has been corrupted and faced issue into those 600 Modules.

But, There is still open issue for which I have raised issue of Hardware Flash Corruption issue for which we didn't found exact root cause and solution for the same.

[phatpaul]

Just curious if your issues are related to a known issue with the flash chips manufactured by XMC?

If the output of:

Code: Select all

esptool.py flash_id

gives Manufacturer: 20,
that means you have a chip manufactured by "XMC".
And if the output of:

Code: Select all

esptool.py read_flash_status --bytes 3

gives Status value: 0xe37bfc,
that is indicative of corrupted status register.

If that's the case, head on over to:
https://github.com/espressif/esp-idf/issues/7994