Experiement to load/execute compiled C code into RAM

[jcsbanks]

I could malloc for then load a tiny binary into RAM and define a function pointer and execute the function.

The C code uses no standard libraries or esp-idf calls, no heap, does not rely on .bss to be initialized, it doesn't even have any includes, it is just computational taking an unsigned int and returning an unsigned int, but since I have the code in C I don't want to convert it to Lua or Javascript if I can do this and I don't want to OTA update the whole binary just to add a 300 byte function. I am used to using C to embed new code in existing binaries on other microcontrollers like this, but without IDA or a simulator for Xtensa I'm a little limited.

Code: Select all

xtensa-esp32-elf-gcc -std=c99 -Os -nostdlib -c test.c

This produces test.o

Code: Select all

xtensa-esp32-elf-objcopy -O binary test.o test.bin

This produces test.bin

Further considerations:
1. Entry point to the single compiled function relative to the start of the bin. I would like it at the start, but the .rodata presently is
2. .rodata would require 32bit alignment if in IRAM unless there is a way to access the same memory as DRAM with an offset
3. Cannot load the bin into IDA to disassemble it to double check what it is doing. There is a ref to memcpy that is not in my code, want to make sure it does not end up in the final bin.

Comments appreciated. Perhaps I should just embed Lua, but I couldn't find any working examples to do that inside an ESP-IDF app and didn't really want to learn Lua. Duktape seems interesting but bulky, and mJS/Mongoose licensing is not ideal for a commercial project where we presently have no money

[jcsbanks]

When I remove things that would use .bss or .rodata, the memcpy is removed and the build can actually be linked with an empty linker script and emit a map file. Would be nice to be able to use .rodata though.

[WiFive]

viewtopic.php?t=4769

[jcsbanks]

heap_caps_malloc(size, MALLOC_CAP_EXEC | MALLOC_CAP_8BIT) fails. Reading the docs suggests that D/IRAM can do data and instructions, but it doesn't say I'm restricted to 32 bit alignment, but can find no explicit way to malloc D/IRAM or may have misunderstood the capabilities. I copied it from DRAM to IRAM but so far whilst it does not crash running the code it does not give expected results just returning a constant so need to investigate more.

[ESP_Angus]

heap_caps_malloc(size, MALLOC_CAP_EXEC | MALLOC_CAP_8BIT) fails. Reading the docs suggests that D/IRAM can do data and instructions, but it doesn't say I'm restricted to 32 bit alignment,

The D/IRAM is some physical RAM which is available on both the executable and data buses, so there is one address on each bus which can map to the same physical RAM. The idea behind the "capabilities based" allocator is to abstract this way, so that you either ask for executable RAM (which has an address on the IRAM bus and requires 4 byte aligned access) or data RAM (which has an address on the data bus), depending on what you need.

Suggest doing heap_caps_malloc(size, MALLOC_CAP_EXEC) and then memcpy() a 4 byte aligned size into this buffer (round up the length of the .bin file if it's not already 4 byte aligned).

I'm not 100% sure that what you're doing will work: taking unlinked object data like this and treating it as a raw executable binary. I guess it probably will work provided you don't use .rodata or any other symbols. You can use "xtensa-esp32-elf-objdump -d test.o" to check that the object file only uses a single .text section and doesn't contain any literal sections (literal sections will have placeholder data waiting to be filled in at link time, and as you don't run the linker there's no way to support these.)

[jcsbanks]

Thanks. I was just using a for loop instead of memcpy, so I had explicit control over alignment, but it seems to work if I use small constants so there is no literal section: the function runs and returns the expected value (with values being passed in and out of the function). It looks like the literal section is used for constants larger than 12 bit since a 24 bit instruction cannot hold a 32 bit value. I'm not used to literal sections so working on understanding them.

Whilst I can do without .rodata and .bss with what feels like less hassle than using a scripting language (and I guess I could add them), I do need what I think the literal section offers to have to handle it.

[jcsbanks]

It works by loading .literal and .text as a blob and then a function pointer which has an offset into the blob where .text is.

Linker script:

Code: Select all

SECTIONS
{
	.literal : {*(.literal)}
	.text : {*(.text)}
}

This is obviously a minimal method with no error checking or any concerns whatsoever!

[rosmianto]

No update on this?

[jcsbanks]

It worked, but I did not continue using it. Doing an OTA update of all the code and data together was easier to use and support for our application. Run time scripts were small and fast by parsing them with std::string so did not have to bulk it with and learn Lua etc.

[Tolomaj]

Hi. I am interested in loading and executing the code from memory.
I've been experimenting a bit with the things that can be done with code loaded into esp32 memory.
But I found some behavior that I don't understand.

<Problem description>
All variables that are not allocated locally seem to not work.
And interacting with them stops the program.
The same goes for the char array, but I haven't experimented much there. However address on the character field was 100% incorrect.

<My theory>
My theory is that offset of file just broke variables addresses because location of variable is shifted.
Or memory is protected for some reason.

<Question>
What causes this behavior?
And how can i achieve that code i load would work as normal c++ ?


This is content of file that is compiled and as .bin loaded to memory in runtime:

Code: Select all

int my_function(); 
int poc();

int main(void (*foo_ptr)(int)) {   // <-- this function is called from main
						//parameter is function that simply print number(for debugging)
    //char arr[] = "abc"; //this doesn't work. "abc" array is not on address (from previous experiments)
    foo_ptr(1);
    my_function(12);
    foo_ptr(2);	// <- prints sucesfuly
    int p = poc(); //<-- esp32 restarts with Error: Core  1 panic'ed (LoadProhibited)
    foo_ptr(3);
    return 5;
}

int c = 1234;   // < - most likely a protected area of ​​memory (theory) or an address shift (theory)

int my_function(int p) {
    int i = 20  + p;
    return i;
}

int poc() {
    return c;
}

This is code i use to allocate memory for code:

Code: Select all

uint8_t* programTMP = new uint8_t[size];
void* program = heap_caps_malloc(size, MALLOC_CAP_EXEC || MALLOC_CAP_8BIT ); // memory alocation for code
programFile.read(programTMP, programFile.size());	// reading prevously loaded file
for (size_t i = 0; i < size; i = i + 4){  	// copy data from bytes to uint32_t.
	uint32_t inst = (programTMP[i+3] << 24) + (programTMP[i+2] << 16) + (programTMP[i+1] << 8) + programTMP[i+0];
	((uint32_t*)program)[i/4] = inst;
}
int (*foo_ptr)(void (*)(int)) = (int (*)(void (*)(int)))(functionAdress); 
Serial.println(foo_ptr(&dummy));	// call function from file. (dummy is function that print integer)