LCD panels with an RGB interface (if it has HSYNC and VSYNC lines) do not have a frame buffer, and the S3 has to maintain full-screen psram buffers and constantly refresh the entire display. I am no expert in the details so I can't offer any tuning advice. There may be some memory-access contention considerations.
I use lvgl on smaller panels (320x480-ish and down), though for a low-frame-rate gui. The smaller panels have frame buffers so you only update small-ish regions as needed. They usually have an spi interface which does not help frame rate, but there are some out there with parallel 16-bit interfaces for snappy updates.
You might ask on the lvgl forum -- a bunch of good folks plus devs:
https://forum.lvgl.io/