I ended up writing SmartMatrix::GFX, a glue driver to allow all my previous code to work on this new backend, which sounds easy, but was a fair amount of work.
Here is the end result:
http://marc.merlins.org/perso/arduino/p ... x_GFX.html
Library: https://github.com/marcmerlin/SmartMatrix_GFX


