This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.<p>PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.<p>The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.
>esp32-s3<p>This microcontroller is a beast, currently using it to do dev work on a pi4.<p>Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
Super cool, thank you for sharing!
This is a really cool project. Thanks for sharing!
9.7 tokens/sec actually seems like a lot! That’s fun!