Yep. Even outside of tracing, there are several different ways that the Linux kernel patches itself:<p>- Static calls: like a call to a global function pointer, except instead of loading a function pointer and doing an indirect call, the code is patched to do a direct call to the destination<p>- Static keys: like an if statement testing a global boolean, except instead of loading a boolean and doing a conditional branch, the code is patched to do either an unconditional branch or a nop<p>- Runtime constants: like a load of a global variable, except instead of loading, the value is patched directly into the code<p>- Alternatives: selects one of multiple possible instruction sequences depending on (usually) whether the CPU supports specific instructions<p>It's really fascinating to see the kind of fun efficient stuff you can do when you have that level of low-level control. Not just code patching but things like RCU as well.<p>> However it is still interesting to write machine code snippets once and execute them many times, like the nested function trampolines.<p>I slightly disagree on this though. In my experience writing code with Clang blocks (which don't use trampolines), they're often useful for code organization even if the callback will only be called once. Therefore, even ignoring security issues, I think GCC choosing a design that required cache flushing was a mistake - certainly in retrospect (as cache flushing has become more expensive over the years), but perhaps even at the time. I did some research, and trampolines were introduced in GCC 2.0, which already included mprotect calls and/or cache flushes on some of the architectures it supported, such as MIPS. However, this was a relatively new development, and on most of the supported architectures it didn't do either of those things. But on MIPS it would do an mprotect every single time a trampoline was created, which can't have been fast.
It's not the cache flush that's expensive, it's loading the new instructions that aren't cached. If you are flushing the very next instruction and then immediately executing it, that's expensive because of the serial dependency, but if you're generating new code, flushing it shouldn't be more expensive than if you were simply accessing new code for the first time. But on old systems you could modify the very next instruction with no penalty because there wasn't a cache. You can still do that and probably faster than those old systems could (they were slower because of not having a cache, everything was an uncached access), it's just a waste of most of the <i>new</i> system's performance.<p>BTW you can do all of this cool stuff in user mode on Linux too (but not on OpenBSD) - you just have to opt in to executable stack and/or writable .text. I could have written the dynamic shift instruction generator I mentioned, but I didn't want to spend the effort, but I imagined having a language with actual support for something like that (like static keys for variables).