I had to scroll a <i>very</i> long way to get to the most important bit:<p>> Running rr record/replay without access to CPU HW performance counters is accomplished using lightweight dynamic (and static) instrumentation. The Software Counters mode rr wiki has more details in case you're curious about some more of the internals.<p>You should move that to the top.
I have tried SO hard to get rr to work for me, including buying a separate pc just to use it...but it just consistently fails so I've basically abandoned it. Something like this would absolutely be a godsend. Just getting something consistently working with Ubuntu is amazing. Does this approach make working in something like WSL viable?<p>I would love if this were upstreamed. Is there a github issue where you discuss the possibility of this with the rr devs? That might be something to add to your readme for everyone else who wants to follow along. Thanks!
Very cool. It’s difficult to praise rr too much, and it just keeps getting better. If you’re not using it, you’re missing out on a superpower.
Well! That eliminates one of if not the biggest problems with rr:) Is there some catch or tradeoff? Performance, maybe?
Author here. Yes there are some tradeoffs:<p>- Performance: slower than using rr with HW counters. Both dynamic and static instrumentation employed by _Software Counter mode_ rr slow things down<p>- Potential fragility: Dynamic and static instrumentation can often make record/replay a bit more fragile<p>- Currently only x86-64 support has been publicly released. I have aarch64 support working reasonably well internally and it allows me for instance run to rr in a Linux VM running on macOS ! I have yet to figure out my strategy for the aarch64 release so watch <a href="https://github.com/sidkshatriya/rr.soft" rel="nofollow">https://github.com/sidkshatriya/rr.soft</a> for any updates.<p>- Currently can run only on a few recent Linux distributions (e.g. Fedora 40/41, Debian Unstable, Ubuntu 24.10) because it relies on robust debuginfod support that is not widespread yet. See <a href="https://github.com/sidkshatriya/rr.soft/wiki#how-does-software-counters-mode-rr-work--why-does-it-require-debuginfod-" rel="nofollow">https://github.com/sidkshatriya/rr.soft/wiki#how-does-softwa...</a> for why debuginfod is required. The debuginfod requirement may be relaxed in the future with more work<p>Regardless of the tradeoffs this allows rr to be used in many more situations i.e. wherever HW Performance counter access is not possible/not reliable/broken.<p>I would love it if more people tried this out and let me know if things worked out well for them (or not) with their programs.
I'm assuming this changes nothing about the lack of io_uring support?
Yes, io_uring is still not supported due to fundamental issues in the overall rr architecture which my modification does not resolve. My modification only addresses the HW counter requirement of upstream rr and the other core aspects of rr remain the same.<p>Normal system calls transition to kernel space and return back from kernel space. They will change your program's memory/process state as soon as they complete. This gives rr an easy boundary when it "can do its thing" to record memory/process state changes or insert results (during replay).<p>When does an io_uring request/response complete ? That's difficult to say. The kernel/userspace when using io_uring communicate with each other by checking a queue head or tail with memory accesses to see if something got added/removed from request/response ring buffer.<p>Think of io_uring and userspace cooperating via memory. (Yes, sometimes "proper" traditional ring crossing system calls are made but what makes io_uring so fast is communicating via memory and not via system calls most of the time). Anyways all this makes it difficult for rr to intervene on the boundary between kernel and userspace because this boundary is elusive when it comes to io_uring. The memory writes cannot be caught by ptrace ! This explanation is simplified of course.<p>There are some plans to deal with io_uring by rr maintainers <a href="https://github.com/rr-debugger/rr/issues/2613" rel="nofollow">https://github.com/rr-debugger/rr/issues/2613</a>
Will if run on latest NixOS?
Probably not at the moment.<p>_Software Counter mode_ rr requires robust support for debuginfod in your Linux distribution.<p>NixOS to my knowledge does not provide debuginfo via a dedicated debuginfod server for <i>all packages</i> so this will make things unreliable for debugging via _Software Counters mode_ rr for packages for which debuginfo is not available via debuginfod.<p>I recently learned about <a href="https://github.com/symphorien/nixseparatedebuginfod" rel="nofollow">https://github.com/symphorien/nixseparatedebuginfod</a> for nix (note: I haven't used this so I don't know how reliable or good this is). Anyways, this project requires setting `separateDebuginfo = true;` for the derivation for the debuginfo to be available via debuginfod. This is an opt in approach but we need pervasive support like it exists in Fedora (and some others).
Does it work with Pernosco ?
Very nice.<p>Has anyone got rr working with python?
rr has always worked with Python in the sense that it can record and replay Python programs.<p>However, when you try to debug the program you can only debug the C code the Python interpreter is written in.<p>I suppose you want to be able to debug the Python code itself. Here is something that could do this <a href="https://pypy.org/posts/2016/07/reverse-debugging-for-python-8854823774141612670.html" rel="nofollow">https://pypy.org/posts/2016/07/reverse-debugging-for-python-...</a> . I don't think the project is active nowadays though. Also I haven't used it so can't say whether it is good or not.<p>It should be possible to built a Python reverse debugger on top of rr. I know this should be possible because I built something for PHP <a href="https://github.com/sidkshatriya/dontbug" rel="nofollow">https://github.com/sidkshatriya/dontbug</a> .<p>There are other fancy (and possibly better) things that are possible -- instead of building a Python debugger atop rr you can record the full trace of the Python program and then for e.g. store the values of important variables at each executed line of the Python program in a database. This would again use rr as the record/replay substrate but with a slightly different approach. This is an area which I've done some work internally but nothing public released yet :-) !
<a href="https://retracesoftware.com/" rel="nofollow">https://retracesoftware.com/</a> Is python specific and seems promising.
Patiently waiting for the day when someone makes something similar for macOS.
It's very difficult for a broad based record/replay software like rr to exist for macOS in my opinion. macOS system interfaces are quite basic in terms of functionality compared to Linux and increasingly locked down.<p>rr uses many advanced features of Linux `ptrace`. Compare `man ptrace` on Linux with that on macOS for example and you will notice that Linux gives a lot of features to `ptrace` that macOS simply does not.<p>There are a large number of other features required for practical record and replay -- I dont think macOS simply provides them also.<p>It's probably possible to build _some_ record/replay system on macOS with constraints, restrictions, workarounds and compromises -- never say never as they say. But I don't think it can be as capable/generic as rr on Linux.
Could this help? <a href="https://developer.apple.com/documentation/xcode-release-notes/xcode-16_3-release-notes" rel="nofollow">https://developer.apple.com/documentation/xcode-release-note...</a><p><pre><code> Instruments 16.3 includes a new Processor Trace Instrument which uses hardware-supported, low-overhead CPU execution tracing to accurately reconstruct execution of the program. This tool provides metrics like duration, number of cycles, and instructions retired for every function executed on the CPU. Timeline in Instruments presents execution flame graph, while detail views provide aggregate-level data like Call Tree or aggregated metrics (min, max, count, sum), divided by function. Traces can be recorded using the new Processor Trace template on supported devices: M4 Mac, M4 iPad, and iPhone 16/16 Pro. Tracing on the device requires additional configuration in the System Settings.</code></pre>
Very cool. Does the dynamic instrumentation handle JIT emitted code?
Still waiting for rr to work more transparently/easily with Haskell.
Is this getting merged back into rr?