8 comments

  • saghm4 hours ago
    &quot;Be careful with mutexes&quot; is good advice, but I&#x27;m surprised it doesn&#x27;t explicitly call out the various channels that tokio provides as alternatives (detailed here: <a href="https:&#x2F;&#x2F;docs.rs&#x2F;tokio&#x2F;latest&#x2F;tokio&#x2F;sync&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;docs.rs&#x2F;tokio&#x2F;latest&#x2F;tokio&#x2F;sync&#x2F;index.html</a>). There are a variety of options that fit different use cases, and you don&#x27;t even need to enable the runtime feature to use them (e.g. if you want to do a single check for completion rather than await). I&#x27;d estimate that at least half of the bottlenecks I&#x27;ve seen with mutexes when using tokio could have been avoided by not even using a mutex at all and instead passing the data that&#x27;s truly needed across different tasks with some type of channel.<p>The other trick I&#x27;ve used a few times that&#x27;s a bit hacky but can get the job done is when reading a snapshot of the data under a mutex is enough without needing to prevent other changes; if that&#x27;s the case, you can just clone the data and drop the mutex to allow other uses move forward at the cost of the data potentially being stale.
    • CoolestBeans1 hour ago
      Tasks and channels is the way. You can get something that feels like programming a real preemptive concurrency model like BEAM languages or golang but with minimal overhead.
    • rusbus2 hours ago
      (I am OP) Both good call outs. Will update the article to include them
      • saghm1 hour ago
        Awesome! I was pretty confident you already were aware of both of those based on the level of knowledge needed for everything else in there, so I mostly was mentioning them here in case some people here might find them useful. Adding them in for others is even better though!
  • 5ersi4 hours ago
    For a true high performance you should use thread busy-spinning, CPU pinning and SPSC&#x2F;MPSC ring buffers.
    • VorpalWay4 hours ago
      It all depends on what you are doing. I do embedded with strict realtime requirements. CPU pinning would not be an option. I have also done software that should use as little resources as possible (but still be quick) to coexist with other software on the same hardware.<p>All of these are different, valid, meanings of high performance. You need context. An interactive IDE is yet another thing that needs to be high performance in yet another way.
      • mahboi1 hour ago
        Also, using 100% CPU without a good reason can cause thermal throttling that makes it slower for the sections that actually need 100% CPU
    • Kenji4 hours ago
      [dead]
  • dist1ll6 hours ago
    When you&#x27;re at a point of tuning Tokio, consider taking a look at ef_vi&#x2F;DPDK + SPDK
    • kev0093 hours ago
      I don&#x27;t think there is a ton of overlap. tokio is appropriate for general userspace apps, ranging anywhere from a CLI, GUI, API or web app. DPDK and SPDK are specialized fast paths for building network data paths and storage solutions that come with tradeoffs: DPDK uses poll mode drivers, outside of the operating system, which have various implications including busy waiting and taking over the interface. That is why DPDK is fast, no kernel&#x2F;userspace context switching and copies, and the drivers are tuned for the polling model. But it&#x27;s not a general purpose building block.
      • dist1ll2 hours ago
        Fwiw with ef_vi you have full control over the event queue - you don&#x27;t need to busy-spin it, you can choose whatever strategy you prefer.<p>&gt; tokio is appropriate for general userspace apps<p>Yep, and for those I wouldn&#x27;t recommend it. But tokio is also widely used in performance-critical infrastructure and web services. For those I&#x27;d say it can definitely be worth taking a second look at kernel bypass.
    • rusbus4 hours ago
      Do you have any resources worth referencing on this? I assume this isn&#x27;t something that works with tokio more of a replace tokio?
  • Tsarp9 hours ago
    One great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.
    • jeffbee8 hours ago
      Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
      • Veserv7 hours ago
        That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload.<p>You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range.<p>The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.
        • jeffbee6 hours ago
          I&#x27;m just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I&#x27;ve seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe&#x2F;uretprobe is 100x more expensive than XRay, at a minimum. HotSpot&#x27;s JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio.<p>1: <a href="https:&#x2F;&#x2F;llvm.org&#x2F;docs&#x2F;XRay.html" rel="nofollow">https:&#x2F;&#x2F;llvm.org&#x2F;docs&#x2F;XRay.html</a> ... is there even a Rust analog to this?
          • RealityVoid5 hours ago
            Huh, it seems xray puts blank trampolines all over your binary? That sounds pretty nifty but I would expect it to be pretty language agnostic, ish? Adding support should be doable for Rust as well, right? Anyways, pretty nifty.<p>I am by no means an expert, but I&#x27;ve recently improved performance for some code and used tracy. They have rust bindings as well. It&#x27;s pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations&#x2F;aquisition tool.<p>Edit: apparently rust already supports xray natively on the nightly.
          • duped6 hours ago
            Not even an analog: <a href="https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;beta&#x2F;unstable-book&#x2F;compiler-flags&#x2F;instrument-xray.html" rel="nofollow">https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;beta&#x2F;unstable-book&#x2F;compiler-flags&#x2F;...</a><p>It&#x27;s worth pointing out though that just tracing function calls isn&#x27;t good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.<p>afaik, C&#x2F;C++ have the same problem.
            • jeffbee4 hours ago
              The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.
              • duped3 hours ago
                The &quot;C++ community&quot;, if it even exists, barely believes in sharing code let alone any library being &quot;dominant.&quot; They&#x27;d have to agree on a build system first, after all.<p>But honestly that&#x27;s a mischaracterization of the situation in Rust. Tokio is popular for networked service backends. If that&#x27;s the wheelhouse you&#x27;re in then yea it might look &quot;dominant.&quot;
                • ablob2 hours ago
                  You don&#x27;t need a build system to share code.<p>You can share with header files and respective (shared) object files regardless of the build system you&#x27;re using. Likewise you could just share the source. None of this needs a build system.
                  • duped1 hour ago
                    I was just being a bit sardonic because the C++ ecosystem is so fragmented that something like tokio couldn&#x27;t really exist. It would be one of three executors in boost, abseil, or folly, and you would never see the kind of downstream ecosystem build on top of them because C++ shops are allergic to external dependencies.
      • nicoburns8 hours ago
        One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.
        • jeffbee8 hours ago
          I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don&#x27;t understand its behavior. Truly the future we were promised.
          • brunoarueira1 hour ago
            I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this <a href="https:&#x2F;&#x2F;github.com&#x2F;brunoarueira&#x2F;thoth-mesh&#x2F;tree&#x2F;main&#x2F;docs&#x2F;adr" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;brunoarueira&#x2F;thoth-mesh&#x2F;tree&#x2F;main&#x2F;docs&#x2F;ad...</a>. This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!
      • rusbus4 hours ago
        Was this in a specific application? I wouldn&#x27;t necessarily expect that histogram to be particularly bad for most applications.
        • jeffbee4 hours ago
          Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.
      • foota7 hours ago
        Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?
        • jeffbee4 hours ago
          If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.
          • loeg1 hour ago
            In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.
      • MomsAVoxell8 hours ago
        If you’re not using eBPF to trace your app you’re doing it wrong.
        • MobiusHorizons2 hours ago
          Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.
        • jeffbee8 hours ago
          The low cost of eBPF tracing is another myth.
          • MomsAVoxell7 hours ago
            1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it&#x27;s a far better path to take than in-app telemetry.
  • denizay56 minutes ago
    Fast Tokioo, drift, drift, drift!
  • jeffbee8 hours ago
    All of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little-known and too easy to violate.
    • cube008 hours ago
      I can&#x27;t say I&#x27;m surprised when I see the 100+ function stack traces that Axum built on Tokio produces.<p>Before you say Axum is &quot;holding it wrong&quot; the project lives under the tokio-rs GitHub org.
      • rusbus7 hours ago
        Note that most of those end up getting inlined in practice
    • prydt2 hours ago
      Do you have any references for these principles for writing Tokio servers? Or just a high level summary of what best practices look like?
  • iberator3 hours ago
    What the hell is Tokio? Articles mentions it like once I was expecting some programing principles from Japan
    • carllerche3 hours ago
      <a href="https:&#x2F;&#x2F;github.com&#x2F;tokio-rs&#x2F;tokio" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tokio-rs&#x2F;tokio</a>
      • cogman102 hours ago
        To further explain. Rust doesn&#x27;t provide a runtime&#x2F;framework for async&#x2F;await, you have to bring your own. Tokio is (I believe) the most popular async&#x2F;await framework for rust.
  • kevinbaiv2 hours ago
    [flagged]