3 comments

  • fsmunoz36 days ago
    This is my attempted at something that makes using Common Lisp with Datastar easier. To test the SDK I made this demo that shows the simulation of the Cassini-Huygens mission using the NASA SPICE toolkit and JPL Horizons API: <a href="https:&#x2F;&#x2F;dataspice.interlaye.red&#x2F;" rel="nofollow">https:&#x2F;&#x2F;dataspice.interlaye.red&#x2F;</a><p>The Datastar API itself is very simple, 3 functions or so, I ended up wasting a lot more time on stuff like leeping the SSE stream open, compression support (zstd only atm), and trying to use CLOS in a way that would fit both Hunchentoot and Clack (not always easy).
    • mark_l_watson36 days ago
      Very nice, thank you. The tests directory is good for testing, and I suggest adding an examples directory with a few very short and complete simple examples.
      • fsmunoz36 days ago
        Thanks! Yes, I think I will add the Horizons JPL demo there, it&#x27;s essentially a 20 line file.
  • kscarlet36 days ago
    &gt; Each SSE connection blocks one worker for its entire duration.<p>Have you tried wookie? Such extreme case of blocking the event loop... negates any benefit of async processing.
    • fsmunoz34 days ago
      An update: I&#x27;ve spent some time taking a much deeper look, and while I can&#x27;t guarantee it&#x27;s perfect, I added a different approach for Clack+Woo, documented here: <a href="https:&#x2F;&#x2F;github.com&#x2F;fsmunoz&#x2F;datastar-cl&#x2F;blob&#x2F;main&#x2F;SSE-AND-WOO.org" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fsmunoz&#x2F;datastar-cl&#x2F;blob&#x2F;main&#x2F;SSE-AND-WOO...</a><p>In short: I&#x27;ve replace the Common Lisp loop (that works for Hunchentoot since it opens threads, but doesn&#x27;t for Woo since it blocks) with a deeper integration into the event loop:<p>&gt; And that was the main change: looking at the innards of it, there are some features available, like woo.ev:<i>evloop</i>. This was not enough, and access to the libev timer was also needed. After some work with lev and CFFI, the SDK now implements a Node.js-style approach using libev timers via woo.ev:<i>evloop</i> and the lev CFFI bindings (check woo-async.lisp).<p>This is likely (almost surely) not perfect or even ideal, but it does seem to work, and I&#x27;ve been testing the demo app with 1 worker and multiple clients.
    • fsmunoz36 days ago
      I haven&#x27;t tried Wookie, since adding Clack+Woo was already a substantial change. Reading <a href="https:&#x2F;&#x2F;fukamachi.hashnode.dev&#x2F;woo-a-high-performance-common-lisp-web-server" rel="nofollow">https:&#x2F;&#x2F;fukamachi.hashnode.dev&#x2F;woo-a-high-performance-common...</a> , where it compares with Wookie, I&#x27;m not sure if it would make a difference: it might be wrong, but &quot;it says:<p>&gt; Of course, this architecture also has its drawbacks as it works in a single thread, which means only one process can be executed at a time. When a response is being sent to one client, it is not possible to read another client&#x27;s request.<p>... which for SSE seems to be similar to what the issue is with Woo. I wrote a bit more on it in <a href="https:&#x2F;&#x2F;github.com&#x2F;fsmunoz&#x2F;datastar-cl&#x2F;blob&#x2F;main&#x2F;SSE-WOO-LIMITATIONS.org" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fsmunoz&#x2F;datastar-cl&#x2F;blob&#x2F;main&#x2F;SSE-WOO-LIM...</a> , and it can be more of a &quot;me&quot; problem than anything else, but to keep a SSE stream open, it doesn&#x27;t play well with async models. That&#x27;s why I added a with-sse-response macro that, unlike with-sse-connection, sends events without keeping the connection open.
      • kscarlet36 days ago
        wookie is built on cl-async, so my hope is that it&#x27;s more tractable to write proper async SSE handler. But I haven&#x27;t looked at whether it&#x27;s possible to keep open connection asyncly.
  • rybosome36 days ago
    Thanks for sharing. I’m curious why the example SPICE application uses Fortran to parse the SPICE data?
    • fsmunoz36 days ago
      The CL-SPICE library I used, that wraps the SPICE C library through CFFI, doesn&#x27;t cover the type of SPICE kernel that I wanted to use for the Comms module. I could try and add it, but it could be more involved than what I expected and put the thing on hold.<p>So I used the FORTRAN SDK for SPICE, since I had used it before, and it&#x27;s reasonably small and easy. The alternative coud be using the C SDK, but I went with FORTRAN since I already had most of the code from a previous project.