For as long as I’ve been involved with nREPL, the question I’ve heard most
often from people building servers and clients has been some variation of
“Will this work with CIDER?”. And for just as long the honest answer was “Try
it and see”. Today I have a better answer for you -
proof, a compatibility suite for nREPL
servers (and clients). I’ve been working on it since the end of September and
proof 0.2 came out a couple of days ago, so I guess it’s high time
I told you a bit about it.
Try It and See
The nREPL protocol is tiny. A handful of ops, requests and responses that are
simple maps, bencode on the wire. You can write a basic nREPL server in a
couple of hours, and lots of people have done exactly that. Making it play nice with
CIDER, Calva, Conjure and the rest of the clients out there is a very
different matter, though. What clients actually expect from a server is
spread across the reference implementation, a draft spec, many years of
client code and lots of GitHub issues. Which status values mean what? Is
ops in describe a map or a list? Where does the end of an interrupted
evaluation go? What happens to a session when its client goes away in the
middle of an eval? Server authors usually find out one bug report at a time.
Client authors don’t have it any easier, as every server has its own quirks,
and for many years the only way to find them was to try your client with every
server you could get your hands on. I’ve been on both sides of this
conversation for well over a decade (CIDER is a client, nREPL is a server), so
believe me - I know the pain.
Bigger Than Clojure
Why bother? Because I’ve always seen nREPL as something bigger than Clojure.
Back in 2019 I wrote that nREPL is similar in nature to
LSP, only much simpler, and I
still believe that. For any language that can evaluate code at runtime, nREPL
is a much simpler alternative to LSP. Implement eval, interrupt and
load-file, throw in completions and lookups if you feel like it, and every
nREPL client out there becomes an editor integration for your language.
That’s not just a dream anymore. These days there are nREPL servers for
Clojure dialects running on Python (Basilisp), C++ (jank), .NET (ClojureCLR),
Go (let-go) and Node.js (nbb), for Janet and Steel Scheme, and even for
Erlang and Elixir (more on those later). Still, “language-agnostic” doesn’t
mean much if everyone who builds a server has to check it against CIDER by
hand.
Better Late Than Never
The idea of a compatibility suite is not new. When nREPL 1.0 came out in 2022
I wrote that improving the protocol spec and the ability to test against
it was where our focus
should go next, and a year later the lack of a compatibility test suite made
my list of things people hate about
nREPL. I even asked if one of you was
looking for a fun small OSS project. Nobody volunteered, and the idea spent
a few more years on the backburner. This year, thanks to the support of
Clojurists Together, I finally made a
very focused push on it.
It started with the spec. Late last year Phil Hagelberg wrote a draft of the
nREPL protocol spec, which landed in May. A spec is
great, but you can’t run a spec against your server, so I started proof. The
two feed each other - every time proof finds a place where the spec and the
clients disagree, or something the spec doesn’t cover at all, it ends up in
a running list, and from there in the spec itself. (A few of
those are already pull requests against it.)
When the spec and the clients disagree, proof sides with the clients. For
instance, the spec recommends returning the ops in describe as a list, but
CIDER and Calva both expect a map. A server that follows the spec to the
letter, but that CIDER can’t connect to, is not much use to anyone. That’s
also why proof is a compatibility suite and not a conformance suite. You
can only conform to an official spec, and nREPL doesn’t have one yet.
Checking Servers
Checking servers is proof’s main job. It’s a single binary1 that talks to
your server over a socket, the same way any client would. It knows nothing
about the server’s internals and it doesn’t care what language the server is
written in. What it does need is a profile - a small TOML file that tells
proof how to start your server and gives it a few snippets of code in the
server’s language (an idea I borrowed from
jupyter_kernel_test):
name = "My server"
[launch]
command = ["my-server", "--port", "0"]
port-pattern = 'nREPL server started on port (\d+)'
[snippets.value]
code = "(+ 1 2)"
value = "3"
Then you install proof and run it:
$ go install github.com/nrepl/proof/cmd/proof@latest
$ proof run my-server.toml
proof starts your server, runs about 80 checks against it and shuts it down
afterwards. (You can also grab a binary from the
releases, if you don’t have Go
around.) Profiles for nREPL, Babashka, nbb, Basilisp, ClojureCLR, jank,
let-go, dialtone and repartee come with proof, so proof run babashka just
works if you have Babashka installed.
The most important decision in proof’s design is how it grades the checks. A
check fails only when some client won’t work properly with your server (or
when users will lose data), and the report tells you which client it is and
links to the client code in question, pinned to a specific commit. If your
server simply does something differently from the reference implementation,
you get a warning instead. Here’s a failure from a run against Babashka
1.13.226:
FAIL interrupt.other-connection An interrupted eval ends on its own connection
the interrupted eval's done went to the connection the interrupt came from
why: neorepl interrupt stops an eval that another neorepl process
started, and that process waits for the eval's done on its own
connection forever if it goes anywhere else.
see: neorepl interrupt stops an eval from another process
https://github.com/nrepl/neorepl/blob/462a182c.../src/neorepl/main.clj#L374-L385
see: nrepl/nrepl#498 https://github.com/nrepl/nrepl/issues/498
The idea is that a failure should always be worth fixing and never be a
matter of taste. If you think some failure isn’t justified, the links show you
exactly what you need to argue with.
Some checks go a step further and send exactly what a particular client
sends, the way it sends it. cider.connect sends the requests CIDER sends
while connecting, calva.eval evaluates code the way Calva does, and so on
for Conjure, vim-fireplace, REPLy, neorepl, clojure-cli.repl and
bb repl --connect. That way the report tells you directly whether those clients will
work with your server.
Once your server passes everything it can pass, you can list the failures you
can’t fix yet as expected failures in its profile and run proof in your CI
with its GitHub Action. When an expected failure starts passing,
proof fails the build and asks you to remove it from the list, so the list
can’t go stale.
Last, but not least, the bundled profiles run every day and their reports end
up in a compatibility matrix, which shows
where the different servers agree, where they don’t and where the spec needs
some work.
Checking Clients
Servers were always proof’s primary focus, but once the foundation was in
place people started asking me how they could check their clients with it.
That made a lot of sense to me - after all, the same knowledge cuts both ways.
proof has two tools for client authors.
proof proxy sits between a client and a server, passes everything along as
is and records it on the way. You use your client for a while (or run its
test suite through the proxy), press Ctrl-C and proof grades every request
the client sent. Here’s a quick neorepl eval going through it:
$ proof proxy -listen 127.0.0.1:7888 babashka
$ neorepl eval --port 7888 '(println "hi") (+ 1 2)'
client
PASS client.bencode Every frame is valid bencode
PASS client.dict Every request is a dict
PASS client.canonical Frames use canonical bencode
PASS client.id Every request has an id
PASS client.active-id An id isn't reused while its request is active
PASS client.field-types Request fields have the right types
PASS client.required-fields Requests carry the fields their op needs
PASS client.need-input need-input is answered with stdin in the same session
WARN client.close Sessions are closed before disconnecting
PASS client.unknown-op Requests use ops the server supports
9 passed, 0 failed, 1 warnings, 0 skipped
The grading rule is the same, just turned around. A client check fails only
when some server breaks, and it links to the server code in question. (That
warning is fine, by the way. neorepl keeps its session around on purpose, so
the next neorepl eval can use it.)
A proxy sees only what goes over the wire, though, so it can’t tell you how
your client copes with the replies of different servers. That’s what
proof serve is for. It’s a small nREPL server for client test suites, which
behaves like nREPL unless you ask it to behave like some other server in some
particular way - send only the value of the last form (like Basilisp, jank and
dialtone), drop what the code prints to stderr, have no interrupt op,
deliver its replies a byte at a time, and so on. None of those scenarios are
made up. Every one of them is something a real server does (or something TCP
can do to any server’s replies). -like turns on all the quirks of a server
at once, so you can run your client’s tests against “jank” without installing
jank:
$ proof serve -listen 127.0.0.1:7888 -like jank
proof in Practice
So far, so good. But does it actually find anything? You bet! Here are some
of the things proof turned up in the past couple of months:
- nREPL itself sent the end of an interrupted eval to the connection the
interrupt came from (fixed on master), drops the evals waiting behind an
interrupted one (#503) and
can leave a session that stops sending
done after its client goes away in
the middle of an eval (#509).
Oops!
- Babashka, nbb, Basilisp, jank and ClojureCLR all had their share of rough
edges, e.g. errors without an
eval-error status, sessions sharing *1 and
the current namespace, and a request that throws taking its whole connection
down.
- CIDER skipped part of its setup when connecting to a server without
cider-nrepl, Calva and Conjure didn’t sort the keys of the dicts they sent, vim-fireplace
and REPLy never close their sessions, and Conjure’s interrupts don’t say
which eval to stop.
Michiel Borkent fixed the Babashka and nbb issues the same day I reported
them (or the next one), Conjure fixed its encoder the next day and Calva
within a week. That’s the Clojure community for you!
I was also super happy to see other people picking up proof before I even got
around to announcing it. Tom Waddington has been running it against his nREPL
implementations for Janet and
Steel Scheme since early October
(nrepl-janet even tests its client with proof serve), Jeaye Wilkerson
suggested using it for
jank’s nREPL server, and Kyle Cesare is already
fixing what it found there.
That’s exactly what I was hoping for!
The projects that benefited from proof the most were my own, though. This
year I built a few toy nREPL implementations, mostly for fun and to see how
well the protocol holds up outside of Clojure:
- nrepl-beam has nREPL servers for
Erlang (dialtone) and Elixir (repartee). proof found that they didn’t answer
close with session-closed, didn’t echo unknown ops and returned only the
last value of several forms, and fixing those uncovered a race after close
and temporary sessions that never stopped.
- mezcaml, my little OCaml client, read
replies as if only one request was ever in flight, so late output from one
eval could end the next one early.
- neat, my language-agnostic client for
Emacs, got late output, errors without
err and unknown sessions wrong.
- nrepl-sci, the nREPL server behind
the “Try it” section of nrepl.org, was the first server I wrote with proof
around from day one, and working on
it mostly came down
to running proof after every change.
And then there’s
neorepl,
the CIDER-inspired terminal REPL I’ve been working on lately, which is just as
happy to be used by coding agents as by humans. neorepl and proof pretty much
grew up together. I ran neorepl through proof proxy and every scenario of
proof serve, which caught a few bugs before neorepl 0.1 was out, and proof
checks every server with the exact requests neorepl sends. It’s also how I
found the interrupt bugs in nREPL above, which hit neorepl harder than most
clients. Its commands are separate processes sharing a session, so
neorepl interrupt stops an eval that some neorepl eval started, and that
neorepl eval used to wait for its done forever.
A Tool for Tool Authors
I guess I should spell out who proof is for. If you’re using nREPL through CIDER
or Calva you’ll probably never run proof yourself (although hopefully you’ll
feel its effects). proof is a tool for tool authors - people writing nREPL
servers, clients and middleware.
I really hope that it will make it a lot easier to grow the nREPL ecosystem.
These days anyone can put together an nREPL server or client in no time with
the help of tools like Claude. Writing the code is no longer the hard part -
knowing whether the result actually works with CIDER, Calva and friends is.
proof gives you (and your coding agent) a fast feedback loop and failures that
explain themselves: what went wrong, why it matters and which client code
depends on it. That’s usually all an agent needs to fix the problem.
Why “proof”?
Naming is hard! Luckily, the nREPL world has a tradition that makes it a bit
easier, as many nREPL clients are named after drinks (CIDER, neat, mezcaml,
chaser and so on). proof is not a client, so it bends the rule a bit, but it still
belongs at the bar. The proof of a spirit is a measure of its strength, which
is more or less what this tool measures for nREPL servers. And running it
gives you proof that your server works with the existing clients (or a list of
the reasons why it doesn’t). Two meanings for the price of one!2
Epilogue
proof is still in its early days, but it already covers the core of the
protocol, the exact requests of the most popular clients and the server quirks
clients have to deal with. Now it needs more users.
So, if you’re working on an nREPL server or client, please take proof out for
a spin and let me know how it went! I’d love to see your server in the
compatibility matrix, and if you think some failure isn’t fair, open an issue
and let’s argue about it. And if you’re a casual Clojure hacker who’d just like
a nice REPL in the terminal, give neorepl a try:
$ bbin install io.github.nrepl/neorepl
Big thanks to Clojurists Together for supporting my work on nREPL, to Phil
Hagelberg for getting the spec going, and to the maintainers of all the
servers and clients who were so quick to fix the problems proof found. You
rock! In the (n)REPL we trust!
That’s all I have for you today. Keep hacking!