As I’ve written many times, I use Org mode to organize a ridiculous percentage of my life and can’t imagine living without it. It’s just so powerful that I keep finding new uses for it. The problem is, though, it’s so big and powerful that it’s hard for a n00b to get started. Experienced Org users always give the same advice: don’t try to learn it all at once; learn and use some small part of it first and then move on to another aspect of Org. Rinse and repeat.
Eric MacAdie has a slightly more refined version of that advice. As he says, “Learn X to accomplish $SOME-IMPORTANT-TASK” is much easier than just “Learn X”. This being tax season here in the U.S., he suggests learning how to use Org lists to track the receipt of tax forms.
That’s exactly how I learned Org. I started by making a check list of the W2s and 1099s that I expected to get and checked them off as they came in. When the list was complete, I could move on to the next stage of preparing my return. But that was just the first step. Later, I started tracking my tax info all year long. I had an Org table for each class of expense and added each tax deductible item to it as I went along. Each row of the table also had a link to the receipt for the item so they’re easily accessed in case of an audit. Of course, being Org tables, I also had a running total automatically. In one case, only a portion of an expense was deductible and Org could take care of that calculation automatically as well.
Finally, I had a summary table that gathered the totals from each of the tables. That table is completely populated automatically by Org as well. At the end of the year, I simply export the required tables to PDF and send them and the W2/1099 documents to my accountant. It’s still not pleasant—tax time never is—but it’s a lot easier than it used to be.
If you’re looking for a task to apply Org to and you live in a jurisdiction like the U.S. with a completely irrational and dysfunctional filing system, tracking your tax data is a good place to start.
I am now coding a bigger thing in Elisp (and I will definitely blog about it when it’s done), but for now let me say something about an issue I encountered along the way. I needed to display an image between the lines (in the literal sense;-)), preferably not interfering with the editing process – so just inserting two newlines and a space with an image display property inbetween was out of the question. (I wanted the user to be able to edit the buffer irrespective of the image display, and actually inserting anything into the buffer would mess with the undo mechanism).
I use a package called org-rainbow-tags which adds random colours to org tags to provide a consistent colour between identical tags. This helps to identify common tags throughout the file but has the side effect of emphasising the lack of my coherent tag ordering.
I would like to order the tags consistently, just for my own peace of mind! 😀
I had assumed that org-mode came with the built-in ability to sort tags but I couldn’t find any evidence of this so I decided to create a method using my own function. My preferred default method is in descending order as I commonly use a year tag which I would always like to be on the right hand side.
Just select the region containing the tags and run my function passing in the universal argument if you fancy ordering the other way!
(defun my/sort-org-tags-region (beg end &optional reversed)
"In active region sort tags alphabetically in descending order.
With prefix argument REVERSE order." (interactive "r\nP")
(unless (region-active-p) (user-error"No active region to sort!"))
(let* ((str (s-trim (buffer-substring-no-properties beg end)))
(wrd (split-string str ":"t" "))
(new
(concat":" (s-join ":" (sort wrd (if reversed #'string< #'string>)))
":")
)
)
(save-excursion
(goto-char beg)
(delete-region beg end)
(insert new)
)
)
)
Regular readers know that I was a Vi/Vim user for many years before I switched to Emacs. I liked Vim and was happy with it but when I started writing in Lisp dialects, trying Emacs seemed liked a worthwhile experiment. I was pretty quickly captured by the Lisp Machine vibe that Emacs provides and have been a devoted user since.
Still, I have always maintained that Vim is a great editor and the proper choice for those who want just an editor and not an operating environment of the type that Emacs provides. The latest iteration of Vim is Neovim, which among other things has replaced the terrible Vim extension language with Lua. I haven’t used it—Emacs devotee, remember—but it seems like a nice evolution of the Vi/Vim family.
To me, the greatest advantage of the Vi family is its composable key sequences that make learning the editing commands relatively simple. I was, therefore, a little nonplussed by this video on Neovim and Astro. If I had to summarize my impression in a single phrase it would be “bringing Doom Emacs to Vim”. The most salient feature to my mind is the substitution1 of Vim’s composable key sequences with a mishmash of Doom-like key sequences many of which aren’t even mnemonic.
The whole thing seems ill conceived. It has a lot of Emacs features but they seem poorly implemented. For example, the first part of the video involves installing Astro and involves several iterations of restarting Neovim. Emacs users don’t have to perform that dance. To channel Dennis Ritchie, if you want Doom Emacs you know where to find it.
Of course, some folks obviously like the system. You might too so take a look at the video. It’s just over 16 minutes so shouldn’t be too hard to fit in.
As part of the current development target of denote, users can
access the commands we provide via a convenient menu-bar-mode entry.
commit 818c6ae08b32c33893f2bbbc3152e3b64be289c5
Author: Protesilaos Stavrou <info@protesilaos.com>
Date: Fri Mar 31 08:32:37 2023 +0300
Define menu with all Denote commands
denote.el | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
Here is the idea:
[ The exact style of the menu is controlled by the toolkit, not the
active theme. ]
Specific entries are available only in the right context, such as to
toggle the denote-dired-mode when in Dired buffers or to show
backlinks when in a text-mode buffer (or its derivatives, like
org-mode).
The menu enhances the accessibility of Denote by making its commands
easier to discover.
Note that this is a development snapshot. I may still tweak things
and also cover the context-menu-mode, though the intent is clear.
UPDATE 2023-03-31 19:51 +0300. I added support for
context-menu-mode. Also made lots of other tweaks.
commit ecccc046beb502efecb9c4f3e3d845b122a8b34c
Author: Protesilaos Stavrou <info@protesilaos.com>
Date: Fri Mar 31 19:50:31 2023 +0300
Document integration with context-menu-mode (also see 2a1d5eb)
README.org | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
commit 2a1d5eb8f668724e5b37e085e64ff00d44f3eb1a
Author: Protesilaos Stavrou <info@protesilaos.com>
Date: Fri Mar 31 19:43:24 2023 +0300
Add code to integrate menu with context-menu-mode
Enable it with:
(add-hook 'context-menu-functions #'denote-context-menu)
Read what I published about the menu:
<https://protesilaos.com/codelog/2023-03-31-emacs-denote-menu/>.
denote.el | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)
Denote is a simple note-taking tool for Emacs. It is based on the idea
that notes should follow a predictable and descriptive file-naming
scheme. The file name must offer a clear indication of what the note is
about, without reference to any other metadata. Denote basically
streamlines the creation of such files while providing facilities to
link between them.
Denote’s file-naming scheme is not limited to “notes”. It can be used
for all types of file, including those that are not editable in Emacs,
such as videos. Naming files in a constistent way makes their
filtering and retrieval considerably easier. Denote provides relevant
facilities to rename files, regardless of file type.
Here's a few things that I've gotten help with figuring out during the last few
days. Both things are related to my playing with tree-sitter that I've written
about earlier, here and here.
You might also be interested in the two repositories where the full code
is. (I've linked to the specific commits as of this writing.)
This made it a little bit difficult to capture the relevant parts of each
section to implement consult-cabal. I thought a pattern like this ought to
work
but it didn't; I got way too many things captured in type. Clearly I had
misunderstood something about the wildcards, or the query syntax. I attempted to
add a field name to the anonymous node, i.e. change the sections rules like this
That worked, but listing all the sections like that in the query didn't sit
right with me.
Luckily there's a discussions area in tree-sitters GitHub so a fairly short
discussion later I had answers to why my query behaved like it did and a
solution that would allow me to not list all the section types in the query. The
trick is to wrap the string in a call to alias to make it a named node. After
that it works to add a field name to it as well, of course. The section rules
now look like this
With that in place I could improve on the function that collects all the items
for consult-cabal so it now show the section's type and name instead of the
string representation of the tree-sitter node.
State in a consult source for preview of lines in a buffer
I was struggling with figuring out how to make a good state function in order
to preview the items in consult-cabal. The GitHub repo for consult doesn't
have discussions enabled, but after a discussion in an issue I'd arrived at a
state function that works very well.
The state function makes use of functions in consult and looks like this
(defunconsult-cabal--state()"Create a state function for previewing sections."(let((state (consult--jump-state)))(lambda(action cand)(when cand
(let((pos (get-text-property 0 'section-pos cand)))(funcall state action pos))))))
The trick here was to figure out how the function returned by
consult--jump-state actually works. On the surface it looks like it takes an
action and a candidate, (lambda (action cand) ...). However, the argument
cand shouldn't be the currently selected item, but rather a postion (ideally a
marker), so I had to attach another text property on the items (section-pos,
which is fetched in the inner lambda). This position is then what's passed to
the function returned by consult--jump-state.
In hindsight it seems so easy, but I was struggling with this for an entire
evening before finally asking the question the morning after.
Whenever Emacs or Org mode is mentioned on Hacker News, there is usually at least one comment from someone who said they started learning it, but had a hard time sticking with it. It is easier to learn a new technology if you have a goal to use it for something, especially something non-technical. “I ... Read more
During time, I encountered two edge cases that didn’t work with the implementation I originally wrote.
These were self-inflicted edge-cases that related to some idiomatic Ruby📖.
The first edge case was as follows:
After my last post I thought I'd move on to implement the rest of the functions
in haskell-mode's major mode for Cabal, functions like
haskell-cabal-goto-library-section and
haskell-cabal-goto-executable-section. Then I realised that what I really
want is a way to quickly jump to any section, that is, I want consult-cabal!
What follows is very much a work-in-progress, but hopefully it'll show enough
promise.
Listing the sections
As I have a tree-sitter parse tree to hand it is fairly easy to fetch all the
nodes corresponding to sections. Since the last post I've made some
improvements to the parser and now the parse tree looks like this (I can
recommend the function treesit-explore-mode to expore the parse tree, I've
found it invaluable ever since I realised it existed)
That is, all the sections are children of the node called sections.
The function to use for fetching all the nodes is treesit-query-capture, it
needs a node to start on, which this case should be the full parse tree,
i.e. (treesit-buffer-root-node 'cabal) and a query string. Given the
structure of the parse tree, and that I want to capture all children of
sections, a query string like this one works
"(cabal (sections (_)* @section))"
Finally, by default treesit-query-capture returns a list of tuples of the form
(<capture> . <node>), but in this case I only want the list of nodes, so the
full call will look like this
As I envision adding more things to jump to in the future, I decided to make use
of consult--multi. That in turn means I need to define a "source" for the
sections. After a bit of digging and rummaging in the consult source I put
together this
(defvarconsult-cabal--source-section
`(:name"Sections":category location
:action ,#'consult-cabal--section-action
:items ,#'consult-cabal--section-items)"Definition of source for Cabal sections.")
which means I need two functions, consult-cabal--section-action and
consult-cabal--section-items. I started with the latter.
Getting section nodes as items for consult
It took me a while to work understand how this would ever be able to work. The
function that :items point to must return a list of strings, but how would I
ever be able to use just a string to jump to the correct location?
The solution is in a comment in the documentation of consult--multi:
:items - List of strings to select from or function returning
list of strings. Note that the strings can use text properties
to carry metadata, which is then available to the :annotate,
:action and :state functions.
I'd never come across text properties in Emacs before, so at first I
completely missed those two words. Once I'd looked up the concept in the
documentation everything fell into place. The function
consult-cabal--section-items would simply attach the relevant node as a text
property to the strings in the list.
My current version, obviously a work-in-progress, takes a list of nodes and
turns them naïvely into a string and attaches the node. I split it into two functions, like this
(defunconsult-cabal--section-to-string(section)"Convert a single SECTION node to a string."(propertize (format "%S" section):treesit-node section))(defunconsult-cabal--section-items()"Fetch all sections as a list of strings ."(let((section-nodes (treesit-query-capture (treesit-buffer-root-node 'cabal)"(cabal (sections (_)* @section))"
nil nil t)))(mapcar #'consult-cabal--section-to-string section-nodes)))
Implementing the action
The action function is called with the selected item, i.e. with the string and
its properties. That means, to jump to the selected section the function needs
to extract the node property, :treesit-node, and jump to the start of it. the
function to use is get-text-property, and as all characters in the string will
have to property I just picked the first one. The jumping itself I copied from
the navigation functions I'd written before.
(defunconsult-cabal--section-action(item)"Go to the section referenced by ITEM."(when-let*((node (get-text-property 0 :treesit-node item))(new-pos (treesit-node-start node)))(goto-char new-pos)))
Tying it together with consult--multi
The final function, consult-cabal, looks like this
(defunconsult-cabal()"Choose a Cabal construct and jump to it."(interactive)(consult--multi '(consult-cabal--source-section):sort nil))
Conclusions and where to find the code
The end result works as intended, but it's very rough. I'll try to improve it a
bit more. In particular I want
better strings - (format "%S" node) is all right to start with, but in the
long run I want strings that describe the sections, and
preview as I navigate between items - AFAIU this is what the :state field
is for, but I still haven't looked into how it works.
The Free Software Foundation has announced that Eli Zaretskii, the de facto lead Emacs developer, is the winner of the Award for the Advancement of Free Software. It’s well deserved and I offer him my wholehearted congratulations.
Zaretskii has provided exactly the quiet, steady leadership that the project needs. He devotes huge amounts of time to Emacs for free and we should all be extraordinarily grateful. The award is the least the community can do for him and I’m glad he received it.
beframe enables a frame-oriented Emacs workflow where each frame has
access to the list of buffers visited therein. In the interest of
brevity, we call buffers that belong to frames “beframed”.
In short: beframe your buffers, not your outlook. Oops that is for
the philosophy blog! 🙃
A beframed buffer menu
The command beframe-buffer-menu produces a dedicated buffer with a
list of buffers for the current frame. This is the counterpart of
beframe-switch-buffer. When called with a prefix argument (C-u
with default key bindings), it prompts for a frame whose buffers it
will display.
Frame-specific scratch buffer
The user option beframe-create-frame-scratch-buffer allows
beframe-mode to create a frame-specific scratch buffer that runs the
initial-major-mode. This is done upon the creation of a new frame
and the scratch buffer is named after the frame it belongs to. For
example, if the frame is called modus-themes, the corresponding
scratch buffer is *scratch for modus-themes*. Set this user option
to nil to disable the creation of such scratch buffers.
The user option beframe-kill-frame-scratch-buffer is the counterpart
of beframe-create-frame-scratch-buffer. It kills the frame-specific
scratch buffer after the frame is deleted. Set this user option to
nil to disable the killing of such buffers.
Assuming and unassuming buffers
Beframe makes it possible to add or remove buffers from the list of
buffers associated with the current frame. This provides for a
flexible workflow where buffers can be initially beframed yet
consolidated into new lists on demand.
Assuming buffers
To assume buffers is to include them in the buffer list associated
with the current frame.
The command beframe-assume-frame-buffers (alias
beframe-add-frame-buffers) prompts for a frame and then copies its
buffer list into the current frame.
The command beframe-assume-buffers (alias beframe-add-buffers)
adds buffers from a given frame to the current frame. In
interactive use, the command first prompts for a frame and then asks
about the list of buffers therein. The to-be-assumed buffer list is
compiled with completing-read-multiple. This means that the user
can select multiple buffers, each separated by the crm-separator
(typically a comma).
The command beframe-assume-buffers-all-frames prompts with
minibuffer completion for a list of buffers to assume. The
interface is the same as that of beframe-assume-buffers except
that there is no prompt for a frame: buffers belong to the
consolidated buffer list (all frames).
The command beframe-assume-all-buffers-no-prompts unconditionally
assumes the consolidated buffer list.
Unassuming buffers
To unassume buffers is to omit them from the buffer list associated with
the current frame.
The command beframe-unassume-frame-buffers (alias
beframe-remove-frame-buffers) prompts for a frame and then removes
its buffer list from the current frame.
The command beframe-unassume-buffers (alias
beframe-remove-buffers) removes buffers from the current frame.
In interactive use, the to-be-unassumed buffer list is compiled with
completing-read-multiple. This means that the user can select
multiple buffers, each separated by the crm-separator (typically a
comma).
The command beframe-unassume-all-buffers-no-prompts unconditionally
unassumes the consolidated buffer list, but preserves the list
stored in the user option beframe-global-buffers.
commit dfa4678c208e1e5c41413f2d39416f84c21f28ff
Author: Tony Zorman soliditsallgood@mailbox.org
Date: Sat Mar 4 11:48:17 2023 +0100
Add the ability to sort the buffer list
Some completion libraries, like consult, give the user the option to
sort the list of buffers according to some strategy. For example,
sorting by visibility—in the sense that one is first shown hidden
buffers, then visible ones, and only then the current buffer—may be
preferrable when deciding to switch buffers via consult-buffer.
Since beframe.el can be used as a consult source (see the manual),
endowing beframe–buffer-list with an arbitrary sort function greatly
improves the synergy between the two libraries.
Marcin Borkowski (mbork) has been doing more writing lately and has felt the need for a thesaurus. I very seldom use a thesaurus for reasons that I’ve written about previously but it’s sometimes convenient to have one. My typical use case is to avoid reusing the same word too often in close proximity. A thesaurus is good in that situation because I’m just interested in finding an alternative word, not shades of meaning.
Borkowski is trying to decide between two thesauruses, le-thesaurus and mw-thesaurus. Le-thesaurus provides a simple list of synonyms from which you can choose and it will automatically replace the word at point. Mw-thesaurus presents a nicely formatted buffer that gives synonyms and information about shades of meaning. It’s much more comprehensive than le-thesaurus but doesn’t automatically replace the word at point.
I have two thesauruses installed: mw-thesaurus and power thesaurus. Power thesaurus operates much like le-thesaurus but uses a community sourced thesaurus. I almost always use power thesaurus because it best fits my default use case and because when I want shades of meaning and context I turn to Websters 1913, most recently through the newly built-in dictionary-search.
Another problem with mw-thesaurus that I didn’t know about until I read Borkowski’s post is that it globally sets org-hide-emphasis-markers to t, which hides all Org emphasis markers in all buffers. That’s not something I want, and I’ve occasionally noticed it happening in the past but I didn’t know why. Borkowski solved that by editing the source code but perhaps there’s a better way.
Regardless of which one(s) you install, it’s often handy to have a thesaurus available without having to leave the comfort of Emacs. Any of those discussed here are fine and there are doubtless other good ones.
Over the years I’ve heard countless complaints about nREPL and its numerous
perceived flaws. I dawned me today that it might be a good idea to write this
“living”1 article that will try to catalog and address some of them.
I’ll focus mostly on the common complaints in the Clojure community, but I’ll also
touch upon some of the broader perceived issues with the nREPL protocol itself.
Bencode sucks
A lot has been said about the choice of
Bencode for encoding data between the
client and the server. The format was very primitive they said. JSON is better!
EDN is better! X is better!
Of course, back when this decision was made EDN didn’t even exist and for most
editors dealing effectively with JSON was a very tall order. For clients
like vim, Emacs, etc it was much easier to write a simple bencode implementation
than to have to create a fast (and correct) JSON parser in VimScript or Emacs Lisp.
Admittedly, I struggled for a while to get even the CIDER bencode parser working properly!
Bencode has many limitations, but it was the widest common denominator and in my
opinion its choice contributed a lot to the early success of nREPL.
I find it funny that even though the Clojure nREPL implementation eventually
provided JSON (Transit) (see https://github.com/nrepl/fastlane) and EDN
(built-in) transports, practically no one uses them.2
I also can’t expect any language-agnostic protocol to adopt a niche format like EDN and
expect any form of wide adoption.
It’s not a real REPL (a.k.a. Rich doesn’t like it)
REPL stands for something - Read, Eval, Print, Loop.
It does not stand for - Eval RPC Server/Window.
It is not something I “feel” about. But I do think we should be precise about what we call REPLs.
While framing and RPC orientation might make things easier for someone who just wants to implement an eval window, it makes the resulting service strictly less powerful than a REPL.
The above is from a 2015 conversation around the introduction of Clojure’s socket REPL server and what (nREPL) problems does it solve. I’ve seen this conversation cited many times over the years to support the thesis that nREPL is problematic in one regard or another.
So, is nREPL a real REPL or not? Does this even matter if it does what you need it to do?
Well, nREPL wasn’t meant to be a “real REPL”. It was meant to be a REPL server
that makes easy for people to build tools (e.g. Clojure editor plugins)
upon. And framing requests and responses makes a huge difference there. Sure, it
might look appealing to be relying only on eval for everything, but you still
need to be able to process whatever you got from eval. It’s also pretty handy to be able to
match requests and responses that originated from them.
I guess nREPL’s approach was eventually “redeemed” when the socket REPL didn’t gain broad
tool adoption, and was followed up by prepl that framed the input. Still, it
didn’t gain broad adoption for development tooling as was hard to deal with its
unstructured output.
Now things have come full circle with a recently introduced Clojure REPL (framed request, framed response).
To wrap it up here - we can either be dogmatic or practical. I always go for being practical - it might not be a real REPL, but it sure gets real work done.
It’s a 3rd-party application
Can’t argue with that one. That obviously puts nREPL at an disadvantage when it comes to Clojure’s built-in socket REPLs or REPLs that get “injected” by clients (e.g. the now defunct unrepl).
On the bright side - this also untangles nREPL from the Clojure’s release cycle. And makes contributing to the project a lot easier.
By the way, I always felt this particular concern to be quite overblown as nREPL is small self-contained library with no external dependencies. Also it’s easy to have it only in development profile (where it’s typically used). Technically it’s also possible to “upgrade” any socket REPL to an nREPL, but we never finished the work in this direction. (see https://github.com/nrepl/nrepl/commits/injection) I guess there wasn’t enough demand for this.
One more thing - tools like CIDER and Calva have been injecting nREPL during
development automatically for quite a while, which blurs quite a bit the line
between built-in and third-party. The point I’m trying to make here is that
there’s a fine line between potential issues and real issues. I prefer to focus
on the latter group.
Middleware suck (insert here every possible reason why)
Maybe they do. Maybe they don’t. Obviously they are concept that predates nREPL and people have been divided about it pretty much its entire existence.
I’m a bit biased, having worked with middleware for many years (long before I
got interested in Clojure) and I’ve always liked the idea of being able to
process requests and responses like a pipeline transforms them. But I also
realize that comes with a certain learning curve and the ordering of the
pipeline steps is not always clear, which can result in various problems.
I guess no middleware is more controversial than
piggieback, the one responsible for
ClojureScript support. Love it or hate it, there’s no denying that the decision
to organize nREPL’s functionality around middleware made it possible to provide
ClojureScript support (ClojureScript didn’t even exist at the time
when nREPL was created) without having to touch the nREPL codebase. It also made
possible some fairly interesting workflows like hosting a Clojure and a
ClojureScript on the same nREPL server. And, of course, this also means that anyone can
provide a replacement for piggieback if they wish to.
At the end of the day, however, one should not forget
that middleware is just an implementation detail in the reference Clojure nREPL server
and you can implement the nREPL protocol without middleware. Babashka’s nREPL server is a nice example of such an approach.
I was hoping that by now there would also be more native ClojureScript nREPL servers, but alas. Maybe this year?
It’s Clojure-only (or optimized for Clojure)
No, it’s not. Just take a look at the nREPL protocol ops (sans session management ops):
completions
eval
interrupt
load-file
lookup
stdin
Which one of them seems Clojure-specific to you?
Yeah, the protocol design was obviously influenced by Clojure and we never gave
much thought on how to standardize nREPL certain responses, but nREPL was always meant
to be a language-agnostic protocol.
Admittedly it didn’t gain much traction outside of the Clojure & broader Lisp community, that’s a different matter altogether.
The protocol is poorly documented
Well, that’s more or less true as of today, but I plan to address this eventually. In particular the structure of responses is mostly
undocumented, which is forcing some people reverse engineer it from existing nREPL implementations. That response structure also needs
a bit of consideration as ops like lookup have responses that are somewhat linked to common Clojure var metadata.
If you’ve always wanted to contribute to open-source projects - that’s a great way to get started.
There’s no standard compatibility test suite
Basically, we need a standard test suite that checks whether nREPL implementation conform to the nREPL protocol specification.
This would make it much easier to build nREPL servers and to verify their proper work.
That’d be certainly nice to have and I think it’s very doable. Perhaps one of you, dear readers, is looking for a fun small OSS project?
First of all, nREPL is mainly designed for Clojure, tested against Clojure, and implements features that Clojure needs in a pretty specific Clojure way.
That’s certainly not how I see things. I believe Chas Emerick wouldn’t agree with the assessment either.
Many misconceptions about nREPL originate from a conflation of the nREPL protocol and the reference Clojure implementation. When I took over nREPL
I expanded the documentation greatly and made the protocol more prominent here and there.
Still, a lot of damage has already been done and we’ll need to do more to separate those clearly in the minds of most people.
I see this quite linked to documenting the protocol clearly and building a standard nREPL compatibility test suite.
Epilogue
No software is perfect and neither is nREPL. It grew organically over the course
of many years and it definitely accumulated a bit of weirdness and cruft along
the way. And a lot of charm, plus a track record of stability and robustness.
Would we have done everything the same way knowing
what we know today? Probably not. But nREPL still gets the job done and it’s
still one of the better options for tool authors who don’t want to reinvent the
wheel every time.
I get that it’s always fun to complain about something4 and to work on exciting
new alternatives5, but I think in the spirit of Clojure it’s also not a bad idea
to appreciate the value of stability and reliability. And keep improving a good
thing.
Next time you’re very frustrated with nREPL consider trying to find a solution for your problem that everyone
in the community might benefit from. New is always better until it’s not. In the (n)REPL we trust!
P.S. If I forgot to mention something you hate about nREPL, please share it in the comments.
Admittedly we never adjusted the protocol structure for them, meaning those transports don’t make use of something that can’t be represented with Bencode. ↩
Over the years I’ve heard countless complaints about nREPL and its numerous
perceived flaws. I dawned me today that it might be a good idea to write this
“living”1 article that will try to catalog and address some of them.
I’ll focus mostly on the common complaints in the Clojure community, but I’ll also
touch upon some of the broader perceived issues with the nREPL protocol itself.
Bencode sucks
A lot has been said about the choice of
Bencode for encoding data between the
client and the server. The format was very primitive they said. JSON is better!
EDN is better! X(ML) is better!
Of course, back when this decision was made EDN didn’t even exist and for most
editors dealing effectively with JSON was a very tall order. For clients
like vim and Emacs it was much easier to write a simple bencode implementation (typically a few hundred lines of code)
than to have to create a fast (and correct) JSON parser in VimScript or Emacs Lisp.
Admittedly, I struggled for a while to get even the CIDER bencode parser working properly!
You also can’t expect any language-agnostic protocol to adopt a niche format like
EDN and expect any form of wide adoption. Sure, it’d be a great fit for Clojure,
but it would impose a heavy burden on the clients if they don’t have access to
Java or JavaScript libraries.
Bencode has many limitations, but it was the widest common denominator and in my
opinion its choice contributed a lot to the early success of nREPL.
I find it somewhat funny that even though the Clojure nREPL implementation
eventually provided JSON (Transit) (see https://github.com/nrepl/fastlane) and
EDN (built-in) transports, practically no one uses them.2
Note: Fun trivia - the very first version of nREPL didn’t use Bencode! You
can find the reasons for the adoption of Bencode here.
It’s not a real REPL (a.k.a. Rich doesn’t like it)
REPL stands for something - Read, Eval, Print, Loop.
It does not stand for - Eval RPC Server/Window.
It is not something I “feel” about. But I do think we should be precise about what we call REPLs.
While framing and RPC orientation might make things easier for someone who just wants to implement an eval window, it makes the resulting service strictly less powerful than a REPL.
The above excerpt is from a 2015
conversation
around the introduction of Clojure’s socket REPL server and what (nREPL)
problems does it solve. I’ve seen this conversation cited many times over the
years to support the thesis that nREPL is problematic in one regard or another.
So, is nREPL a real REPL or not? Does this even matter if it does what you need
it to do?
Well, nREPL was never meant to be a “real REPL”. It was meant to be a REPL
server that makes it easy for people to build development tools (e.g. Clojure
editor plugins) upon. And framing requests and responses makes a huge difference
there. Sure, it might look appealing to be relying only on eval for
everything, but you still need to be able to process whatever you got from
eval. It’s also pretty handy to be able to match requests and responses that
originated from them.
I guess nREPL’s approach was eventually “redeemed” when the socket REPL didn’t gain broad
tool adoption, and was followed up by prepl that framed the input. Still, it
didn’t gain broad adoption for development tooling as was hard to deal with its
unstructured output.
Now things have come full circle with a recently introduced Clojure REPL (framed request, framed response).
To wrap it up here - we can either be dogmatic or practical. I always go for being practical - it might not be a real REPL, but it sure gets real work done.
Note: You can find a short comparison of nREPL, the socket REPL and prepl here.
It’s a 3rd-party application
Can’t argue with that one. That obviously puts nREPL at an disadvantage when it comes to Clojure’s built-in socket REPLs or REPLs that get “injected” by clients (e.g. the now defunct unrepl).
On the bright side - this also untangles nREPL from the Clojure’s release cycle. And makes contributing to the project a lot easier.
By the way, I always felt this particular concern to be quite overblown as nREPL is small self-contained library with no external dependencies. Also it’s easy to have it only in development profile (where it’s typically used). Technically it’s also possible to “upgrade” any socket REPL to an nREPL, but we never finished the work in this direction. (see https://github.com/nrepl/nrepl/commits/injection) I guess there wasn’t enough demand for this.
One more thing - tools like CIDER and Calva have been injecting nREPL during
development automatically for quite a while, which blurs quite a bit the line
between built-in and third-party. The point I’m trying to make here is that
there’s a fine line between potential issues and real issues. I prefer to focus
on the latter group.
Middleware suck (insert here every possible reason why)
Maybe they do. Maybe they don’t. Obviously they are concept that predates nREPL and people have been divided about it pretty much its entire existence.
I’m a bit biased, having worked with middleware for many years (long before I
got interested in Clojure) and I’ve always liked the idea of being able to
process requests and responses like a pipeline transforms them. But I also
realize that comes with a certain learning curve and the ordering of the
pipeline steps is not always clear, which can result in various problems.
I guess no middleware is more controversial than
piggieback, the one responsible for
ClojureScript support. Love it or hate it, there’s no denying that the decision
to organize nREPL’s functionality around middleware made it possible to provide
ClojureScript support (ClojureScript didn’t even exist at the time
when nREPL was created) without having to touch the nREPL codebase. It also made
possible some fairly interesting workflows like hosting a Clojure and a
ClojureScript on the same nREPL server. And, of course, this also means that anyone can
provide a replacement for piggieback if they wish to.
At the end of the day, however, one should not forget
that middleware is just an implementation detail in the reference Clojure nREPL server
and you can implement the nREPL protocol without middleware. Babashka’s nREPL server is a nice example of such an approach.
I was hoping that by now there would also be more native ClojureScript nREPL servers, but alas. Maybe this year?
It’s Clojure-only (or optimized for Clojure)
No, it’s not. Just take a look at the nREPL protocol ops (sans session management ops):
completions
eval
interrupt
load-file
lookup
stdin
Which one of them seems Clojure-specific to you?
Yeah, the protocol design was obviously influenced by Clojure and we never gave
much thought on how to standardize nREPL certain responses, but nREPL was always meant
to be a language-agnostic protocol.
Admittedly it didn’t gain much traction outside of the Clojure & broader Lisp community, that’s a different matter altogether.
It’s bloated and has too many (useless) features
Some people feel that over the course of recent years nREPL has become too bloated with the additions of features like:
Support for Unix sockets (as an alternative of using TCP)
Support for EDN (instead of Bencode)
Dynamic middleware loading/unloading
TLS encryption of the TCP connections
The addition of completion and lookup middleware
I get their point. nREPL is definitely bigger and more complex than it used to
be when I took over project 5 years ago. I’ve been quite selective with the approval
of new features and my general approach has been to say “No, thanks.”, unless
there’s a lot of demand or a great justification for something. I’ve also
avoided anything that would introduce external dependencies for the project, so
it’s still 100% self-contained.
Yeah, obviously all of those feature additions are not really “must haves” (but
what is really?) and some of them could have been distributed as external
middleware packages. Still, I felt that a good out-of-the-box experience was
preferable over a minimal core in the context of a tool like nREPL. Time will
tell whether I was right or wrong. You can also share your take on this in
the comments.
Note: Most of those newer features are marked as “experimental” and might be removed down the road if we notice issues with them or they don’t gain traction.
One more thing - it has been pointed out that because of the growing nREPL
codebase it loads slower than it used to (simply because you have to load more
Clojure code). While I acknowledge the problem, I think that the impact
(something like an extra 500ms on application startup from what I saw
somewhere) is not that big given most Clojure workflows don’t involve restarting
applications all the time. I often work with the same instance of a Clojure
application for days.
It doesn’t have enough features
For everyone who thinks nREPL is bloated there’s also someone who feels it
doesn’t have enough features. People would love to see things like “find
references” support, more built-in transports, built-in support for
ClojureScript, debugging, rich data types, etc.
That’s one of the realities of life - no matter what you do you can’t please
everyone. But I keep trying regardless of the odds being stacked against me!
The protocol is poorly documented (lack of formal specification)
Well, that’s more or less true as of today, but I plan to address this eventually. In particular the structure of responses is mostly
undocumented, which is forcing some people reverse engineer it from existing nREPL implementations. That response structure also needs
a bit of consideration as ops like lookup have responses that are somewhat linked to common Clojure var metadata.
If you’ve always wanted to contribute to open-source projects - that’s a great way to get started.
There’s no standard protocol compatibility test suite
Basically, we need a standard test suite that checks whether nREPL implementation conform to the nREPL protocol specification.
This would make it much easier to build nREPL servers and to verify their proper work.
That’d be certainly nice to have and I think it’s very doable. Perhaps one of you, dear readers, is looking for a fun small OSS project?
First of all, nREPL is mainly designed for Clojure, tested against Clojure, and implements features that Clojure needs in a pretty specific Clojure way.
That’s certainly not how I see things. I believe Chas Emerick wouldn’t agree with the assessment either.
Many misconceptions about nREPL originate from a conflation of the nREPL protocol and the reference Clojure implementation. When I took over nREPL
I expanded the documentation greatly and made the protocol more prominent here and there.
Still, a lot of damage has already been done and we’ll need to do more to separate those clearly in the minds of most people.
I see this quite linked to documenting the protocol clearly and building a standard nREPL protocol compatibility test suite.
Epilogue
No software is perfect and neither is nREPL. It grew organically over the course
of many years and it definitely accumulated a bit of weirdness and cruft along
the way. And a lot of charm, plus a track record of stability and robustness.
Would we have done everything the same way knowing
what we know today? Probably not. But nREPL still gets the job done and it’s
still one of the better options for tool authors who don’t want to reinvent the
wheel every time.
I get that it’s always fun to complain about something4 and to work on exciting
new alternatives5, but I think in the spirit of Clojure it’s also not a bad idea
to appreciate the value of stability and reliability. And keep improving a good
thing.
Next time you’re very frustrated with nREPL consider trying to find a solution for your problem that everyone
in the community might benefit from. New is always better until it’s not. In the (n)REPL we trust!
P.S. If I forgot to mention something you hate about nREPL, please share it in the comments.
Admittedly we never adjusted the protocol structure for them, meaning those transports don’t make use of something that can’t be represented with Bencode. ↩
When I’m writing about or in Ruby📖 code, sometimes I want to grab the qualified method name. For example, let’s say I have the following Ruby code:
module Hello
module World
def foo
:bar
end
def self.call
:yup
end
end
end
The qualified method name for the method #foo would be Hello::World#foo. The qualified method name for the singleton method .call is Hello::World.call.
A Ruby documentation convention is that instance methods are prefix with a # and singleton methods are prefixed with a . or ::.
Using treesit-explore-mode, I was able to quickly refine my recursive queries. Below is treesit’s rendering of the Abstract Syntax Tree (AST 📖) of the above Ruby code:
, in a moment of dreary skies and sleeping dogs, I hacked together the following functions:
jf/treesit/qualified_method_name
Copy the qualified method name to the paste buffer (e.g. the kill-ring).
jf/treesit/module_space
Recurse up from a node to create a list of the module/class ancestors.
(require 'treesit)
(cl-defun jf/treesit/qualified_method_name (&key (type "method"))
"Get the fully qualified name of method at point."
(interactive)
(if-let ((func (treesit-defun-at-point)))
;; Instance method or class method?
(let* ((method_type (if (string= type
(treesit-node-type func))
"#" "."))
(method_name (treesit-node-text
(car (treesit-filter-child
func
(lambda (node)
(string=
"identifier"
(treesit-node-type node)))))))
(module_space (s-join "::"
(-flatten
(jf/treesit/module_space func))))
(qualified_name (concat module_space
method_type
method_name)))
(message qualified_name)
(kill-new (substring-no-properties qualified_name)))
(user-error "No %s at point." type)))
;; An ugly bit of code to recurse upwards from the node to the "oldest"
;; parent. And collect all module/class nodes along the way. This will
;; return a series of nested lists. It's on the originating caller to
;; flatten that list.
(defun jf/treesit/module_space (node)
(when-let* ((parent (treesit-parent-until
node
(lambda (n) (member (treesit-node-type n)
'("class" "module")))))
(parent_name (treesit-node-text
(car (treesit-filter-child
parent (lambda (n)
(string=
"constant"
(treesit-node-type n))))))))
(list (jf/treesit/module_space parent) parent_name)))
This is most certainly a rainy day kind of project; one that helped me learn just a bit more about the treesit package.
Postscript
The list returned by jf/treesit/module_space is '(nil ("Hello" ("World"))); which is a ugly but workable. Perhaps someone will write to me with a refactor of this code.
When I’m writing about or in Ruby📖 code, sometimes I want to grab the qualified method name. For example, let’s say I have the following Ruby code:
module Hello
module World
def foo
:bar
end
def self.call
:yup
end
end
end
The qualified method name for the method #foo would be Hello::World#foo. The qualified method name for the singleton method .call is Hello::World.call.
A Ruby documentation convention is that instance methods are prefix with a # and singleton methods are prefixed with a . or ::.
Using treesit-explore-mode, I was able to quickly refine my recursive queries. Below is treesit’s rendering of the Abstract Syntax Tree (AST 📖) of the above Ruby code:
, in a moment of dreary skies and sleeping dogs, I hacked together the following functions:
jf/treesit/qualified_method_name
Copy the qualified method name to the paste buffer (e.g. the kill-ring).
jf/treesit/module_space
Recurse up from a node to create a list of the module/class ancestors.
(require 'treesit)
(cl-defun jf/treesit/qualified_method_name (&key (type "method"))
"Get the fully qualified name of method at point."
(interactive)
(if-let ((func (treesit-defun-at-point)))
;; Instance method or class method?
(let* ((method_type (if (string= type
(treesit-node-type func))
"#" "."))
(method_name (treesit-node-text
(car (treesit-filter-child
func
(lambda (node)
(string=
"identifier"
(treesit-node-type node)))))))
(module_space (s-join "::"
(-flatten
(jf/treesit/module_space func))))
(qualified_name (concat module_space
method_type
method_name)))
(message qualified_name)
(kill-new (substring-no-properties qualified_name)))
(user-error "No %s at point." type)))
;; An ugly bit of code to recurse upwards from the node to the "oldest"
;; parent. And collect all module/class nodes along the way. This will
;; return a series of nested lists. It's on the originating caller to
;; flatten that list.
(defun jf/treesit/module_space (node)
(when-let* ((parent (treesit-parent-until
node
(lambda (n) (member (treesit-node-type n)
'("class" "module")))))
(parent_name (treesit-node-text
(car (treesit-filter-child
parent (lambda (n)
(string=
"constant"
(treesit-node-type n))))))))
(list (jf/treesit/module_space parent) parent_name)))
This is most certainly a rainy day kind of project; one that helped me learn just a bit more about the treesit package.
Postscript
The list returned by jf/treesit/module_space is '(nil ("Hello" ("World"))); which is a ugly but workable. Perhaps someone will write to me with a refactor of this code.
It seems as if every second technical article or tweet that I read is about Large Language Models (LLMs) and, in particular, the various iterations of ChatGPT. In response to this, there are already several Emacs packages providing an interface to ChatGPT.
Over at the Emacs subreddit, ahyatt argues that Emacs is the best platform for LLMs. The main reason for this, he says, is that the primary thing you want ChatGPT to do is integrate with your writing environment and Emacs excels at this. Furthermore, with Emacs, this integration is universal: You can use it with code, documentation, technical papers, emails, and just about any other type of writing.
Ahyatt also mentions that it’s often useful to provide context with a ChatGPT prompt and Emacs makes this easy to do. He goes on to speculate on other advanced use of ChatGPT and how they can be easily enabled with Emacs. Take a look at his post for the details.
Whether or not ChatGPT is the long awaited—or dreaded—revival of serious AI, those who want to experiment with or work on LLMs need a platform from which to do so and as ahyatt argues, Emacs is an excellent choice for that.
Recently I read a post by @nikitonsky about writing a custom REPL for Clojure and Sublime Text.
What got my attention was a way of implementing a protocol over a plain Clojure REPL.
In the post, Nikita describes a way to “upgrade” the connection by sending code that basically implements a new REPL to the main REPL process.
I liked the idea and decided that two can play this game, and started working on a similar thing, but for Fennel.
A few months ago I’ve already proposed the idea of a simple protocol for the Fennel REPL.
The idea was met with a certain amount of skepticism because there’s already an implementation of a such protocol that is language agnostic (in theory) and battle-tested.
The protocol in question is called nREPL and it is widely used in the Clojure ecosystem.
There are several client implementations for various editors, and the protocol is extensible enough to support other languages and various clients.
In fact, Fennel already has an implementation of the nREPL protocol, called jeejah.
However, it has problems.
First of all, nREPL is mainly designed for Clojure, tested against Clojure, and implements features that Clojure needs in a pretty specific Clojure way.
It’s not impossible to implement support for other languages, and there were attempts of various completeness.
Another problem is that nREPL has “n” in its name for a reason.
It’s a Network REPL, which means that the protocol is designed around network communication pipelines.
In principle, it is possible to implement a standard-input/-output or a pipe-based version, but no clients will probably support it.
And Fennel, running on Lua, has some problems with networking.
The luasocket library becomes a dependency and there’s no clear way of implementing asynchronous communication.
Again, not impossible, JeeJah does it, but it’s hard to do it properly.
There’s also bencode.
It’s an encoding that is used by torrents.
Encoding and decoding messages may be slow, especially in a single-threaded REPL.
And we’ll need to re-implement half of fennel.view the serialization function to support all kinds of Lua quirks.
Nikita also has some thoughts on nREPL:
You need to add nREPL server dependency to your app. It also has a noticeable startup cost (~500ms on my machine).
Which is a good point, although it’s not completely true.
Some existing nREPL clients, such as CIDER, can inject the nREPL dependency into your project, so you don’t need to actually ship it with your application.
You develop with nREPL, and use all of its goodies, yet your app has no nREPL once it’s shipped.
So you only need to include the nREPL dependency if you want to ship your application with capabilities for remote code execution.
This is not really possible with Fennel, however.
There’s no package manager for fennel, and neither there is a way to dynamically inject an entire nREPL library from the client.
But what about injecting a smaller and simpler protocol?
Why, yes!
That’s exactly what Nikita is doing in their Clojure-Sublimed plugin, and I’ve decided to do the same in Emacs.
The protocol
If you’ve read the mailing list discussion I linked earlier, you’ve seen that I envisioned line-based, tab-separated messages.
I’ve scraped the idea and went for data-based communication: table in, plist out.
Tables can be interpreted by fennel, we can pattern-match on them, and store arbitrary amounts of data.
Plists are natively understood by Emacs Lisp, which is an implementation language of our client.
So my idea was that I send something like that to Fennel REPL:
I’ll explain why there are three outgoing messages for one incoming, but a bit later.
Right now we need to address the main problem: how do we teach Fennel to work in terms of our protocol?
When I thought about the protocol for the first time, I thought that it will be part of the Fennel REPL by default.
However, Fennel is quite minimalist, and the choice was not to include anything like that.
The REPL is already better than the Lua one.
This is true when we’re talking about interactive usage, but not when we’re talking about machine interaction.
So the answer is - REPL upgrade.
Upgrading the REPL
This technique is using the ability to evaluate in the existing REPL, basically implementing a different REPL at runtime.
Sending it to Fennel REPL starts our own REPL inside of it:
Welcome to Fennel 1.3.0 on PUC Lua 5.4!
Use ,help to see available commands.
Try installing readline via luarocks for a better repl experience.
>> (let [fennel (require :fennel)] (while true (io.write "repl>> ") (io.flush) (print (fennel.eval (io.read)))))
repl>> (+ 1 2 3)
6
However, it’s more of a downgrade, rather than an upgrade.
Our prompt doesn’t support unfinished expressions, comma-commands no longer work, and so on.
The purpose of this example is to give you a general idea of how the upgrade process works.
Reimplementing the REPL from scratch is a tough task, but we don’t even have to do it!
Fennel comes with an extensible REPL already: the fennel.repl function accepts a table with the following callbacks:
readChunk - poll for the user input.
onValues - called when the REPL returns the result of the evaluation,
onError - called when the error occurs,
pp - pretty-printer function
If we implement these functions we’ll get a REPL that works the way we need it to.
Implementing the Proto(col|type) REPL
First things first, we need fennel the library.
We’ll use some of its public API, later on, but the feature we’re interested in the most right now is fennel.repl:
I’ll go through the implementation of each callback but I will not implement the whole protocol as part of this post.
The protocol while small, still has a lot of code, so I’ll only do parts that pose interesting challenges.
It will still be a functional protocol, just not as feature-full as the one I’ve actually implemented for Emacs integration.
First things first, let’s make the read-chunk callback.
In the built-in REPL, it is used to print the prompt and handle user input.
We don’t need the prompt, as this will not be an interactive REPL, but we do need to poll for user input.
Not just that, our input comes in a form of a message, so we also need to parse it.
(fn protocol.read-chunk [parser-state]
(io/write">> ") (io.flush)
(let [message (io/read:l)
(ok?message) (pcall evalmessage)]
(if ok? (casemessage {: id:evaldata} (protocol.acceptid:evaldata)
_ (error "message did not conform to protocol"))
(error (.. "malformed input: " (tostring message))))))
In the read-chunk function we’re reading a line from the client and eval it.
I could have used just the parser here, but you’ll see why having eval here is useful.
If the evaluation was successful, we call case and do pattern matching on the message.
If the message matches any pattern (right now the only one) we know how to process it.
The processing is handled by the accept function, so let’s implement it:
Nice!
Now, obviously, this omits a lot of plumbing, error handling, other operations, and so forth, but it should give you the idea.
Also, there are a lot of unnecessary table lookups, I just wrote the code in such a way so you could evaluate them one after another without forward declarations.
But, as a result, sending each of those code blocks to a plain Fennel REPL gives us a new REPL that acts in terms of a specified protocol.
Now we’ve really upgraded the REPL.
Unfortunately, this will not work as is, not because we’re missing ops or error handling though.
There’s an elephant in the room, consider this message: {:id 1 :eval "(print :foo)"}.
What would happen if we send this message to our REPL?
Now our REPL suddenly waits for user input midway.
So if another protocol message comes, it will be consumed by this read.
So we have to handle this too.
Handling IO
Unfortunately, Lua doesn’t have any way of redirecting output from standard out to something else.
Clojure kinda does:
(with-out-str (.printlnSystem/out"foo"))
Even though we call the println method of the System/out class, this expression still returns a string "foo\n".
It is possible because in the JVM there are ways to configure that.
I’ve tried using io.output in Fennel, but it can only be set to a file, we can’t pass it a table that implements the necessary methods:
So we need to set up IO in such a way that it works as expected when we communicate through a protocol, yet wraps the IO inside of the user’s code.
Fortunately, we can do that:
It’s a bit of code, but what it essentially does is this:
store the original values of stdin, stdout, stderr,
define a bunch of replacement functions, like env.print, env.io.write,
capture original *FILE metatable in fd and override it’s metamethods.
Unlike tables, all file objects in Lua share the same metatable, so we only need to redefine metatable entries for the stdin file.
Though it may depend on the implementation of the Lua runtime being used.
The env argument here is a table that will be used in the fennel.repl, so we need to change that bit of code.
Because we don’t want to actually modify the real _G table, we’ll need to copy it first:
Great, now we get a message with the print OP, and the data to be printed.
The client then can detect such message and act accordingly.
We can even add support for different descriptors, hide stderr messages, or color them differently.
But what about the opposite operation?
How do we read input from the user?
This is a bit tricky.
The general idea is the same, we need to tell the client that the REPL is awaiting input, but we also need to wait in such way that other messages don’t get confused with user input.
You can see that in the set-io function, the io.read calls protocol.read if the descriptor is stdin.
Let’s implement protocol.read now:
Unfortunately, Lua doesn’t have an inbuilt filesystem library, so this method is not cross-platform.
It will probably work under BSD systems, and Macs, but not under Windows, unless you’re using WSL, but still not guaranteed.
I don’t have a Windows PC, so I can’t test, but if you know a cross-platform, or even just a special case for Windows, for creating a named FIFO pipe, and attaching to it, it’d be great if you reach me.
This pretty much concludes our very basic, yet fully functional protocol!
The IO part is the trickiest part because there may be specifics to how files are read, but again, we can’t expect that the target application will have luafilesystem installed.
But there’s a way to work this around.
So we do have a final small change to make in our read-chunk function.
Making the protocol extensible at runtime
You may remember that I’ve chosen eval over parser in the REPL to work with incoming messages.
There was a specific reason for it - the message can actually be not for the target application, but for the REPL itself!
And we don’t even need any kind of special-casing for it, only a nop OP:
(fn protocol.read-chunk [parser-state]
(io/write">> ") (io.flush)
(let [message (io/read:l)
(ok?message) (pcall evalmessage)]
(if ok? (casemessage {: id:evaldata} (protocol.acceptid:evaldata)
{: id:nop""} "\n"_ (error "message did not conform to protocol"))
(error (.. "malformed input: " (tostring message))))))
We also need to expose the protocol table in the _G table, and fennel for convenience:
(set _G.proto-replprotocol)
(set _G.fennelfennel)
With these changes, we can send a message like (do (print "hi!") {:id 1 :nop ""}) and it will be processed by the eval, and the print call will be evaluated outside of the main protocol handling.
Thus, we can get fancy!
Here, for example, how we can make our protocol respond in JSON:
Now, if we send this to the REPL, we will get nothing back, since the OP in question is nop, however, the format function will be redefined.
So if we send one more message, the usual eval one, we’ll get back JSON messages in response:
This is why languages with the capability to run any code at runtime are cool in my opinion.
We can build our application while it is running.
Any protocol method, that we’ve defined can be re-implemented at runtime.
So if you’re running Windows, you can redefine protocol.read to support your particular platform without even asking me to update the code.
(But please, if you know a more portable way of handling input, send me a message).
As a final change, let’s remove the >> prompt from the read-chunk, and we’re done.
The prompt is not needed, it was only to help you differentiate what messages we’ve sent, and what messages we receive.
For the machine the prompt will only get in the way:
(fn protocol.read-chunk [parser-state]
(let [message (io/read:l)
(ok?message) (pcall evalmessage)]
(if ok? (casemessage {: id:evaldata} (protocol.acceptid:evaldata)
{: id:nop""} "\n"_ (error "message did not conform to protocol"))
(error (.. "malformed input: " (tostring message))))))
Finally, we can add the call to proto-repl to our protocol code, so right after we’ve sent it the new REPL is started automatically:
(proto-repl)
Now, for real, this concludes the basic protocol implementation.
This is, in fact, what I started with, and then gradually made it more mature by extending the number of operations the protocol supports, and making it more robust.
Not every protocol method is exposed to be changed in my implementation though.
This was more like a demo that you can follow, but in the real version of the protocol, it is a single giant function that gets sent to the REPL in one go.
Now, we can talk about the client, because without the client our protocol has no value.
The client
My editor of choice is Emacs for many reasons.
And this is one of them - we can make all kinds of applications in Emacs, including a custom client for a custom protocol.
We’re like full-stack developers now, but our backend is written in Fennel and our frontend will be written in Emacs Lisp.
As I mentioned at the beginning of the post, I’ve chosen plists because Emacs understands them natively.
Still, our messages come in as strings, so we’ll need to parse them.
But how will we organize our client to begin with?
Comint
The current implementation of the Fennel REPL in the fennel-mode package uses the inbuilt comint package to do all of the heavy lifting.
Comint is Emacs’ generic interface for providing user interaction to asynchronous processes.
The selling point is that you can start almost any interactive program in Comint, set the prompt regexp and it will work.
However, the problems start when we begin building a machine-to-machine interface over a human-to-machine interface.
As with any kind of interface it involves parsing the output.
Our protocol is no different here, we’re still going to parse output, so what’s wrong with doing it via comint, especially since it already knows how to do the majority of things?
Comint even has the redirection facility to deal with such tasks specifically.
The answer is - there’s no message queue that spans over both comint and the target process.
For example, you can send a long-running code to the REPL that will print something after a long period of time, like this one:
(do (for [i 1 1000000] nil) (print "foo""bar""baz"))
Once the loop is completed the stdout of the process will contain a string like this one: foo\tbar\tbaz.
Comint will grab it and print it to the REPL.
All good.
But if we set up a redirect while this loop runs, that, say, queries for the completions from the REPL, we can get into a funny situation.
Imagine, user typed f and pressed the Tab key.
The current implementation of completion support in fennel-mode package will use a comint redirect mechanism, and for this case, it will send the following expression ,complete f to the REPL.
The output of the ,complete f command will be fn\tfor\tfcollect\tfaccumulate, another tab-separated line, but our REPL is busy at the moment, running the loop.
What can go wrong?
What happens is that the ,complete f is buffered by the process stdin, and not processed unless the REPL is able to read the message.
The comint redirect, however, is waiting for the output from the process and grabs what’s first to appear there.
So it grabs the output from the print function and uses it as the data for the completion engine.
While the REPL gets the output from the ,complete command:
This is a race, and the last request from comint wins the first output from the process.
It just so happens that both messages used a format similar enough for completion to work.
In most cases, completion will silently fail, and the REPL will just lose the results of the expression.
Comint is not suited for working with a protocol like ours, but we can still reuse a fair bit of comint features in our client.
Instead, we’ll have two separate processes - one for the server that implements the protocol, and one for the REPL that acts as a fake process with all comint goodies.
Here’s what our architecture will look like:
By separating input from output, and effects from parsing, we avoid all problems with comint racing over process output and misparsing different commands.
And the input doesn’t have to come from comint, we can send messages via the input sender from anywhere, so code interaction in the buffer is easy to implement too.
Notice how ID comes into play here - we read user input, format it as a message, and assign it an ID, which we send to the server.
While doing so, we register a callback for the message in a hash table, that uses message IDs as keys, and callback as values.
Once the server has processed the message, it responds with several messages of its own, all of which include a callback.
We then parse these messages in the output filter, which looks for the callback in the hash table.
If the callback was found, it is being called with the data from the message according to the protocol.
And the callback can print the result back to the REPL, to a completion engine, or to any other target buffer.
This is also the reason why our protocol answers with special messages accept and done before and after the data-oriented ones, like values, print, read, and such.
All protocol operations require a callback to reach the user, but once the message is processed fully its callback can be released.
This is what the done OP is for in our protocol, but what about accept?
Right now in my design, accept is a noop, but I can see some potential later use cases.
One of them is for implementing asynchronous timeouts.
For example, when we register a callback we can store the maximum amount of time that is meaningful for such a callback to exist.
Once the server answers with the accepted OP, we can check if the time since callback registration exceeds the lifespan of the callback.
If not, we continue waiting for other messages, otherwise, we unassign it.
Another possible use for the accept OP is to cancel previous callbacks.
This is not particularly useful in the context of a single-threaded Lua application, as it will always process messages sequentially, but nothing prevents us from implementing an asynchronous REPL.
In an asynchronous context, it is quite possible that a message will be accepted before the previous one was fully processed.
And in cases like when we query for completions, we are interested only in the latest results, so we can use accept to cancel the previous completion callback.
But enough talk, let’s write some code!
Server process
Now, I must say, that in the same way that we didn’t implement a full-blown protocol, we won’t implement a full-blown client.
However, I will try to provide a complete enough example that you’ll be able to make your own client like this in the future if you wanted to.
We start simple - we need a way to start the server process that will act as a pipe:
(defvarproto-repl--upgrade-code"...whole protocol code...")
(defvarproto-repl-process" *proto-repl*")
(defvarproto-repl--message-callbacksnil)
(defvarproto-repl--message-id 0)
(defunproto-repl--start-server (command)
"Start the Fennel REPL process.
COMMAND is used to start the Fennel REPL. Sends the upgrade code
to the REPL and waits for completion via a callback." (let ((proc (make-process:nameproto-repl-process:bufferproto-repl-process:command (split-string-shell-commandcommand)
:connection-type'pipe:filter#'proto-repl--process-filter)))
(buffer-disable-undoproto-repl-process)
(with-current-bufferproto-repl-process (setqmode-line-process '(":%s")))
(message"Waiting for Fennel REPL initialization...")
(setqproto-repl--message-callbacksnil)
(setqproto-repl--message-id 0)
(proto-repl--assign-callback#'proto-repl--start-repl)
(send-stringproc (format"%s\n"proto-repl--upgrade-code))))
First of all, let’s address the elephant in the room - the protocol code.
Our package needs to somehow obtain the protocol code and send it to the REPL process.
I’ve chosen to store it as a string, even though it is a very long string.
The reasoning behind this is that I want this file to be both self-contained and because package management in Emacs is a mess, and there’s no reliable way to obtain a file that is located in the same package unless they’re both in the same directory, which may not be the case1.
Not the most elegant solution, but it works.
Next up, we define two more variables, one for process buffer name, and another one for storing callbacks.
For simplicity’s sake, I’ll use an associative list for callbacks, but in the real client, a hash table with fast equality function should be used.
And the main piece of code - the function that starts the process.
Code should be mostly self-explanatory, but in case it’s not, the main idea is to start the process and assign a process filter to it.
The process filter will be the next thing we’ll implement, so I’ll save the explanation for later.
After that, we do a few setup steps and send the message to the process with the upgrade code.
Sending messages is another bit part of the client, and we’ll look into them as well.
Process filtering
The process filter is a function that accepts two arguments, the process being filtered and the data that was read from it.
By specifying a custom process filter we actually prevent any output from the process in the process buffer, so some kind of logging should be implemented.
And here’s our first challenge.
Our protocol works on a one message per line basis, yet the output from the process is received in hunks instead of lines.
It is quite possible to receive a message that contains the start of the next message without its end.
In a case like this, a custom encoding would be a much better choice than using a line-based protocol, but we can workaround the problem by implementing buffering.
(defvarproto-repl--message-bufnil)
(defunproto-repl--buffered-split-string (string)
"Split STRING on newlines.
If the string doesn't end with a newline character, the last (or
the only) line of the string is buffered and excluded from the
result." (let ((strings (string-linesstringt)))
(whenproto-repl--message-buf (setcarstrings (concatproto-repl--message-buf (carstrings)))
(setqproto-repl--message-bufnil))
(if (string-suffix-p"\n"string)
strings (setqproto-repl--message-buf (car (laststrings)))
(nbutlaststrings))))
First, we split the string on newlines, removing empty lines, as they’re irrelevant to us.
Next, if we have something in our buffer, we concatenate it with the first line, as it is a leftover from the previous hunk.
Then we check if the string ends with a newline, if it is we return lines as is.
If not, we set the buffer to the last incomplete line and return everything except for the last line.
This way, the protocol is much more resilient to process buffering, and the fact that Emacs doesn’t read line by line on most systems.
Yes, our protocol is line-based and it’s not exactly machine-friendly, but it is much easier than using some kind of encoding, like bencode.
Once we have a list of lines, we can begin processing them.
Here’s our process filter:
(defunproto-repl--process-filter (_message)
"Parse the MESSAGE and process it with the callback handler." (dolist (message (proto-repl--buffered-split-stringmessage))
(let ((message (substringmessage (string-match-p"(:id "message))))
(when-let ((data (condition-casenil (car (read-from-stringmessage))
(error nil))))
(when (plistpdata)
(proto-repl--handle-protocol-opdata)))))
(with-current-bufferproto-repl-process (goto-char (point-max))
(insertmessage)))
In this function, we’re going to work on a line-by-line basis because the first thing we do is split the incoming message.
However, the input massaging doesn’t stop there - for each message we want to strip everything that is not part of it.
If you remember, before we had our IO wrapped, the output from io.write was right before one of the messages.
This is partly why we do it.
In fact, this should never happen, but we can’t control it before the protocol is initialized, especially because by default Fennel REPL is quite verbose.
If we send a partially complete expression to the base Fennel REPL it will display a .. prompt, so while sending our protocol line by line we’ll get back a lot of dots preceding the initialization message:
So we have to deal with it.
This function is also the right place to add logging, but I’m omitting it for the sake of simplicity and just emitting everything to the process buffer.
After we’ve split the input, and truncated it we parse it with the read-form-string function.
This again is where some errors might lurk - if our protocol ever would return two messages in a single line, we’ll lose the last one.
So you might want to check for that.
If the message was read successfully, we also check if it is a plist with the plistp function, and pass it to the protocol handler.
The protocol handler is the meat of our client.
It manages callbacks for each message and handles all supported protocol OPs.
(defunproto-repl--handle-protocol-op (message)
"Handle protocol MESSAGE.
Message contains an id, operation to execute, and any additional
data related to the operation." (let ((id (plist-getmessage:id))
(op (plist-getmessage:op)))
(when-let ((callback (proto-repl--get-callbackid)))
(pcaseop ("accept"nil)
("done" (proto-repl--unassign-callbackid)
(unless (proto-repl--callbacks-pending?)
(proto-repl--display-prompt)))
("eval" (let ((values (plist-getmessage:values)))
(funcallcallback (format"%s\n" (string-joinvalues"\t")))))
("print" (proto-repl--print (plist-getmessage:data)))
("read" (let ((inhibit-messaget))
(write-region (read-string"stdin: ") nil (plist-getmessage:data))))
("error" (proto-repl--display-error (plist-getmessage:type)
(plist-getmessage:data)
(plist-getmessage:traceback)))
("init" (proto-repl--unassign-callback 0)
(funcallcallbacknil))))))
Even though our protocol is quite small, we do have a few OPs to support.
As I’ve mentioned the accept OP is a nop, but the done OP is quite important.
You can see, that we unassign the callback for the message, and then check if there are pending callbacks.
This allows us to avoid drawing the prompt in the REPL because pending callbacks mean that the REPL is still busy.
Another special OP is the init one.
It has a special callback with the ID 0, and this callback is strictly for the REPL initialization.
Other callbacks should be pretty much self-explanatory.
Let’s write functions for dealing with callbacks:
(defunproto-repl--get-callback (id)
"Get a callback for a message with this ID." (cdr (associdproto-repl--message-callbacks)))
(defunproto-repl--unassign-callback (id)
"Remove callback assigned to a message with this ID." (setqproto-repl--message-callbacks (assoc-delete-allidproto-repl--message-callbacks)))
(defunproto-repl--assign-callback (callback)
"Assign CALLBACK and return the id it was assigned to." (let ((idproto-repl--message-id))
(add-to-list'proto-repl--message-callbacks (considcallback))
(setqproto-repl--message-id (1+id))
id))
(defunproto-repl--callbacks-pending? ()
"Check for callbacks that still waiting for the DONE message."proto-repl--message-callbacks)
With these, we can do all operations on callbacks that our client requires.
The proto-repl--assign-callback is an interesting one.
It increments the ID after it assigned the callback, and returns the ID of the callback itself.
This is our main interface for sending messages to the REPL.
User interaction
With the server part mostly done, we can make the comint part that is an actual user interface to our protocol-based REPL.
Let’s start with the mode for the REPL:
(require'fennel-mode) ;; for font-lock(defvarproto-repl-buffer"*Fennel Proto REPL*")
(defvarproto-repl-prompt">> ")
(define-derived-modeproto-repl-modecomint-mode"Fennel Proto REPL""Major mode for Fennel REPL.
\\{proto-repl-mode-map}" (setqcomint-prompt-regexp (format"^%s"proto-repl-prompt))
(setqcomint-prompt-read-onlyt)
(setqcomint-input-sender'proto-repl--input-sender)
(setqmode-line-process '(":%s"))
(setq-localcomment-end"")
(fennel-font-lock-setup)
(set-syntax-tablefennel-mode-syntax-table)
(unless (comint-check-proc (current-buffer))
(let ((proc (start-processproto-repl-buffer (current-buffer) nil)))
(add-hook'kill-buffer-hook (lambda ()
(when-let ((proc (get-buffer-processproto-repl-process)))
(delete-processproc)))
nilt)
(insert";; Welcome to the Fennel Proto REPL\n")
(set-marker (process-markproc) (point))
(proto-repl--display-prompt))))
There’s a lot to dig in, but the main part is where we start a new process with the start-process function.
This process is what the comint will use for it’s internal implementation of input handling and such.
Let’s look at the fennel-repl--input-sender:
(defunproto-repl--input-sender (_input)
"Sender for INPUT from the REPL buffer to REPL process." (let* ((id (proto-repl--assign-callback#'proto-repl--print))
(mesg (format"{:id %s :%s %S}\n"id"eval" (substring-no-propertiesinput))))
(send-stringproto-repl-processmesg)))
This function is responsible for sending the input to the server as a message, so it formats user input like one.
Before sending the message it assigns the callback, and then the rest of our system should just work.
The proto-repl--display-prompt is a simple function that just prints the prompt to the REPL buffer:
This function is almost identical to the prompt displaying one, but it is mostly for the simplicity’s sake.
In fact the real one is a bit more complicated because we want to print stuff before the prompt if we’re sending code not from the REPL but from the buffer.
I’ve left this out as it’s not that hard to implement.
So the last thing we need is to actually start the REPL in a buffer:
(defunproto-repl--start-repl (_)
"Start the REPL." (message"Fennel Proto REPL initialized")
(with-current-buffer (get-buffer-createproto-repl-buffer)
(proto-repl-mode)
(pop-to-buffer (current-buffer))))
This one is a callback we’ve used when starting a server.
And this one is the funtion for the end user:
On the left is the REPL buffer, which is responsible for input handling, and sending messages.
On the right is the server buffer, which shows the process log.
As you can see, we’ve got the usual messages for all our expressions.
New Fennel REPL integration for Emacs
Sure, the implementation of the client and the protocol has a lot of room for improvement, but it is enough for the purpose of this post.
As a matter of fact, I’m almost done working on a proper implementation, and most of the code here is greatly simplified version of what I’ve already made for the fennel-mode package.
For now it lives in a separate module, but I have high hopes on completely replacing inbuilt comint-based client with this protocol based one, once I test it more.
It will probably still live in a separate module for a while even after I will consider it mostly complete, so users would be able to try it in their environments, but my hopes are high.
There is one problem though - this protocol implementation may not work in some contexts.
For instance, if you’re already providing a custom REPL in your application, that is based on the fennel.repl and implements its own readChunk, that, for example, doesn’t read from stdin.
One such example is the LÖVE game engine, there’s an implementation of the REPL that polls for input on a separate thread.
If we send our protocol-based REPL to such a custom REPL, we’ll negate all the efforts made to make the REPL work in a non-blocking way.
For Clojure, this works because the REPL already lives in its own thread, but there are no threads in Lua apart from what coroutines provide.
So I’m still thinking about how to handle the situation when the upgrade is impossible.
This is why the official protocol is much better than such a custom one, but alas.
You can already try this REPL if you check out the proto-repl branch in the fennel-mode project.
If you’re using straight.el, or Quelpa, or Emacs version recent enough to have the package-vc-install function, you can try installing fennel-mode and supplying the proto-repl branch to the recipe.
If you want to experiment with the code that is provided in this article for example for the purpose of building a similar protocol for another language, this very article is written in the literate style, and you can grab it here.
Run the org-babel-tangle on it, and you should get the protocol.fnl file with all the necessary functions, and proto-repl.el file with the Emacs Lisp code.
You’ll need to put the protocol code to the proto-repl--upgrade-code variable, as I’ve excluded it from the article’s text since it was making that particular code block too long.
Thank you for reading!
I hope this was interesting and useful.
As always, if you have any thoughts you want to discuss on the matter, feel free to contact me via one of the ways mentioned on the about page.
Package managers like straight.el or quelpa use recipes to specify what files to use during the build process.
It is possible that users of the package may not notice that the recipe needs to be changed.
This can be handled by the package archive like MELPA, but I’d rather not bother with it right now. ↩︎
Now that I started to devote more time to writing, I need a thesaurus more and more often. What I usually did was to go to my browser and use thesaurus.com – but as we all know, leaving Emacs is always a pain. I fired M-x package-list-packages and it turned out that there are several packages to look up synonyms from the comfort of Emacs.
I am restarting the development of MCT with the sole intent to keep it
in good working condition for the handful of users who like it. I do
not plan to expand the scope of the package. Instead, I am removing
certain features of dubious value and am cleaning up the code (which
was in a good state, all things considered).
To recap:
MCT is a layer of interactivity on top of the default Emacs
minibuffer and the Completions buffer. It treats the two as a
unified space and provides commands to intuitively move between
them.
MCT provides a facility for “live completions” which automatically
update the Completions buffer given certain conditions. Users
have access to options for a passlist and blocklist which further
control what should be live updated.
On 2022-04-14 I had announced the discontinuation of the project’s
development:
https://protesilaos.com/codelog/2022-04-14-emacs-discontinue-mct/.
The reason was that Emacs 29 was assuming certain features that MCT
had. I thought that Emacs 29 would provide an MCT-equivalent
experience and was making way for that eventuality. It turns out
that I was mistaken: MCT is still more featureful and might show the
direction for future developments on Emacs 30.
Now the other news:
I still think `` by Daniel Mendler is the best User Interface
for the minibuffer. Where relevant in my Emacs coaching sessions, I
always recommend Vertico: https://protesilaos.com/coach. It is
robust and highly configurable. What MCT does, Vertico does better.
My plan for my personal config is to have two modules, one for
Vertico and another for MCT, so that I can use the former by default
and the latter when needed. (Again, I want to maintain MCT but will
not add major new features.)
I have removed support for Emacs 27. This was a mistake from day
one. Emacs 27 cannot show the Completions buffer in one-column
view and thus lacks the primary user-facing aspect of MCT.
I have removed everything that has to do with in-buffer completion
(completion-in-region). While it is nice to have a uniform
interface for completions in the minibuffer and inside buffers, the
latter was never good enough. This is not an MCT problem, but how
inherently limited the Completions are in the scenario where the
minibuffer is not activated. I will continue using corfu by
Daniel Mendler.
For a brief period of time I experimented with various extensions to
the core MCT functionality (e.g. integrating with avy.el. These
are all removed. There now is a single ‘mct.el’ file. It works
fine for what it was originally designed to do: enhance the
minibuffer UI.
The new Emacs 29 support for tree sitter awesome. Since I'm an evil user, I made a small
package to take advantage of the new facilities. If you are interested, the package is on
GitHub here. From the README:
This package provides some basic additional actions for evil-mode using the new built-in
tree sitter library. The package only works with Emacs 29 or greater. To activate, just
run M-x evil-ts-mode. This was so easy to do, that it hardly merits a package. But perhaps
it will be useful to someone.
In visual mode, you can select a if/try/etc statement with s. So when you are inside an if
statement, the sequence vas will select it. Similarly, f selects a function and c selects
a class. On these cases, there is no difference between inner and outer text objects.
In normal state, you can move to the beginning or the end of a class with [c and ]c.
Similarly, [f and ]f moves you to the start or end of a function. And [w moves you to the
start of a sentence, and ]w to the end (these last two bindings are not great, but ]s is
usually taken for navigating spelling errors). I created this package for my own personal
use, so the default bindings may not be what you want. Of course, you can change that. The
mode map is evil-ts-mode-map.
When I wrote about formatting Clojure buffers1 I also mentioned that Clojure
LSP2 can take care of this among other things. Since my working days are
pretty much Clojure days, it’s time to check out how LSP can help me in this
regard and whether the experience is worth an extra tool.
Installing Clojure LSP is trivial so I will leave you on your own with that.
After the installation the first thing I tried is the obvious M-x
eglot in a Clojure buffer, but all I got was a connection timeout error as
a reply. Weird, I thought, this has never happened with LaTeX. However, the
solution was simple: I just needed to increase the default value of
eglot-connect-timeout to 60.
Emacs can leverage many goodies from Clojure LSP thanks to Eglot, but some of
them out of the box interfere with CIDER. For instance, the integration with
ElDoc is better left in the capable hands of Eglot in order to avoid duplicates
when displaying documentation.
(setq-defaultcider-eldoc-display-for-symbol-at-pointnil)(defunmu-cider-disable-eldoc()"Let LSP handle ElDoc instead of CIDER."(remove-hook'eldoc-documentation-functions#'cider-eldoct))(add-hook'cider-mode-hook#'mu-cider-disable-eldoc)
In a similar fashion I put Clojure LSP in charge of completions. This approach
is useful when in a Clojure buffer all I need is some poking around to
understand the code in front of me. There is no need to jack-in with CIDER to do
just that.
(defunmu-cider-disable-completion()"Let LSP handle completion instead of CIDER."(remove-hook'completion-at-point-functions#'cider-complete-at-pointt))(add-hook'cider-mode-hook#'mu-cider-disable-completion)
I am sure CIDER can, and probably will, embrace Clojure LSP some day, but one
cannot blame its developers for these nuisances. On the contrary, we should be
grateful for the ease of customization at our disposal, which is one more reason
to appreciate CIDER, if you ask me.
Anyway, now I can get rid of mu-cljfmt-format-buffer and rely on M-x
eglot-format-buffer for my Clojure formatting needs. A happy ending
indeed, because removing code from my init.el is always a pleasure.
As many of you know, I’m fascinated by those who use Emacs for projects that don’t involve science or engineering. One such aspect that I always enjoy reading about is using Emacs for creative writing, by which I mean novels, short stories, poetry, and the like.
That may seem like an edge case but there are plenty of writers who write their novels with Emacs, including such A-list writers as Vernor Vinge and Neal Stephenson. There’s little reason to wonder why. Emacs has superb editing and linking capabilities that makes writing and organizing a piece of prose easier than it would be with most other writing platforms.
John Urquhart Ferguson is another writer who uses Emacs and he’s written a package, Org Novelist, to make the note taking and story organization easier. A full-blown novel can have several minor characters and locales that are hard for the author to keep track of. Similarly, ideas may occur to the author that don’t involve whatever part of the story is currently being worked on and a way of capturing those notes is needed. Ferguson’s package is intended to help with that. Best of all, the package let’s you write your prose in Org mode.
Ferguson has a short announcement on reddit but to really get a feeling for what the package is about and what it can do, take a look at the README on his GitHub repository. The package is not on MELPA or any of the other package repositories so you’ll have to download it from GitHub.
Ferguson stresses that the package was written for his own use with his particular workflow but he’s releasing it in case others find it useful. From the README, that seems likely to me so if you’re interested in a package to help you with your creative writing, take a look.
Moldable Emacs: what is the public API of this Elisp buffer?
Lately I have been reading the Elisp source code that ships with
Emacs 29. The addition (and the general excitement) of treesit.el
made me willing to come up with a plan to support that in my
moldable-emacs.
I like that Emacs contributors keep a stoic discipline in distinguish
public and private functions in a library. The convention is that
my-function belongs to the public API while my--function is not.
As a user often I care only about the public interface. So I thought:
wouldn't be nice to have a view/mold that shows me only what I need?
You can see in the below video how the resulting mold helped me
exploring cl-lib.
This filters out some of the complexity of Elisp libraries I don't
know.
I hope to generalize this little by little for other languages so I
can get the essence of the files I need to read.
And I wonder if this could help me evaluate the quality of an API as
well. It may be of help if Emacs warns me while I write my library and
tells me that the API is getting bloated, no?
Anyway, this has been good for my Elisp exploration.
This is the twentyfifth anniversary of Curl, one of those quiet utilities that everyone uses but that never makes a big splash. Along with wget, Curl is one of the two premier standalone applications for downloading content from the Web.
Its 25th anniversary is reason enough to give Curl and its developer, Daniel Stenberg, a shout out but there’s an even better reason: hardware setup porn. The Hackers Stations site has a page on Stenberg’s home office and the kit he uses. He’s a full time open source developer who works from home so it’s obviously important that he has a comfortable and efficient setup. Another reason Stenberg deserves an Irreal shout out is that he’s an Emacs user.
There’s no real point here, of course, but if you’re like me, you take voyeuristic pleasure in seeing other people’s setups. Working on a cross platform application, as he does, Stenberg has machines running Linux, Windows, and macOS so his setup is pretty complete. The page even describes all the knickknacks on his desk.
Definitely worth a couple minutes of your time if you enjoy seeing how other developers work.
I would like to propose a collaborative project to highlight and promote Emacs packages on GNU ELPA and NonGNU ELPA. This will most likely take the form a webzine, but that is not fixed.
The idea would be to regularly highlight new packages that have been added to these archives and touch on packages that have been updated. Packaging-related news from readers or maintainers could also be find a place. If possible, it would be nice to
What the precise goals and parameters should be hasn’t been defined, but I think it would make sense to differentiate the initiative from other projects, for example like Sacha Chua’s Emacs News by going into greater detail on the new packages or This Month in Org by not focusing too much on Org-related developments.
As all of this is rather vague, and the idea is to have a collaborative projects I would invite anyone interested to participate on a little mailing list I created to organise the zine.
There’s a JUXT Safari video featuring Ellis Kenyo on the state of Doom Emacs. It’s not really an introduction to Doom but, rather, a status report on where Doom is now and what new things are being worked on.
I’m a complete innocent as far as Doom is concerned. I’ve never tried it and the main thing I know about it is that it has a large and dedicated userbase. Even so, the video was informative and gave a reasonable summary of what the project is about and how things works. One sort of un-Emacsy thing about the package is that certain administrative functions are performed outside Emacs. That, of course, is an anathema to us hardcore “everything in Emacs” fanatics but, again, I have no experience with it so I don’t know what it’s really like.
As a side issue, the Minions are insisting that I point out Kenyo’s Emacs theme is a prime example of everything that’s wrong with dark themes. It features the horrible and unreadable dark gray and dark blue on black that I described in a recent Red Meat Friday post. It’s too bad because, at least for my aging eyes, it makes it impossible to read the examples and configuration that Kenyo offered.
Regardless, the video is interesting and useful. It runs for 31 minutes, 25 seconds so you’ll need to set aside some time.
A few days ago I posted on r/haskell that I'm attempting to put together a Cabal
grammar for tree-sitter. Some things are still missing, but it covers enough to
start doing what I initially intended: experiment with writing an alternative
Emacs major mode for Cabal.
The documentation for the tree-sitter integration is very nice, and several of
the major modes already have tree-sitter variants, called X-ts-mode where X
is e.g. python, so putting together the beginning of a major mode wasn't too
much work.
Configuring Emacs
First off I had to make sure the parser for Cabal was installed. The snippet for
that looks like this1
The built-in elisp documentation actually has a section on writing a major mode
with tree-sitter, so it was easy to get started. Setting up the font-locking
took a bit of trial-and-error, but once I had comments looking the way I wanted
it was easy to add to the setup. Oh, and yes, there's a section on font-locking
with tree-sitter in the documentation too. At the moment it looks like this
One of the reasons I want to experiment with tree-sitter is to use it for code
navigation. My first attempt is to translate haskell-cabal-section-beginning
(in haskell-mode, the source) to using tree-sitter. First a convenience
function to recognise if a node is a section or not
(defuncabal--node-is-section-p(n)"Predicate to check if treesit node N is a Cabal section."(member (treesit-node-type n)
'("benchmark""common""executable""flag""library""test_suite")))
That makes it possible to use treesit-parent-until to traverse the nodes until
hitting a section node
(defuncabal-goto-beginning-of-section()"Go to the beginning of the current section."(interactive)(when-let*((node-at-point (treesit-node-at (point)))(section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p))(start-pos (treesit-node-start section-node)))(goto-char start-pos)))
And the companion function, to go to the end of a section is very similar
(defuncabal-goto-end-of-section()"Go to the end of the current section."(interactive)(when-let*((node-at-point (treesit-node-at (point)))(section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p))(end-pos (treesit-node-end section-node)))(goto-char end-pos)))
Emacs 29 Build Command
TBH, I've been building emacs "manually" for months.
I have scripts for this, of course. Naturally, they are broken. It turns out not to be all that hard to get me to bail out from fixing (even) my own scripts, and revert to building Emacs by hand. Here's what I ran this evening, to make a new snapshot from the emacs-29 branch:
(export BIF=/d/emacs-build/install \ SLUG=29-$(git rev-parse --short HEAD); \ (.
Intro prelim: this is a following my css-in-cljs method from my garden css has ascended, which was exceptionally good for the first draft/version of the project but eventually should cede to normal CSS once you are ready for hardening the product. The goal here is to remove all the per-html CSS rules that were so perfect for Dev with a single CSS file that can be called from all the html files in the project and is both far more efficient file-size wise, and the Right Way™ to do it for caching and synchronization.
I had been meaning to give ChatGPT a good try, preferably from Emacs. As an eshell fan, ChatGPT seemed like the perfect fit for a shell interface of sorts. With that in mind, I set out to wire ChatGPT with Emacs's general command interpreter ( comint).
I had no previous experience building anything comint-related, so I figured I could just take a peek at an existing comint-derived mode to achieve a similar purpose. inferior-emacs-lisp-mode ( ielm) seemed to fit the bill just fine, so I borrowed quite a bit to assemble a basic shell experience.
From then on, it was mostly about sending each request over to the ChatGPT API to get a response. For now, I'm relying on curl to make each request. The invocation is fairly straightforward:
There are two bits of information needed in each request. The API key, which you must get from OpenAI, and the prompt text itself (i.e. whatever you want ChatGPT to help you with). The results are not too shabby.
I've uploaded the code to GitHub as a tiny chatgpt-shell package. It's a little experimental and rough still, but hey, it does the job for now. Head over to github to take a look. The latest iteration handles multiline prompts (use C-j for newlines) and basic code highlighting.
Let's see where it all goes. Pull requests for improvements totally welcome ;-)
There are several packages which have a switch-to-buffer-like command that
also suggests recent files in addition to existing buffers. The oldest one it
probably the built-in ido.el with its concept of "virtual buffers" you can
enable with (setq ido-use-virtual-buffers t). The
consult package's consult-buffer commands
even go a step further and provide even more sources for "virtual buffers" next
to recent files, e.g., bookmarks, registers, project files, or home-brewn
sources.
I've used ido.el back in the days and over multiple intermediate steps
settled on the typical and awesome completion combo of
vertico,
corfu, and
marginalia. I've also tried
consult for some time but its previews,
i.e., that it immediately shows the completion candidate in a buffer, is a bit
too dynamic for my taste. Same for its search capabilities like
consult-ripgrep where I prefer a more static UI like that of
rg. That said, what I've immediately missed
after removing the consult package is the support for recent files in the
(now again stock) switch-to-buffer command. I'm so used to switching to
virtual buffers; I never visit files in projects I'm working using C-x C-f
because C-x b always worked, too. I never had to ask myself "did I already
find that file in this emacs session?" and then choose the right command, i.e.,
find-file or switch-to-buffer.
However, reading the docs showed that since ages (Emacs 20.3) one can define a
custom read-buffer-function which is then used by read-buffer, the internal
workhorse of the commands switch-to-buffer / pop-to-buffer. So the
question is how to sneak in recent files in addition to actual live buffers.
The answer for me was to use a completion table which tries to do buffer
completion and if that runs out of candidates, switch to a completion table for
recent files. Such a "try this and if it won't work, try that" completion
table can be composed from existing tables using completion-table-in-turn.
That works great for me but won't do the trick if you have a buffer foo.txt
and also a different recent file foo.txt. In such a case, when you rather
want a completion table composed of other tables which should have the same
priority, you can use completion-table-merge instead of
completion-table-in-turn.
Long story short, here's the code which I'm using now:
(defconst th/read-buffer-or-recentf-command-alist
'((kill-buffer buffers)
(switch-to-buffer buffers-except recentf)
(pop-to-buffer buffers-except recentf))
"Alist with entries of the form (CMD . COMPLETES).
COMPLETES is a list defining what's completed where entries can
be:
- `buffers': completion for all buffers
- `buffers-except': completion for all buffers except the current one
- `recentf': completion for recent files which will be found on demand")
(defun th/read-buffer-or-recentf (prompt &optional
def require-match predicate)
(let* ((tables (or
(mapcar
(lambda (syms)
(pcase syms
('buffers #'internal-complete-buffer)
('buffers-except (internal-complete-buffer-except
(current-buffer)))
('recentf (completion-table-dynamic
(lambda (s) recentf-list)))
(unknown (error "Unknown case %S" unknown))))
(cdr (assoc this-command
th/read-buffer-or-recentf-command-alist)))
(list #'internal-complete-buffer
(completion-table-dynamic
(lambda (s) recentf-list)))))
(completion-table (apply #'completion-table-in-turn tables)))
;; `read-buffer-to-switch' (called by `switch-to-buffer') already sets
;; `internal-complete-buffer' as `minibuffer-completion-table' using
;; `minibuffer-with-setup-hook' before `read-buffer-function' is invoked by
;; `read-buffer', so we'd be restricted to buffers by default. Therefore,
;; append a function setting our completion table.
(minibuffer-with-setup-hook
(:append (lambda ()
(setq-local minibuffer-completion-table completion-table)))
(when-let ((result (completing-read prompt completion-table
predicate require-match nil
'buffer-name-history def)))
(cond
((get-buffer result) result)
((file-exists-p result) (buffer-name (find-file-noselect result)))
(t result))))))
(setq read-buffer-function #'th/read-buffer-or-recentf)
This custom read-buffer-function just arranges that a suitable completion
table is used, then does the completion using completing-read and if the
result is no buffer but an existing file, it finds it and returns the
corresponding new buffer.
The defconstth/read-buffer-or-recentf-command-alist allows for defining
what's suggested by completion on a command by command basis. For example,
completion for recent files makes totally no sense for kill-buffer while
completing the current buffer is not too sensible for switch-to-buffer and
pop-to-buffer. If there's no entry in the alist, the default behavior is to
offer completion for all buffers and then for recent files.
It's important to note that in contrast to virtual buffers in ido.el or
consult, this read-buffer-function is in effect whenever emacs queries for
a buffer, not just in certain buffer switching commands. As an example, when
doing M-x append-to-buffer RET now I can also complete a recent file instead
of an existing buffer which will be opened on demand. That's quite cool. A
caveat is, however, that the position where the current region will be appended
depends on where point has been when I visited that file the last time because
I use save-place-mode. If I wouldn't use it, the region would be prepended
to the beginning of the file/buffer which would also be awkward. But I guess
the actual inconsistency is due to the fact that append-to-buffer inserts at
the location of point which doesn't quite fit to the "append" in its name.
Anyway, after a few days of using it, I must say, it works extremely well for
me. One thing which doesn't work completely is marginalia annotations.
During completion, buffers are annotated as usual, but when recent file
completions kick in, those are not. I've tried making them respond to
metadata requests which didn't work out because right now, emacs' completion
framework doesn't really support combining completion tables which have
different metadata, e.g., one says the category is buffer, the other says
it's file. Well, that's probably a sensible assumption in all cases but this
one. :-)
Emacs isn’t my only obsession, I like to create digital art and for that I use ArtRage. The interesting thing about ArtRage is that each brush stroke or editing action can be recorded in a text file or script for later playback. I use this facility for creating time-lapses.
The ArtRage manual describes the following:
Script Files
ArtRage scripts are simple Unicode UTF16 text files which can be edited using Notepad, Notepad+++, TextEdit, or any similar program
“similar program”?! now let me think….. what do I have available :)
For a nice clean time-lapse ArtRage now has the ability to filter out the rest of the UI elements and leave the canvas fixed in the center of the screen. For the most part this works very smoothly however sometimes the recording hasn’t quite registered some strokes properly or some reference files have moved, in this case I may get an error such as:
At which point it is time to break out emacs and delve into the recorded text script file. For a typical piece of art the script can get quite large, for my last portrait it was 174M with 605012 lines. How does emacs handle this?, well it had no problem and opened it in a split second!
I think to resolve this current issue around ReferenceImageXForm I am just going to remove all occurrences as for a time-lapse playback I don’t usually want to show the references anyway.
First of all lets isearch for ReferenceImageXForm, now that I have isearch-lazy-count turned on it gives me 1/36 in the minibuffer. Lets just step through each one and remove the whole line.
Also I think I would like to remove the initial reference image load which starts off as:
and then has many lines of image data and then finishes with an enclosing brace. Of course this is easy to remove in emacs by leveraging:
(kill-sexp &optional ARG INTERACTIVE)
Kill the sexp (balanced expression) following point.
now a quick save and playback and the timelapse runs perfectly.
I remember a while ago when I was not quite aware of the capabilities of emacs and I was more Windows bound I had to look for specific text editors that could handle large files and then save them in the requisite Unicode UTF16 format. They were slow, it was a hassle, I couldn’t define any macros e.t.c, but emacs just does this out of the box!
Maybe I should petition the ArtRage team to reference emacs in their documentation!
As I alluded many times in the past, I use Org mode’s clocking feature almost all the time. However, there are times I don’t. One of such cases is a rare situation when I don’t have access to my computer. Since I started commuting using public transport, I do some reading in a streetcar or bus. When I get to my laptop again, I want to update my clocking data. How to do that?
As many of you know, I spent many years as a Vi/Vim user. A bit more than 15 years ago, I moved to Emacs, mostly I think, because I was moving a lot of my coding to various Lisps and loved the idea of an editor that was programmed and extensible in Lisp. As I’ve written before, when I moved to Emacs I went all in learning and internalizing the native Emacs key sequences rather than fall back to Vim by using Evil. I’ve never regretted that decision but one thing I do miss is the easy Vim command repeat bound to the . key.
Emacs has its own repeat commands but the situation is complicated by Emacs’ complex commands. One nice feature in Emacs, though, is that some commands with a long prefix can be repeated without retyping the prefix on subsequent invocations. Being Emacs, it is, of course, possible to set up your own repeatable bindings but this has to be done in advance.
Karthik Chikmagalur has an interesting post that shows how to build and invoke a repeatable key sequence on the fly. He does this by cleverly leveraging the Ctrl+h helper function that is called when you type Ctrl+h after inputting part of a binding. The only downside—if it is one—is that you have to type Ctrl+g to get out of the repeat mode.
If you’re like many (most?) Emacsers and are striving to be as efficient as possible, this may be something you’ll like. It requires only eleven lines of code so it’s easy to add to your init.el.
Effects of speech-enabling set-minibuffer-message - I like how T.V Raman's discovering more about Emacs (even old features) by adding speech to more things. =) If you use Emacspeak and would like prefix keys echoed faster, set echo-keystrokes to a small value
Update:Jami
has won this year's Award for Project of Social Benefit, presented
by the Free Software Foundation "to a project or team responsible for
applying free software, or the ideas of the free software movement, to
intentionally and significantly benefit society. This award stresses
the use of free software in service to humanity."
Jami is free/libre software for universal communication that
respects the freedoms and privacy of its users. An official GNU
package, Jami is an end-to-end encrypted secure and distributed
communication tool for calling, conferencing, messaging, and file
transfer. Jami has end-user applications across multiple operating
systems and platforms, as well as multiple APIs and a plugin system
for building upon and extending Jami as a framework for secure and
private communication.
This talk gives an update on what's new in and about Jami since
bandali's "Jami and how it empowers users" talk at LibrePlanet
2021.
I will add the presentation video once the conference recordings
have been processed and published by the Free Software Foundation.
LibrePlanet is a conference about software freedom,
happening on March 19-20, 2023. The event is hosted by the Free
Software Foundation, and brings together software developers, law and
policy experts, activists, students, and computer users to learn
skills, celebrate free software accomplishments, and face upcoming
challenges. Newcomers are always welcome, and LibrePlanet 2023 will
feature programming for all ages and experience
levels.
First a bried introduction for the uninitiated and then the
announcement of the new feature.
Denote is a simple note-taking tool for Emacs. It is based on the idea
that notes should follow a predictable and descriptive file-naming
scheme. The file name must offer a clear indication of what the note is
about, without reference to any other metadata. Denote basically
streamlines the creation of such files while providing facilities to
link between them.
Denote’s file-naming scheme is not limited to “notes”. It can be used
for all types of file, including those that are not editable in Emacs,
such as videos. Naming files in a constistent way makes their
filtering and retrieval considerably easier. Denote provides relevant
facilities to rename files, regardless of file type.
Backronyms: Denote Everything Neatly; Omit The Excesses. Don’t Ever
Note Only The Epiphenomenal.
I just pushed the following commit and want to share with you the
commit message further below:
commit 62c6853480d62b00e4c838e148730d1fedf6235f
Merge: 94a3ffd d7dc32c
Author: Protesilaos Stavrou <info@protesilaos.com>
Date: Mon Mar 20 11:23:24 2023 +0200
MAJOR UPDATE towards version 2.0.0: Merge branch 'signature'
[...]
README.org | 148 ++++++++++++++++++++++++++++++++++++++++++++++-------------
denote.el | 152 +++++++++++++++++++++++++++++++++++++++++++++----------------
2 files changed, 231 insertions(+), 69 deletions(-)
Signatures are an optional extension to Denote’s file-naming scheme.
They are arbitrary strings of alphanumerical characters that can be
used to establish sequential relations between files at the level of
their file name (e.g. 1, 1a, 1b, 1b1, 1b2, …).
Files that have the signature follow this scheme (though read the
documentation of ‘denote-prompts’ for possible permutations):
DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSION
As a reminder, Denote’s default file-naming scheme is:
DATE--TITLE__KEYWORDS.EXTENSION
[ Denote can be used to rename any file, not just to create notes. As
such, the file-naming scheme is a powerful, yet low-tech invention
to facilitate searching and filtering. ]
For the time being, signatures are not added to a file’s front matter
and are not shown anywhere else beside the file name. This is done on
purpose to simplify the implementation and make sure we define clear
use-cases for this feature (it is easier to add new functionality than
refactor/break existing one).
Users can create files with signatures either by (i) modifying the
‘denote-prompts’ user option to affect the standard ‘denote’ command,
or (ii) by generating such files on demand with the command
‘denote-signature’ (alias ‘denote-create-note-using-signature’).
Signatures are treated as quasi-identifiers when renaming files that
include them. This means that they are not touched by Denote. The
user must manually update the signature which, in theory, should not
be done if notes already have a predefined sequence.
Signatures are backward-compatible, meaning that existing users are
not impacted by their inclusion.
The signature extension was discussed at length on the GitHub mirror
in issue 115: https://github.com/protesilaos/denote/issues/115.
Thanks to Stefan Thesing, Mirko Hernandez, Noboru Ota (nobiot),
Xiaoxing Hu, nbehrnd, Elias Storms, and 101scholar for helping me
reason about this feature, understand its scope, and prototype its
implementation.
The inclusion of the ‘signature’ branch into ‘main’ does not mean that
we are done with the development of this feature. We are simply
making it available to more users while preparing for the release of
version 2.0.0 of Denote.
Please note that planet.emacslife.com aggregates blogs, and blog authors might mention or link to nonfree things. To add a feed to this page, please e-mail the RSS or ATOM feed URL to sacha@sachachua.com . Thank you!
