A few days back, I hacked up a little minor mode for Emacs to use an LLM for spell checking. (Before you get a heart attack — I did not say “editing”, “getting ideas” or “research”: Just spell checking. Using an LLM for actual writing isn’t just a bad idea — it leads to abhorrent results.)
I’ve been running a random selection of blog posts I wrote for the Comico read through blog to see whether the thing works well — and… yes! But also no.
OK, positives first:
1) For simple typos, it’s very good indeed — it finds them all.
c) For word substitution errors, like “to” instead of “too”, it’s also very good.
II) For grammar errors, it’s more mixed. It’s good at, for instance, basic noun/verb agreement, so if I’ve written “they writes”, it fixes that. But if the noun clause is long and complex, it doesn’t always react. I’ve experimented with adding/removing an “s” from the verb in sentences like that, and it sometimes lets both versions through without comment.
⁵) Since I’m usually giving the LLM the complete blog post to proof-read, it knows what area I’m writing about, and so it’s able to correct me when I misspell names, too. I hadn’t even considered that, but that was a very nice surprise. So when I’m writing about comics and casually mention “the art looks a bit like Kriegstein”, it knows to correct that to “Krigstein”. This is something that traditional spell check can’t do at all. (And amusingly enough, running this article through the LLM doesn’t tag that “they writes” up there as an error, because it gets that that was an example of an error.)
So that’s all very good. So what’s the bad bits? Well, it’s an LLM, so it’s slow. That’s not a major problem, though, since I’m just using it to give a blog post a look-over to weed out the worst spelling mistakes. But I’ve added some code to strip out (longer) HTML fragments before sending the text over to the LLM, and this helps a bit.
The biggest problem is that an LLM is an LLM and will randomly decide to not follow the instructions in the prompt. Sometimes it adds random elements to the text here and there. Sometimes it says that I should fix a word from “identical” to “identical”. Sometimes it removes a paragraph. This doesn’t happen often — perhaps in 5% of the cases? But…
LLMs are unreliable! It cannot be! *gasp*
(Somebody speculated that this unreliability is a feature for the LLM companies — because you know that you may get bad results, you get extra happy the times you get a good result. The slot machine thing? I think that’s probably the case — some people get really addicted to their LLMs…)
Fortunately, these things can be programmed around — I’ve now added code to do sanity checks to the output from the LLM before using it. So if the LLM decides to go off piste, the package detects that, and then you can just do the query again.
It used to be that doing the same thing and expecting a different result was a sign of insanity, but with an LLM, that’s fine.
It’s a brave new world out there.
It’s also … odd … to type commands in Emacs and then they cost money.
Anyway, I’ve pushed these fixes to Microsoft Github, so there you go.
The consult Emacs package provides asynchronous search commands such as consult-fd, consult-find, consult-grep, consult-git-grep, and consult-ripgrep. For these commands, Consult does not necessarily start a new search for every change to the minibuffer input. Instead, it uses configurable debounce and throttle delays to control when asynchronous processes start, while a separate refresh delay controls how frequently asynchronous results are pushed to the completion UI.
The first time I tried Consult, I thought to myself: this feels slow compared to Counsel (I had a similar feeling about Emacs settings such as show-paren-delay). With Counsel, results updated immediately on every keystroke, while Consult seemed to hesitate for a fraction of a second before updating the list.
As it turns out, this behavior is entirely intentional. The current Consult defaults are conservative by design.
Aggressive asynchronous search
If you prefer lower latency and speed, these variables can be made more aggressive:
These values reduce the amount of time Consult waits before starting another asynchronous search and allow the completion UI to refresh much more frequently.
These values do not make the underlying search programs execute faster. External tools like ripgrep already use highly optimized, multi-threaded algorithms to scan files across multiple CPU cores. Instead, these variables speed up the feedback loop inside Emacs itself. They reduce the waiting period before Consult spawns the external process, and causing the Emacs UI to pull in new asynchronous data and redraw the screen at a much faster rate.
Variable: consult-async-input-debounce
(setq consult-async-input-debounce 0.05)
This setting forces Consult to wait 0.05 seconds after your last keystroke before launching an asynchronous process. Because this acts as a debounce delay rather than a fixed polling interval, every new input resets the timer. The search only executes once the full quiet period elapses.
This prevents Emacs from spinning up redundant background tasks for every intermediate character you type, which minimizes CPU overhead and keeps the UI responsive.
Variable: consult-async-input-throttle
(setq consult-async-input-throttle 0.1)
The consult-async-input-throttle variable acts as a hard rate limit for how often Consult starts asynchronous processes. With this value, a new process starts at most once every 0.1 seconds, regardless of how fast you type.
To break down the difference between the two concepts:
Debouncing is an idle timer. It waits for you to stop typing. If you keep hitting keys, the timer keeps resetting, and the search will not run until you stop typing.
Throttling is a strict speed limit. It enforces a maximum execution rate. Even if you pause just long enough between words to constantly trigger the debounce timer, the throttle ensures the system will not exceed the permitted rate.
Variable: consult-async-refresh-delay
(setq consult-async-refresh-delay 0.05)
The consult-async-refresh-delay variable controls how frequently Consult updates the completion UI with results from asynchronous commands. This value makes Consult refresh at intervals as short as 0.05 seconds when new results require an update. It does not cause Emacs to redraw continuously when there is nothing new to display.
A low refresh delay can make search results appear faster, but redisplay itself has a cost. Emacs redisplay is expensive enough that highly frequent updates can increase CPU and garbage-collection activity.
Choosing values
If you prefer asynchronous searches to feel as responsive as possible, these values offer a good balance between low input latency and resource usage. Consult responds quickly to minibuffer changes and updates the completion UI with minimal delay.
These settings are well suited to fast computers plugged into a power source. On a laptop, the aggressive process creation and continuous UI updates will keep the CPU active and drain your battery much faster than the conservative defaults. On slower computers or when searching very large projects, the default values may provide a better balance by avoiding unnecessary searches while input is still changing.
The optimal values depend on typing speed, project size, search-tool performance, power constraints, and the cost of processing and redisplaying asynchronous results in Emacs. Lowering the delays is therefore best viewed as an explicit trade-off: less input latency in exchange for more frequent asynchronous work.
… I’ve lost track. I’ve been using Jinx for my spelling correction for a while now and really like it. When I first started, my main problem was that the selection procedure described in the documentation didn’t work for me. I was almost certain that was because of an interaction with Ivy; my commenters agreed and offered some things to try.
After some experimentation, I still wasn’t able to get that aspect of Jinx working. I decided to wait for Emacs 31 to revisit the issue. Emacs 31 is here but I still haven’t addressed the problem. Most of the reason for that is that I’ve developed some efficient muscle memory for dealing with the problem. I have jinx-correct bound to Ctrl+; and when I call jinx-correct, I continue holding down Ctrl and can easily move down in the list of alternatives by pressing n. Usually the first choice is the one I want so it’s only a single press. Then I type Return to update the text.
That seems like a little more work than simply pressing a number but I don’t need to move my hands or reach to do it so it seems easy to me. I’ll probably do a bit more research and get the default method working when I get a chance but in the mean time, I’m pretty happy with how it’s working for me now.
I’m really enjoying how easy it is to put an “unknown spelling” in a variety of places—the current file, the current directory, or in a personal global dictionary—and am happy I switched. If you haven’t tried it yet, you should give it a spin.
The other week I made a calendrical display for the eink thing I have. The problem is that I don’t really use a calendar — I guess the main reason is because I hate planning, but the other reason is because I really dislike all the calendar entry interfaces I’ve used.
I haven’t used all that many, though, so I may well be missing some ingenious calendar out there. But what I’ve been resentfully using is the standard Google calendar on Android.
I filmed one of my interactions with this thing, and I swear — I didn’t try to make it more awkward than it is for me:
I’d be climbing up the walls if I were to try to use it more than I do now — I just put the absolute bare necessities in there, like dentist appointments and Boris shows. (One coming up in November!)
But now I have this calendar in the hall… wouldn’t it be nice to have it display more stuff? Stuff that I really should be remembering, anyway?
There’s also org-gcal, if you’re in the Org ecosystem… but… I’m not.
Does a command line interface to Google Calendar work? Because I want to have the calendar in my pocket, too. And… Yes, there’s gcalcli. (Which I noticed hasn’t been updated for two years and is probably going to be deprecated, but it works for now, so I haven’t bothered to redo my code.)
After going through the byzantine process on Google Cloud Console to create an “application”:
Yes! You have to create a logo! Behold my magnificent logo!
Anyway, after all that… tada:
Yes! The system works!
It does the normal Do What I Mean Thing — “fri 9” means “next Friday at 9am”, and “aug 11 1930” means “half past seven on August 11”, etc:
I’m freee! I never have to enter a calendar entry on the mobile phone ever again!
Also, public service announcement: doomemacs.com is not affiliated with doomemacs.org (see this Github discussion). Just in case someone reading this happens to be the person who made doomemacs.com, please consider setting up a redirect to doomemacs.org for more awesomeness and less confusion. Thank you!
A few years ago I wrote about a setup I coded for myself for when I want to make paying my bills a tiny bit less painful. One thing I mentioned then was the primary selection. For those of you who don’t know it: under X.org, you can select a portion of text (for example by dragging the mouse over it) and then paste it in the same or another application with clicking the middle button. It is extremely useful for quick copy-and-paste operations where you do not have to issue a separate “copy” command (usually C-c, but often also found on mouse context menu). And by the way, Wayland also supports this. Sometimes I find it useful to use it even without the mouse.
Like many nerds, I feel so embarrassed whenever I see that I’ve misspelled something. But some years back when I started producing text in huge amounts for things like the Eclipse Comics read-through I realised that my options were: 1) Remain neurotic about this stuff and produce one blog post per week, or 2) Start caring less and produce a lot of blog posts.
I think I made the correct decision — I just don’t have time to proff-read all this stuff, so typos will abound. It’s just not feasible to put as much work into that stuff as you’d do for a properly published article. The reader gets what they pay for.
But now there’s a way to have your delicious cake and eat it too — just ask an LLM for help.
NOOOO! *audience runs in slow motion towards camera*
I know, I know. Whenever I read somebody in deep AI psychosis who thinks they’re being AI sceptical saying “well, of course you shouldn’t let the LLM write your articles. Instead I let it do the research, and then I make it provide an outline, and then I write some text based on that, and then it does formats and restates what I wrote. IT”S A WHOLE DIFFERENT THING! I WROTE IT!!!!”
No. You should never let an LLM do those things: Not the research, not the outline, not the formatting, not anything. Reading the output of LLMs is abhorrent to people. You should never subject anybody to an LLM generated text, even if you’ve done “quality control”.
LLM-generated text gives people the ick. It’s better to have an awkwardly phrased article you’ve written yourself than subjecting somebody to a Claude text. It’s not just the ick — it’s the horror.
However, you can use LLMs for spell- and grammer check:
I’ve whipped up a li’l minor mode for Emacs (and put it on Microsoft Github). The default prompt is very clear that I want the LLM to do nothing but copy edits — no structural changes, and no other helpful suggestions. Just fix grammer and tippos.
And then you can interactively go through the fixes and toggle back to the original ones if you disagree.
Yes, yes, sure — Grammarly, etc — stuff like this already exists. But I just don’t like using them. I use the built-in Emacs spellcheck, but it doesn’t catch everything. Having an LLM give the text a quick once-over for obvious mistakes seems nice.
While testing this minor mode now, I see that LLMs haven’t changed as much as I thought — they’ve gotten so much better on programming the last half year, it’s incredible. For stuff like this, though, it’s still pretty unreliable. Any two runs won’t necessarily fix up the same thing, or the same thing the same way. And getting it (I’m using Claude Sonnet 5) to output it in the requested form is not reliable at all — I’ve been reformulating the prompt, trying to get it to output it in the format I the mode needs, and… I think I’m at 95% now?
So, like all with all non-programming tasks, LLMs are still pretty much toys. But useful ones! For instance, this blog post is totally free of errors.
I while ago I added a minor mode, haskell-ng-project-mode, to my Haskell mode.
The idea I had was that I could use it to add keybindings for running various
tools (via compile), e.g. I bound cabal build and cabal test, p b and
, p t respectively. Then I hooked in the minor mode in my haskell-ng-mode.
Soon I realised it'd be nice to have those shortcuts available in dired too,
so I hooked it in there as well. When I later wanted to re-run the tests after
modifying a JSON file that was part of a golden test I realised that this wasn't
a very nice solution at all.
After looking around a bit I found emacs-multi-compile. I removed the minor mode
and add this configuration instead
I've had some vague thoughts that it'd be rather easy to write something
slightly more custom, but emacs-multi-compile works very well for me so I'm
very happy with it.
Jumping between implementation and test, again
I wrote about this a while ago. At the time I came up with a way of bending
ff-find-other-file to my will, but soon after that I ran into a case where
there may be 2 "other files" – one of the Haskell projects at work put unit
tests in files ending in Spec.hs and property tests in filed ending with
Prop.hs. Unfortunately this is a situation that ff-find-other-file can't
handle.
I ended up hacking together consult-kith to deal with it. The configuration to
handle two kinds of test files looks like this
As many of you know, my idea of a theme is to set the background to “oldlace” and the cursor to red. For the rest of it, I just take what Emacs gives me. That sounds pretty haphazard but it works out pretty well. It has enough contrast to be easy to read without searing my eyeballs and has enough syntax highlighting to be useful without being overpowering.
Prot (Protesilaos Stavrou) has a much more refined sense for these things. He considers what colors go well to together and things like contrast levels. Just the oposite of my “do with me what you will” approach. Nevertheless, I mostly don’t bother much with formally defined themes although I do occasionally like to take a look at them.
The reason for this post is Prot’s latest release of his Doric Themes. They’re sort of minimalist, like my non-theme theme but still very nice looking if you don’t want your Emacs screen to look like an arcade. The doric-almond variation is, in particular, very similar to my color scheme. Mine has a little more contrast: the foreground text is a bit darker and the keywords in my code buffers are a blue that’s slightly brighter than doric-almond’s light green. Still, it’s a theme that I’d be comfortable with if I hadn’t taken the easy way.
Some people have multiple themes and change them according to the time of day, their mood, the phase of the moon, or whatever. I’d hate that probably for the same reason that I don’t do well with trying to use different editors: I’d always be confused as to what’s what.
If you’re looking for a minimalist theme and my method seems too minimal, take a look at Prot’s Doric Themes. There are several variations in both light and dark mode.
More vc stuff, and this week's rabbit hole was git worktrees (Bozhidar Batsov's Working with Git Worktrees in Magit is what sent me down this particular rabbit hole!). It was an itch that manifested itself while reading this blog and mainly seeing that worktrees seemed to make a good deal of sense to me (especially as I am quite a subversion veteran). I have always felt uncomfortable with the default git in-situ branch switching and prefer to branch develop away from the mainline in a physical separate directory and this is just what worktrees facilitate. So how to do this with built-in vc-mode tools?, well as always this requires a little work and yes, I could use Magit, but I'm seeing how far I can push the built in vc.
The funny thing is that vc already understands worktrees perfectly well: a worktree is just a directory holding a .git pointer file instead of a .git folder, vc-git resolves that transparently, and diffs, commits and vc-dir all quietly do the right thing per worktree. What vc cannot do is actually seemingly manage them. There is no command to create one, switch between them, move one, list them, or throw one away, and the branch side is nearly as thin. Magit does all of this beautifully, of course - but I am stubborn about staying inside core Emacs where I can, so I spent an evening teaching vc-dir some new tricks. And yes, I do understand that vc in Emacs is in fact a generic abstraction and works well across all different types of version control, hence the reduction to a core subset of general source code commands, with a few extra added on of course for those obvious missing pieces, but I want to add the less obvious pieces, and certainly for git, worktrees is probably going to be one of them. Right lets get on with this and write some code!
Everything funnels through one small worker that shells out via vc-git-command and then drops you straight into vc-dir in the new tree:
Here is what I implemented:
;;;; -> vc-git-worktree-core
;;(defunmy/vc-git-branches ()
"Return a list of local branch names in the current git repository."
(let ((root (vc-git-root default-directory)))
(when root
(with-temp-buffer
(let ((default-directory root))
(vc-git-command (current-buffer) 0 nil
"branch""--format=%(refname:short)"))
(split-string (buffer-string) "\n" t)))))
(defunmy/vc-git--worktree-root ()
"Return the git worktree root for `default-directory'.
Signal a `user-error' when not inside a Git repository."
(or (vc-git-root default-directory)
(user-error"Not in a Git repository")))
(defunmy/vc-git-worktree-entries ()
"Return the worktrees of the current repository as a list of plists.
Each plist has keys :path, :head, :branch, :detached, :bare,
:locked and :prunable, parsed from `git worktree list --porcelain'."
(let ((root (my/vc-git--worktree-root))
entries current)
(with-temp-buffer
(let ((default-directory root))
(vc-git-command (current-buffer) 0 nil
"worktree""list""--porcelain"))
(goto-char (point-min))
(while (not (eobp))
(let ((line (buffer-substring-no-properties
(line-beginning-position) (line-end-position))))
(cond
((string-prefix-p "worktree " line)
(when current (push current entries))
(setq current (list :path (substring line 9))))
((string-prefix-p "HEAD " line)
(setq current (plist-put current :head (substring line 5))))
((string-prefix-p "branch " line)
(let ((ref (substring line 7)))
(setq current
(plist-put current :branch
(if (string-prefix-p "refs/heads/" ref)
(substring ref 11)
ref)))))
((string= line "detached")
(setq current (plist-put current :detached t)))
((string= line "bare")
(setq current (plist-put current :bare t)))
((string-prefix-p "locked" line)
(setq current
(plist-put current :locked
(if (> (length line) 6)
(substring line 7)
t))))
((string-prefix-p "prunable" line)
(setq current
(plist-put current :prunable
(if (> (length line) 8)
(substring line 9)
t))))))
(forward-line 1)))
(when current (push current entries))
(nreverse entries)))
(defunmy/vc-git--default-worktree-path (root branch)
"Return a default new-worktree path for BRANCH next to ROOT.
The default is a sibling directory of ROOT prefixed with the project
name, e.g. <parent>/myproj-feature for branch feature of myproj.
Slashes in BRANCH become dashes so hierarchical branches cannot
collide; an empty BRANCH falls back to <project>-worktree."
(let ((project (file-name-nondirectory
(directory-file-name (expand-file-name root)))))
(expand-file-name
(concat project "-"
(if (string-empty-p branch)
"worktree"
(replace-regexp-in-string "/""-" branch)))
(file-name-directory (directory-file-name (expand-file-name root))))))
(defunmy/vc-git-worktree-add (path branch &optional new-branch start-point)
"Create a Git worktree at PATH checking out BRANCH, then open `vc-dir'.
An empty BRANCH checks out HEAD. With prefix argument NEW-BRANCH,
create BRANCH as a new branch from START-POINT (defaulting to HEAD)
instead of checking out an existing branch or revision."
(interactive
(let* ((root (my/vc-git--worktree-root))
(branches (or (my/vc-git-branches) '()))
(make-new (if current-prefix-arg t nil))
(branch (if make-new
(read-string "New branch name: ")
(completing-read "Checkout branch (empty = HEAD): "
branches nil nil)))
(default-path (my/vc-git--default-worktree-path root branch))
(path (read-file-name "Worktree location: "
(file-name-directory default-path)
nil nil
(file-name-nondirectory default-path)))
(start (and make-new
(completing-read "Start point: "
(delete-dups
(append (list "HEAD") branches
(my/vc-git-tags)))
nil nil nil nil "HEAD"))))
(list (expand-file-name path) branch make-new start)))
(my/vc-git-worktree--create path branch new-branch start-point))
(defunmy/vc-git-worktree--create (path branch new-branch start-point)
"Create a Git worktree at PATH and open `vc-dir' there.
BRANCH names an existing branch or revision, unless NEW-BRANCH is
non-nil, in which case BRANCH is created from START-POINT first."
(let ((root (my/vc-git--worktree-root))
(expanded (expand-file-name path)))
(let ((default-directory root))
(cond
(new-branch
(when (string-empty-p branch)
(user-error"New branch name must not be empty"))
(vc-git-command nil 0 nil "worktree""add""-b" branch expanded
(if (string-empty-p start-point) "HEAD" start-point)))
((string-empty-p branch)
(vc-git-command nil 0 nil "worktree""add" expanded))
(t
(vc-git-command nil 0 nil "worktree""add" expanded branch))))
(message "Created worktree %s" expanded)
(vc-dir expanded)))
(defunmy/vc-git-worktree-checkout (path branch)
"Check out existing BRANCH in a new worktree at PATH, then open `vc-dir'.
The Magit-style `magit-worktree-checkout' first action: the location
defaults to a sibling directory named <project>-<branch>."
(interactive
(let* ((root (my/vc-git--worktree-root))
(branches (or (my/vc-git-branches) '()))
(branch (completing-read "Checkout branch in new worktree: "
branches nil nil))
(default-path (my/vc-git--default-worktree-path root branch))
(path (read-file-name "Worktree location: "
(file-name-directory default-path)
nil nil
(file-name-nondirectory default-path))))
(list (expand-file-name path) branch)))
(when (string-empty-p branch)
(user-error"Branch must not be empty"))
(my/vc-git-worktree--create path branch nil nil))
(defunmy/vc-git-worktree-branch (path branch start-point)
"Create new BRANCH from START-POINT in a new worktree at PATH.
Then open `vc-dir'. The Magit-style `magit-worktree-branch' first
action: the location defaults to a sibling directory named <project>-<branch>."
(interactive
(let* ((root (my/vc-git--worktree-root))
(branches (or (my/vc-git-branches) '()))
(branch (read-string "New branch name: "))
(start (completing-read "Start point: "
(delete-dups
(append (list "HEAD") branches
(my/vc-git-tags)))
nil nil nil nil "HEAD"))
(default-path (my/vc-git--default-worktree-path root branch))
(path (read-file-name "Worktree location: "
(file-name-directory default-path)
nil nil
(file-name-nondirectory default-path))))
(list (expand-file-name path) branch start)))
(my/vc-git-worktree--create path branch t start-point))
(defunmy/vc-git-worktree-move (path new-path)
"Move the Git worktree at PATH to NEW-PATH (`git worktree move').
Interactively, prompt for a worktree, defaulting to the current one."
(interactive
(let* ((entries (my/vc-git-worktree-entries))
(paths (mapcar (lambda (e) (plist-get e :path)) entries))
(current (my/vc-git--worktree-root)))
(unless paths (user-error"No Git worktrees found"))
(let ((path (completing-read "Move worktree: " paths
nil t nil nil current)))
(list path (read-file-name
"Move to: "
(file-name-directory
(directory-file-name (expand-file-name path)))
nil nil)))))
(let* ((root (my/vc-git--worktree-root))
(expanded (expand-file-name path))
(target (expand-file-name new-path)))
(let ((default-directory root))
(vc-git-command nil 0 nil "worktree""move" expanded target))
(message "Moved worktree %s to %s" expanded target)
(when (derived-mode-p 'vc-dir-mode) (vc-dir-refresh))))
(defunmy/vc-git-worktree-switch (path)
"Interactively select an existing Git worktree and open `vc-dir' in it."
(interactive
(let* ((entries (my/vc-git-worktree-entries))
(table (mapcar (lambda (e)
(cons (plist-get e :path) e))
entries))
(current (my/vc-git--worktree-root)))
(unless table (user-error"No Git worktrees found"))
(let ((completion-extra-properties
(list :annotation-function
(lambda (cand)
(let ((e (cdr (assoc cand table))))
(when e
(concat " ["
(or (plist-get e :branch)
(and (plist-get e :detached)
"detached")
"?")
(when (plist-get e :bare) ", bare")
"]")))))))
(list (completing-read "Switch to worktree: " table
nil t nil nil current)))))
(vc-dir path))
(defunmy/vc-git-worktree-remove (path &optional force)
"Remove the Git worktree at PATH, prompting first for confirmation.
With prefix argument FORCE, pass --force to git.
Interactively, prompt for a worktree, defaulting to the current one."
(interactive
(let* ((entries (my/vc-git-worktree-entries))
(paths (mapcar (lambda (e) (plist-get e :path)) entries))
(current (my/vc-git--worktree-root)))
(unless paths (user-error"No Git worktrees found"))
(list (completing-read "Remove worktree: " paths
nil t nil nil current)
(if current-prefix-arg t nil))))
(let* ((root (my/vc-git--worktree-root))
(expanded (expand-file-name path))
(main-root (plist-get (car (my/vc-git-worktree-entries)) :path))
(here (file-truename default-directory))
(gone (file-truename expanded)))
(when (yes-or-no-p (format "Remove worktree %s%s? "
expanded (if force " (forced)""")))
(let ((default-directory root))
(if force
(vc-git-command nil 0 nil "worktree""remove""--force" expanded)
(vc-git-command nil 0 nil "worktree""remove" expanded)))
(message "Removed worktree %s" expanded)
(when (derived-mode-p 'vc-dir-mode)
(if (string-prefix-p gone here)
(vc-dir (or main-root root))
(vc-dir-refresh))))))
(defunmy/vc-git-worktree-prune ()
"Prune stale Git worktree metadata (`git worktree prune')."
(interactive)
(let ((root (my/vc-git--worktree-root)))
(let ((default-directory root))
(vc-git-command nil 0 nil "worktree""prune"))
(message "Pruned worktrees in %s" root)
(when (derived-mode-p 'vc-dir-mode) (vc-dir-refresh))))
(defvar-localmy/vc-git-worktree-list--root nil
"Repository root shown in the current worktree list buffer.")
(defvarmy/vc-git-worktree-list-mode-map
(let ((map (make-sparse-keymap)))
(define-key map (kbd "RET") #'my/vc-git-worktree-list-visit)
(define-key map (kbd "g") #'my/vc-git-worktree-list)
(define-key map (kbd "q") #'quit-window)
map)
"Keymap for `my/vc-git-worktree-list-mode'.")
(define-derived-modemy/vc-git-worktree-list-mode special-mode "Worktrees""Major mode for listing Git worktrees.
\\{my/vc-git-worktree-list-mode-map}")
(defunmy/vc-git-worktree-list-visit ()
"Open `vc-dir' for the worktree on the current line."
(interactive)
(let* ((here (line-number-at-pos))
(btn (or (button-at (point))
(save-excursion
(beginning-of-line)
(next-button (point) t))))
(ok (and btn (= here (line-number-at-pos (button-start btn))))))
(if ok
(vc-dir (button-label btn))
(user-error"No worktree on this line"))))
(defunmy/vc-git-worktree-list ()
"List Git worktrees of the current repository in a dedicated buffer.
RET on a path opens `vc-dir' there; `g' refreshes the list, `q' quits."
(interactive)
(let* ((root (if (derived-mode-p 'my/vc-git-worktree-list-mode)
(or my/vc-git-worktree-list--root
(my/vc-git--worktree-root))
(my/vc-git--worktree-root)))
(default-directory root)
(entries (my/vc-git-worktree-entries)))
(unless entries (user-error"No Git worktrees found"))
(with-current-buffer (get-buffer-create "*vc-git worktrees*")
(let ((inhibit-read-only t))
(erase-buffer)
(my/vc-git-worktree-list-mode)
(setq my/vc-git-worktree-list--root root)
(setq default-directory root)
(insert (format "Worktrees for %s\n\n" root))
(dolist (e entries)
(let* ((path (plist-get e :path))
(head (or (plist-get e :head) ""))
(short (if (> (length head) 7) (substring head 0 7) head))
(flags (string-join
(delq nil
(list (when (plist-get e :bare) "bare")
(when (plist-get e :detached) "detached")
(when (plist-get e :locked) "locked")
(when (plist-get e :prunable)
"prunable")))
",")))
(insert-button path
'action (lambda (btn)
(vc-dir (button-label btn)))
'follow-link t)
(insert (format " [%s] %s%s\n"
(or (plist-get e :branch) "HEAD")
short
(if (string-empty-p flags)
""
(concat " (" flags ")")))))))
(goto-char (point-min))
(pop-to-buffer (current-buffer)))))
(with-eval-after-load 'vc-dir
(define-key vc-dir-mode-map (kbd "Z a") #'my/vc-git-worktree-add)
(define-key vc-dir-mode-map (kbd "Z b") #'my/vc-git-worktree-checkout)
(define-key vc-dir-mode-map (kbd "Z c") #'my/vc-git-worktree-branch)
(define-key vc-dir-mode-map (kbd "Z g") #'my/vc-git-worktree-switch)
(define-key vc-dir-mode-map (kbd "Z k") #'my/vc-git-worktree-remove)
(define-key vc-dir-mode-map (kbd "Z l") #'my/vc-git-worktree-list)
(define-key vc-dir-mode-map (kbd "Z m") #'my/vc-git-worktree-move)
(define-key vc-dir-mode-map (kbd "Z p") #'my/vc-git-worktree-prune))
Some interactive commands: Z a for the general case, Z b for checking out an existing branch, Z c for a new branch from a start point, plus Z m to move a tree, Z g to switch between them with branch annotations in the completion, Z k to remove one (prefix forces, and a vc-dir sitting inside the removed tree falls back to the main root rather than dying), Z p to prune stale metadata, and Z l for a listing buffer where RET jumps to vc-dir. The letters deliberately mirror Magit's own worktree map, so Z b, Z c, Z g, Z m and Z k all do what a Magit user's fingers expect; Z a, Z p and Z l are extras with no Magit equivalent. The same set hangs off C-x v z … globally.
So the whole loop, start to finish, never leaves vc: Z c to spin up the worktree, hack away, C-x v v to commit, Z g back to main, merge the branch, and Z k to remove the worktree.
If any of this sounds useful, the pattern to steal is a small one: vc-git-command with default-directory bound to the worktree root does almost all of the heavy lifting, and vc-dir on the resulting path gives you the status buffer for free. The rest is just prompts and keybindings.
Language-server-protocol (lsp) is standard to communicate information about a
current programming project to an editor integration, powered by a “language
server” for the specific programming language. Emacs has two major
implementations of this editor functionality: eglot (built-in) and lsp-mode.
Getting this set up is usually one of the first things that new users will want
to do, but in my experience one can get by for a surprisingly long time using
Emacs without needing to reach for eglot or lsp-mode. That is because many of
the functions that lsp-servers provide can be provided by Emacs through other
means.
As someone who quite often moves between different programming languages, I find
the need to set up each lsp-server to a bit clunky. Despite being presented as a
simple one-stop solution, you often need to research exactly what server to use,
how to set it up, and to tailor it to your machine in some ways. For me, this
only makes sense in a few languages that I use often, and even then it might not
be the first thing I take care of.
In-buffer completion
The most obvious functionality is auto-complete (aka in-buffer completion). In
Emacs this is provided on multiple levels, which is sometimes confusing for new
users. There are many ways to show in-buffer completions, but they are all
powered by the same back-end functionality: completion-at-point-functions
(capf). This is a list of functions that run in order, only returning a value if
there is something to complete.
Some major modes (most notable perhaps lisp-interaction-mode) come with their
own buffer-local values of capf1 that provide buffer-specific completions.
completion-at-point will then move on to the global value if the buffer-local
value of capf includes the value t. It is therefore this global value that we
are often interested in modifying.
For this we can use the cape-package — standing for “Completion At Point
Functions”. Specifically, we can make liberal use of the cape-dabbrev-function.
This capf makes use of Emacs’ built-in dabbrev-functionality. Dabbrev will scan
all the words in the buffers of your choice and give you completions for words
that you have already used. This means that, while working on your project, you
have quick and easy access to all the keywords and function names in your
currently opened buffers.
This also works as a form of autocomplete for writing prose, and so we can
combine these two using cape-wrap-super:
This will combine the output cape-dabbrev and cape-dict2 into one capf that
can run at the same time (and therefore not block each other). For a simpler
contribution we also use the cape-keywordcapf that comes with a number of
pre-configured programming language3 keywords. This is a bit of a pre-lsp
solution, where each editor provided support for each programming language. To
set up cape, simple add something like this to your init file:
This is the order in which cape will try your capfs, with the first matching
blocking the rest. As mentioned earlier, locally bound capfs will supersede
these global values, and so python-completion-at-point will run before any of
these in python-mode buffers.
Another improvement we can do is to change the value of
cape-dabbrev-buffer-function. The value of this variable is the function that
returns the buffers to scan for dabbrev results. By default this is
cape-same-mode-buffers, meaning that only buffers in the same mode (org-mode,
python-mode, et cetera) will be used for results. This is a good default, but I
would rather have always have some relevant completion candidate than nothing.
So personally I set this value to cape-text-buffers, which will return all
buffers in text-mode or prog-mode buffers. This means extra niceties like
referring to practically anything you are working on when writing a commit
message in git.
Jump to definition and references
A second major feature of lsp-servers is “jump to definition” that allows you to
go to the code that creates a function (its definition) to inspect its inner
workings. There are a few different ways to do this.
The first is the package named dumb-jump that uses ag, rg, or simply grep to
find definition-matching strings. This works similarly to cape-keyword in that
it uses a pre-configured set of language-syntax elements to match
function-defining cases.4 I however do not personally use this.
Instead I use another of Minad’s wonderful additions to the Emacs’ ecosystem;
consult. I must confess that I slept on consult for far too long. I had it in
my config for a long time, but the number of (largely stand-alone) functions
made it a bit overwhelming to learn.5 What Consult offers is a suite of
improved versions of preëxisting Emacs utilities (and a few new ones building
upon said features), largely based on the completing-read.
Emacs has a built-in feature called imenu that scans the buffer’s structure for
top-level features (Org-mode and markdown headers, or in this case function
definitions) and allows you to quickly move to them. Consult of course expands
on this by instantly showing you the place you will move to, but it also offers
an extension of this feature: consult-imenu-multi (as opposed to the regular
consult-imenu). This will not only scan the current buffer, but also all other
same-mode buffers. Better yet, it integrates with Emacs’ project.el, and so only
checks buffers that belong to the same project (exempli gratia git repository).
As I usually have the relevant buffers open in a project I am working on, this
works very well as an alternative to dumb-jump, and the preview and subsequent
confirmation mean that I am not worried about jumping to the wrong place —
although of course at the cost of speed.
Then we of course have the venerable xref built into Emacs. This works a bit
differently depending on the programming language backend, and plugs into an
lsp-server automatically if it is available. Use xref-find-references6
Otherwise we can use etags-regen to automatically create and refresh the TAGS
file that keeps track of the project structure:
The reason why I usually prefer using imenu is because it requires no trip to
inspect the state of any files on the system — everything is already loaded and
available, and perhaps 90% of the time it includes the things that I need (since
they are things I am working on).
Syntax checking
Here we can use the built-in flymake or the standalone flycheck. Both of these
make use of external syntax-checking tools for each language, and so do require
some more setup on the host machine than the above solutions. In fact, Flycheck
can even act as its own lsp-server just for syntax checking. But my experience
has still been that this is easier to deal with than lsp-servers that need to be
configured, started, reconnected, that crash, or have some other issue that need
to be dealt with; A simple unix-style shell command that prints text (and that
is then processed by Flymake or Flycheck) rarely needs much setup or maintenance
in my experience. This is a matter that I really just want out of my mind.
Conclusion
There are only so many things that can be done merely through these static
solutions. For example, neither of our in-buffer completion or
jump-to-definition solution will complete external libraries; only buffers that
have already been opened. There is no hover information.
I am not ideologically opposed to the use of language servers, and I will
sometimes intentionally start one if I expect to be working on a larger code
base for a longer amount of time. But tools like the ones above are nevertheless
incredibly useful because they work in all buffers and contexts.
One of the strengths of Emacs is how it acts as a sort of force-multiplier for
any task interacting with text, and how it builds upon improvements in one area
throughout the entire editor. Having some of the most critical lsp-like
abilities available anywhere you want without any extra effort is incredibly
powerful. This is especially noteworthy for things like cape-dabbrev since it
has made me so spoiled for autocompletion for everything I write.
I am sure that there are lots of different solutions and tools that I have
missed that would work to replace even more lsp-based ones, and probably
solutions that work better and/or are more elegant than the ones I use — the
answer may even be to just use lsp. But honestly I only think about the matter
of language server when I am talking with others. In day to day life I rarely
feel the need for them at all. ❦
If you are using the built-in switch-to-buffer command (by default bound to
C-x b) I highly recommend switching it to consult-buffer instead. Instant
“previews” is super useful, and I often just spam C-n (N being right next to B)
to cycle through my open buffers instead of starting to write the exact name
that I want.
James Cherti has another excellent post, this time on optimizing Emacs startup with use-package. The idea is that many configurations load a substantial number of packages and that this can cause a significant increase in startup time.
Most of you know by now that we here at the Irreal bunker refuse to worry about that because, after all, we hardly ever restart Emacs. Still, plenty of people do worry about it and there’s nothing wrong with learning a few easy to implement principles that will speedup Emacs startup without having to obsess about tiny details.
The TL;DR is to get use-package to defer loading of packages until they are actually needed. Most Emacers who aren’t complete n00bs already know this. The question is, how exactly, to specify the use-package commands to achieve that.
The obvious answers aren’t the right ones. For example, the :defer keyword seems like the natural answer but Cherti explains why this isn’t usually necessary and that it doesn’t do what you probably think it does. It’s much better to use one of the commands that implicitly defer loading but also arrange for autoloading the required functions when they’re needed.
Cherti also explains how to get use-package to add logging information to the *Messages* buffer to help with understanding the load process. There’s a lot of information in Cherti’s post and it serves as a handy reference for how to use use-package to control the loading of packages at startup time. If you’re at all worried about startup time or just want to understand how to use use-package to control package loading, take a look at his post.
The theme for the Emacs Carnival this September is "Games". Thanks to Raymond Zeitler for hosting!
One of the things I love about the Emacs community is people's sense of humour and whimsy. There's a grand tradition of people hiding Easter eggs in software, a bit of hidden fun that might make someone smile.
But isn't this just bloat and frivolity? One might argue that the code that lets us play M-x tetris, dunnet, or doctor in Emacs is a waste of time and space compared to More Serious Things, but on the other hand, the Tetris source code could be a good jumping-off point if you find yourself wanting to make an interactive real-time simulation that displays graphical results. Want something more modern? Maybe DOOM on Emacs!
Besides, the code for games isn't actually loaded until you choose to play it. It's okay for something to be fun by itself instead of needing to be productive. There's something wonderful about a tool that people choose not just for work but also for enjoyment.
Then take a step back and marvel at this community of people who try things out, tinker, follow that crazy idea that won't let go of them, and share what they came up with. If you write Emacs Lisp (or want to learn), rummage around in the source code for ideas on how to do things. See if you have any ideas that tickle your fancy. You might be able to get Emacs to do things text editors don't usually do! =)
Check out the Emacs Carnival page for Games to see other posts!
These are my minimalist themes. They use few colours and will appear
mostly monochromatic in many contexts. Styles involve the careful use
of typography, such as italics and bold italics.
If you want maximalist themes in terms of colour, check my ef-themes
package. For something in-between, which I would consider the best
“default theme” for a text editor, opt for my modus-themes.
I drew inspiration for both themes from plants that exist around my
house. They both use the same colours in different shades and
sequence. doric-lilac has a light background while doric-borage
has a dark background.
More commands to load a theme
The commands doric-themes-rotate-light and doric-themes-rotate-dark
are convenience wrappers of doric-themes-rotate. As their name
suggests, they are restricted to one kind of theme in the collection.
Similarly, doric-themes-load-random-light and doric-themes-load-random-dark
are variants of doric-themes-load-random.
As an Emacs user, your configuration can easily grow from a few lightweight adjustments to a massive, hundred-package IDE. Without careful management, Emacs startup time can degrade from sub-second execution to several seconds, or minutes, in the worst cases. Eager package loading is one common source of startup overhead. This guide explains how Emacs loads libraries, how use-package configures package loading, and how deferred loading can reduce startup time.
Before we dive in, please consider sharing this article on your website, blog, Mastodon, Reddit, X, Linkedin, or other social media platforms.
Why does Emacs startup get slow?
A naive package declaration looks like this:
;; The kirigami Emacs package provides a unified method to fold and unfold;; text in Emacs across a diverse set of Emacs modes.
(use-package kirigami)
By default, this causes use-package to load the package during initialization, unless the declaration contains a deferring keyword or another option that changes the loading behavior.
Conceptually, the eager loading is equivalent to:
(require 'kirigami)
The require function ensures that a feature is loaded.
When many packages are loaded this way, the time spent loading and evaluating their code contributes to increasing startup time.
What is Autoloading?
Autoloading registers a function without immediately loading the library that defines it. The library is loaded only when the function is called. For example:
;; Simplified representation of an autoload registration
(autoload 'kirigami-global-mode "kirigami""Global mode for Kirigami."t)
If M-x kirigami-global-mode invokes the autoload, Emacs loads the kirigami.el library ("kirigami", the second argument), and then calls the actual kirigami-global-mode function. This allows the library's startup cost to be deferred until the function is actually needed.
Explicit vs. implicit deferral in use-package
The use-package macro makes package configuration and loading declarative. By default, a declaration is eager, meaning the package is loaded immediately while Emacs processes the configuration during startup, rather than waiting until the package is needed. A deferring keyword or another loading mechanism can postpone loading until later.
The explicit :defer keyword
Deferral using :defer t (Not necessary)
Note: Explicitly using :defer t is often unnecessary. Keywords such as :commands, :bind, :hook, :commands, and :mode establish loading triggers automatically. These keywords are discussed below.
If a package does not have another loading trigger, :defer t prevents use-package from loading it immediately:
(use-package kirigami
:defert)
This prevents the package from being required during startup. Without an autoload or another loading trigger, the package will remain unloaded until something else loads it.
Idle deferral: Numeric argument (Not ideal)
A numeric value to :defer schedules the package to be loaded after Emacs has been idle for the specified number of seconds:
;; Load kirigami after Emacs has been idle for 20 seconds
(use-package kirigami
:defer20)
Note: This is not ideal because loading a heavy package after a few seconds will block the event loop and can cause a freeze in the user interface. If multiple packages are deferred via idle timers, they can trigger sequentially during a single idle period, increasing the delay. To prevent unprompted blocking operations, it is best practice to configure explicit autoloads, such as :hook, :bind, :commands, or :mode. (These keywords are discussed below.)
Instead of manually adding :defer t to every use-package declaration, you can use trigger keywords to configure deferred loading automatically. These keywords generate autoloads, ensuring the package evaluates only when you interact with its components:
:commands: Generates autoloads for specific interactive commands. Emacs loads the package when you execute the command (for example, via M-x).
:bind: Maps keys to commands and automatically creates autoloads for them.
:hook: Adds a package function to a hook and arranges for deferred loading.
:mode: Adds a file-pattern entry to auto-mode-alist and defers loading until the mode is needed.
Example: Autoloading on major mode file extensions
Instead of loading Markdown mode immediately, use :mode to defer it until you open a .md file:
Note: Specifying :defer t is unnecessary here, as :mode automatically defers package loading.
Example: Autoloading on keybindings
The :bind keyword maps keys to commands and automatically creates autoloads for them. The package remains deferred until you press one of the defined key combinations:
If you do not want to define global keybindings but still need to defer loading until a command is called interactively (for example, via M-x), use the :commands keyword. This generates the necessary autoloads:
Instead of eagerly loading outline-indent-minor-mode (indentation-based code folding) at startup, you can configure Emacs to load and activate it on demand. This ensures the package evaluates only when a python-mode, python-ts-mode, or yaml-ts-mode buffer is opened:
This also loads outline-indent when you invoke the command manually via M-x outline-indent-minor-mode, outline-indent-backward-same-level, or outline-indent-forward-same-level.
Note: There is no need to specify :defer t separately. The :hook keyword adds outline-indent-minor-mode to python-mode-hook, python-ts-mode-hook, and yaml-ts-mode-hook.
The unconditional use-package :init block
Code inside the use-package:init section is evaluated before the package is loaded. An autoloaded function called from :init can itself cause the package to load immediately, defeating the intended deferral. Therefore, :init is generally appropriate for settings that must be established before the package loads, while package-dependent function calls belong in :config or another deferred trigger (e.g., with-eval-after-load).
(Similarly, code inside the :preface block evaluates before package loading, but it also executes during byte-compilation. This distinction makes :preface the correct location for code required to satisfy the byte-compiler, such as defvar statements, helper macro definitions, or declare-function calls that prevent compiler warnings for deferred libraries.)
Forcing immediate load with :demand
With global deferral enabled, you will occasionally encounter packages that must be loaded immediately (e.g., themes, keybinding managers, or daemon-side servers). You can override global deferral on an individual basis using :demand t:
(use-package tomorrow-night-deepblue-theme
:demandt; Force immediate loading:config;; Load the tomorrow-night-deepblue theme
(load-theme 'tomorrow-night-deepblue t))
Note: If a declaration specifies both :demand t and :defer t (or triggers), :demand t takes precedence and forces eager loading.
The use-package :after keyword
The use-package:after keyword allows you to defer the initialization of a package until one or more specified target packages have been fully loaded. This is practical for:
Add-on packages that extend a base minor mode.
Integration packages that bridge two distinct tools together.
Consider embark-consult, a package that integrates embark with consult. You only need the embark-consult package after embark and consult are loaded:
In this scenario, embark-consult remains entirely inactive during startup. Emacs won't evaluate the embark-consult configuration until it has successfully loaded both embark and consult.
Although :defer t is present, evaluating (kirigami-global-mode 1) during :init triggers the autoload for kirigami-global-mode. The package is therefore loaded during initialization, defeating the intended deferral.
Loading after the Emacs init phase
If you prefer to load a package after the init phase, use the :hook keyword:
Using :hook (after-init . kirigami-global-mode) adds the mode to after-init-hook. This ensures the package remains deferred while Emacs evaluates the rest of your init file, guaranteeing the mode is active before the editor is ready for user interaction.
Global laziness: use-package-always-defer (Not recommended)
You can configure use-package to automatically assume :defer t for declarations that do not otherwise establish eager loading:
(setq use-package-always-defer t) ; NOT RECOMMENDED
This changes the default behavior so that use-package declarations are deferred unless a declaration explicitly requires immediate loading or establishes another loading mechanism.
Note: Global deferral breaks packages that rely on background execution, global hooks, or immediate side effects to function correctly. You will find yourself tracking down silent failures and adding :demand t throughout your initialization file to force eager loading for things like themes, modeline managers, and daemon servers. Instead of relying on a blanket setting that obscures the loading state, setting use-package-always-defer to nil and explicitly deferring packages where appropriate results in a more deterministic setup.
Diagnostics and verification
To check whether a package feature has been provided in the current Emacs session, use the featurep function. For example, for the kirigami package:
(featurep 'kirigami)
Evaluates to nil when the feature has not been provided.
Evaluates to t when the feature has been provided.
Debugging and macro expansion optimization
If you need to diagnose loading order or macro expansion issues, use-package provides built-in settings to help debugging and performance.
Setting use-package-expand-minimally to t causes the use-package macro to generate less boilerplate code. If you byte-compile your initialization file, this reduces the size of the compiled file and can yield slight performance improvements:
(setq use-package-expand-minimally t)
Setting use-package-verbose to t forces use-package to log package loading events to the *Messages* buffer. This is useful for verifying whether deferred packages are loading when expected:
(setq use-package-verbose t)
Setting use-package-enable-imenu-support to t causes imenu to see use-package declarations. This allows you to quickly jump to specific use-package blocks using M-x imenu:
(setq use-package-enable-imenu-support t)
By default, the :hook keyword automatically appends -hook if you omit it. Setting use-package-hook-name-suffix to nil disables this behavior. Disabling this forces you to write fully qualified hook names (e.g., (python-mode-hook . outline-indent-minor-mode)):
(setq use-package-hook-name-suffix nil) ; NOT RECOMMENDED
Note: It is not recommended to set use-package-hook-name-suffix to nil if you maintain a pre-existing configuration that relies on the default implicit behavior. Modifying this variable globally instantly breaks every existing :hook (mode . function) declaration that omits the -hook suffix. Unless you are building a configuration from scratch or can rewrite and validate every hook assignment in a single pass, leaving the default behavior intact is the most practical decision.
Startup profiling: measuring the impact
To benchmark startup optimizations, measure startup at a point later than the end of init-file processing.
use-package can collect loading statistics. Enable statistics gathering after use-package has been loaded but before the use-package declarations are evaluated:
(setq use-package-compute-statistics t)
After restarting Emacs, execute: M-x use-package-report
This displays a tabulated buffer with timing information for the use-package phases.
Advanced profiling with benchmark-init
While use-package-report offers insights into declarative package loading, the third-party package benchmark-init (available on MELPA) provides a more granular view of the entire Emacs startup process. It tracks the time spent loading individual files and executing functions across the whole initialization sequence, allowing you to see exactly where you can influence startup time.
Way back before Vim 6.0, Bram Moolenaar added the UserGettingBored autocmd to the Vim documentation. It was a joke and the command didn’t actually do anything—or rather, it was never invoked and didn’t really exist. It was like an Emacs hook that was documented but didn’t actually occur. The documentation said “When the user hits CTRL-C. Just Kidding.” In 2013 the documentation changed to “When the user presses the same key 42 times. Just kidding! :-)” and, there were various other tweaks along the way. Finally, in 2022 an unofficial plugin was created by Mike Smith that actually implemented the autocmd and did something. You can read what it was in Hahn’s post.
None of this matters at all, of course, except to provide evidence that Vim users can be as whimsical as us Emacs users. You probably won’t see anything like this in VSCode. It wouldn’t be “professional”.
A quick little update to
next-gh-pr.el. Since writing
the first version I've found it to
be really useful, but I've also bumped into one small issue. Sometimes I'll
create a draft PR before I remember to start the changelog entry, and so
then I want to insert the current PR link rather than the next one.
Now, sure, I can just go and copy the URL and create the link in the
changelog, but where's the fun in that when I can solve the problem in
Emacs Lisp? So, with this in mind, I've just updated the
package so that, with a prefix argument to the command, it will include a
link for the latest PR rather than for the next one.
This does, of course, make the name a little less correct, but I'm not going
to rename the package or the command for this one minor difference in
intent.
I am still using my Kinesis Freestyle Edge keyboard. I think I’ve decided on the Glove80 as a replacement, but the price and the inability to easily return it are giving me second thoughts. Its style is very different, and I know I will have a hard time adjusting. Still as someone pointed out to me it’s not like I’m forced to throw away my current keyboard, which is true. I see myself using my Kinesis for work on the Mac while trying the new Glove80 on Linux for blog posts and more leisure-time typing.
I’ve made a few adjustments, however:
First, the spread between the two halves is much wider. This is something I was considering when I first got this keyboard, but I didn’t follow through with it. I got used to the wider split within a couple of days. It does make my shoulders feel better and more natural, especially when I’m standing at my desk. With that, I also started using the keyboard’s full tenting capability at all times: I used to have the keyboard flatter when sitting down.
The mouse is in the middle, and since I use it for quick bursts between otherwise typing sessions, it’s fine. When I game, I move it back to the right. I can basically ignore the entire right half of my keyboard while gaming, since most of the action happens on the left.
Gone is the Caps Lock as Ctrl. Caps is just Caps again. I’m still pressing it now and then out of habit, but for the most part, I switched back to using the original Control keys. I try to force myself to use my thumb, which means I have to curve my thumb into my left palm if I use the left side, or I just go all the way to the right and use my Ctrl key there. This is the most difficult part because it’s not just muscle memory: the Ctrl key is out of the way, and it can slow me down.
I also realized that the Windows key, which also acts as the Command key on my Mac, should be way more prominent than this keyboard allows. Thanks to PopClip on the Mac, I don’t use it much for copy-pasting on macOS (and since you have to use your mouse anyway to highlight text in Emacs, having PopClip is just a natural extension that I’m happy to have).
Speaking of the mouse, I know some of you will wrinkle your nose in disgust — but I started using it more in Emacs. Sometimes when I want to select text further down or up from where I am with the mark, it’s just easier to grab it and highlight the text this way instead of using the keyboard. I am also in the process of rediscovering the menus in Emacs (yes, those menus that all of you Emacs gurus love to hide), especially when it comes to reminding myself of useful shortcuts I keep forgetting — like registers, the mark ring, and other org-mode tricks I keep forgetting about.
My pinky still annoys me, but less. I seem to have this habit now of letting it hover over the keyboard instead of resting on the shift key. It’s almost as if I lost my ability to place my left hand correctly and naturally at times. It’s one of the things I hope the Glove80 might help with.
As the result of a long standing discussion on the Emacs devel list, Emacs 31 comes with what amounts to a starter kit called newcomers-presets1. The idea is to provide newcomers with a set of useful configurations to help them ease into Emacs. It is, however, much more than training wheels for n00bs. The presets described in it are ones that most experienced Emacs users are already using.
I know this because last April Sacha Chua wrote a splendid post that describes what’s in it. I somehow missed it back when she first posted it but she mentioned it in the latest Emacs News. As I’ve said before, I don’t usually write about things that she’s already covered in Emacs News but I broke that rule yesterday so I might as well break it again, especially since her post is especially worth your attention.
When newcomers-presets was first introduced I didn’t pay too much attention to it but then Prot recommended it for experienced users as well as beginners. The problem was that I was unclear on what exactly it did. Then I saw the reference to Chua’s post. It has a comprehensive list of what it does along with a explanation for each setting.
I won’t be using it because I’m not a fan of starter kits for experienced users and because even if I like everything in it today, it could change tomorrow. It’s better, in my opinion, to see what’s in it and add the things you think are useful to your own configuration. Chua includes a handy “Copy code” button with each item to make that easy.
I’ll admit that until fairly recently I had no idea git worktrees existed.
They’ve been part of git for a decade1 and I never once needed them. Feature
branches did the job just fine - create a branch, do the work, merge it, delete
it, and start over.
What finally introduced me to worktrees was, of all things, AI coding agents.
Tools like Claude Code create a worktree for each task, so several agents (or
several sessions of the same agent) can work on the same repo in parallel
without stepping on each other - or on you. Suddenly my projects directory was
full of cider-this and cider-that siblings, and I figured I should
understand what’s actually going on there.
Worktrees vs Branches
A branch is just a movable pointer to a commit, and making one is basically
free. The catch is that a repository has a single working directory, so working
on two branches means switching that directory back and forth. You know the
routine: stash your half-done work (or commit it), check out the other branch,
do the thing, check out the first branch again, unstash. It works, but it’s
tedious, and it gets worse when the two branches leave your project in
different build states and every switch means recompiling half the world.
A worktree gives you an additional working directory attached to the same
repository:
Now ~/projects/cider-smart-targeting is a full checkout of that branch, while
your main checkout stays exactly where it was. The object database, refs,
stashes and remotes are all shared - a worktree is not a clone, so fetching in
one is fetching in all, and creating one is nearly instant. Each worktree gets
its own HEAD and index, and git enforces one simple rule: a branch can only
be checked out in one worktree at a time.
When are they worth it? Whenever two things need to happen at the same time: a
long test run on one branch while you work on another, reviewing a PR without
disturbing your half-done work, or - the reason everyone is talking about them
these days - AI agents doing their thing in isolation. The price is pretty
modest: your working files exist on disk more than once, and anything that
isn’t tracked by git (dependencies, build caches, node_modules and friends)
has to be set up again in each worktree.
If branches have never felt limiting to you, that’s fine - they didn’t for me
either, for over a decade. Worktrees are one of those features you don’t miss
until your workflow changes.
What About Jujutsu?
While we’re on the topic of working copies - the most interesting thing
happening in version control right now is
Jujutsu (jj), a git-compatible VCS that makes
the problem worktrees solve mostly go away. In jj the working copy is a
commit, snapshotted automatically as you work. There’s no staging area and no
stash, because there’s no uncommitted state that could be lost or get in the
way - switching contexts is always safe. It also has proper support for
parallel working directories (jj workspace), plus an operation log that makes
practically everything undoable. That last bit is part of why the AI agent
crowd has taken an interest in it too.
You can use jj on top of an existing git repository (they call this a colocated
repo), and your colleagues - and Magit - will keep seeing a normal git repo.
I’m still just an observer here, but it’s clearly a project to keep an eye on.
Worktrees in Magit
Back to Emacs land. Magit has had worktree support for years, hiding behind Z:
Key
Command
Description
Z b
magit-worktree-checkout
Check out an existing branch in a new worktree
Z c
magit-worktree-branch
Create a new branch and worktree in one go
Z g
magit-worktree-status
Jump to another worktree’s status buffer
Z m
magit-worktree-move
Move a worktree
Z k
magit-worktree-delete
Delete a worktree
The best part is that there’s nothing else to learn. Each worktree gets its own
Magit status buffer, and every Magit command operates on the worktree the
current buffer belongs to. Z g is the only switching mechanism you need, and
even that is just a shortcut for visiting another status buffer.
A few practical tips from my (admittedly recent) experience:
By default the status buffer doesn’t list your worktrees. Fix that with:
Now every status buffer shows all worktrees of the repo, and you can hit
RET on any of them to jump there.
Create worktrees as siblings of the main checkout with descriptive names
(cider-smart-targeting next to cider), not nested inside it - nesting
confuses grep, find and plenty of other tools.
Magit’s branch selection annotates branches that are checked out in another
worktree with the worktree’s path, and refuses to check them out a second
time (that’s git’s rule, not Magit being difficult). If you ever wondered why
a branch “won’t check out”, that’s usually why.
Each worktree is its own project as far as project.el (or Projectile) is
concerned, so project switching, per-project buffers and search all work
naturally.
Anything that’s not tracked by git doesn’t come along for the ride. You’ll
have to install dependencies once per worktree and start with a cold build
cache. For Elisp that costs you nothing; for a big JVM or JS project it’s the
main downside of the whole approach.
When you’re done with a worktree, delete it with Z k (or with
git worktree remove from the command line). If you just delete the
directory by hand, git worktree prune will clean up the leftover
bookkeeping.
Closing Thoughts
These days my workflow with agents usually looks like this: an agent does its
work in a worktree, I review the changes there in Magit (often while another
task is running in a second worktree), and the worktree goes away once the
branch is merged. Maybe one day I’ll find other uses for worktrees - we’ll see.
Are you using git worktrees - and did you discover them the same roundabout way
I did? I’d love to hear about it in the comments!
That’s all I have for you today. Keep hacking (in parallel)!
They were introduced in git 2.5, released all the way back in July 2015. ↩
I just installed Alpine Linux on an old laptop. Works great!
Last year, I complained about all of the lightweight Linux distributions being based on Debian. Alpine is not based on Debian and is very lightweight indeed; it uses busybox in place of many GNU utilities, musl libc in place of glibc, and openrc in place of systemd. Alpine is so small that lots of folks use it for containers and things.
Prot: Tu peux prendre le lien correct cette fois.Fabrice: Ah, voilà, je suis prêt avec.Sacha: Ah, ok.Ok.Nous sommes en direct.Prot: Ah, nous sommes en direct.Très bien.Sacha: Donc, je vais passer à vos notes.Bonjour à toutes et à tous et bienvenue au 30eépisode d'Emacs Chat.C'est la suite de notre conversation avec Fabrice Nissen.Merci de nouveau Fabrice d'être là et merciégalement à Prot de nous rejoindre.Donc, on continue.Lors de la conversation précédente, nous avonsparlé de ton expérience sur Windows et sur WSL,des thèmes de publication vers HTML ou vers PDF,de ton kit de configuration Leuven, des petites améliorationset des automatisations et des formations et descours particuliers en français, en anglais, ennéerlandais et en espagnol.C'est incroyable, ça!Du coup, si ça ne te dérange pas, j'essaie d'ensavoir plus sur ton flux de travail, puis repasserà une discussion sur l'enseignement d'Emacs,de nombreux changements et de grandesincertitudes à l'époque de l'IA.Qu'en penses-tu?Depuis notre conversation, tu as ajouté beaucoupde fonctions, d'outils, de préréglages (presets)de gptel à ta configuration.C'est un sujet très tendanceYouTube a évidemment beaucoup de videode ce genre en anglais, mais pas encore en français, je pense.Peux-tu nous montrer ton flux de travail avec l'IA sur Emacs?Fabrice: Volontiers. Merci pour l'invitation et l'image bouge.Merci pour l'invitation pour cette trentième fois.Donc, je vais partager mon écran.Prot: Merci.Fabrice: Alors, je tape share, voilà.Et donc, ce sera celui-ci.Est-ce que vous voyez mon écran ?Prot: Oui, je peux voir ton écran.Fabrice: Ok, très bien.Très bien.
3:00gptel
Fabrice: Donc en fait, sur gptel, j'utilise pas encorebeaucoup, beaucoup, beaucoup, mais de plus en plus.Je suis aussi un peu novice, un peu, mais j'aieffectivement, et récemment d'ailleurs, faitquelques petites améliorations de layout au standard.Donc j'ai, par exemple, utilisé le raccourciC-<F1> pour lancer…Je n'ai pas lancé le chat.Je vois que je n'ai pas ma dernière configuration.C'est peut-être l'occasion aussi de parler,juste un petit break.Vous voyez que j'utilise Helm.Je ne sais pas ce que vous utilisez vous.Je suis extrêmement content de Helm, ce qui mepermet de taper.Je ne dois plus savoir où se trouve un fichier,dans quel répertoire.Il suffit que je connaisse quelques morceaux de son nom.Et je vais retrouver directement le fichier ou lebuffer actif.Donc ici c'est gptel et ce sera le point el.Il est là.Donc ici, je vais lancer et je vais faire vbrapidement pour le keychord eval-buffer.Voilà, donc il est évalué.Donc je vais avoir un look un peu différentmaintenant si je recommence.Voilà, C-<F1>.Donc on peut lui dire bonjour.C-c C-c pour envoyer.Voilà.Et donc ici il répond.Donc une petite utilisation que j'ai faite, c'estgptel-highlight-mode qui permet de mettre laréponse avec un certain fond et une petite barreici dans la frange pour bien identifier ladifférence entre les questions et les réponses.Je peux lui demander d'écrireUn exemple de code Python.S'il fait comme d'habitude, il va faire la suitede Fibonacci.Voilà.Et donc, qu'est-ce qu'il a fait ?Ah non, les nombres premiers.Ici, c'est quelque chose que je commence àutiliser de plus en plus, le buffer deconversation gptel.Une petite customisation que j'ai faite.D'ailleurs, j'ai envoyé un issue à l'auteur degptel pour éventuellement qu'il l'introduisedirectement dans...Dans gptel, c'est le fait de recoloriser lescodes blocs dans les réponses en utilisant lacouleur que j'ai dans mon thème pour Org Mode.Donc ici, c'est exactement comme dans un fichierOrg Mode.J'ai le fond jaune pour le code et lesdélimiteurs ici, de début et de fin, en gris.Et ça, c'est très important, je trouve, pourmettre en évidence comme il faut le code bloc.Et le fait, bon ici ce n'est pas de l'Emacs Lisp,mais si j'avais demandé, si je pouvais demanderde faire la version, donnez-moi la version del'Emacs Lisp.C-c C-c.Et l'avantage évidemment de faire ici, donc à lafin de la réponse, il y a un scroll automatiquevers le bas.Ah d'accord.Et ce qu'il fait aussi, c'est qu'il fait unformatting pour que ça reste moins de 80caractères.Et donc ici, l'avantage, c'est que c'est dansEmacs et que c'est l'Emacs Lisp.Je peux directement faire C-x C-e. Et jepourrais tester les fonctions.Je pourrais faire ici.Ça me permet d'éviter de faire des copier-collerentre ChatGPT ou Copilot ou n'importe lequel.Ici j'ai tout directement là dedans.C'est extrêmement pratique.Sacha: Tu as aussi des fonctions pour ajouterdes contextes, des fonctions, d'autres chosesdans ta configuration.Au lieu d'utilisation de conversation buffer, tuas aussi utilisé les fonctionnalités gptel danston code.Fabrice: Il y a le rewrite que j'utiliseparfois, ou alors pour répondre avec le contexte,il y a notamment le add context ou add context file.Je n'ai pas encore beaucoup joué avec, donc jesais qu'on peut aller dans Dired et puis fairegptel-add-file, add context file, je crois.Et je l’ai déjà bindé sur C-c g, donc tout cequi est C-c g et c’est gptel.J’ai déjà utilisé le raccourci avec un petit a,donc C-c g a pour ajouter du contexte.Mais je ne l'ai pas encore vraiment utilisé, doncje n'ai pas encore beaucoup d'expérience avec.Par contre, je fais parfois dans des fichiers, jepeux faire, si je prends un fichier Org, on peutaller dans le fichier ici avec mes petites notesd'aujourd'hui.Ça pourrait m'arriver ici de dire je veuxtraduire ces deux paragraphes en anglais parexemple.Je vais les sélectionner.Je vais faire C-u C-c RET pour avoir lemenu de gptel.Là, je vais changer la directive puisque ladirective ici c'est tu es un assistant, etc.Je vais changer la directive.Je vais en fait utiliser le S, Set System Message.Là, j'ai une série de prompt, en fait, de systemprompt, de directives, ça s'appelle, sous gptel.Donc, une série de directives, de prompt qui sont prêts.Mais ici, je vais faire un spécifique.Donc, je vais encore faire une fois s. Et là, jevais taper, par exemple, traduit.Là, quand on a fini, on peut faire C-c C-c.Ici, on voit la directive qui est là.Pour l'instant, si je ne change rien, il vaenvoyer ma sélection et il va insérer la réponse.Là, je ne vois pas très bien.Il faut que je déplace un truc.Il va insérer la réponse à la fin.Je pourrais faire ça, par exemple, pour voircomment il a traduit.Comme ça, je garde au-dessus l'input.Je vais faire effectivement RET ici.Parfois, c'est un peu compliqué à lire.Prot: Qu'est-ce qu'il m'a fait ?Fabrice: On dirait qu'il a pris plus delignes.Sacha: Il y a un problème des démos en direct.Fabrice: Je vais bien le refaire.C-u C-c RET.Je vais bien sectionner ma région.La directive est là.Donc, je vais faire ici Enter.Voilà.Donc j'ai vu que l'input était, la section étaittoujours sélectionnée, donc j’ai fait C-w pourl'entrée, et puis j'ai gardé ce qu'il y a.Donc ici, voilà, typiquement, il a fait le jobfacilement, donc je suis nouveau restant dansDirectement dans Emacs, on fait des copies-colleravec d'autres interfaces externes.Sacha: Je vois que tu peux aussi remplacer lasélection avec les résultats ou peut-êtrerediriger l'output à un autre tampon pourcomparaison.Fabrice: Exact.Donc, je n'ai pas encore tout utilisé, mais jepourrais ici essayer.On va essayer.On va traduire en français, par exemple.Donc, j'ai bien sélectionné.Je vais faire C-u C-c RET pour avoir denouveau le menu.Ou alors, M-x, j'ai fait tel menu.Donc là, je vais changer et je vais mettre ladirective 2 fois S. Et cette fois-ci, je vaistraduire en français.C-c C-c, voilà.Et donc, je vais mettre ici le respond in place.Et donc, en théorie, il va écraser ma sélectionpar sa réponse.Donc, je vais faire le i ici.Et donc, en fait, il faut toujours lire ladernière ligne ici.On voit ce qu'il va faire.C'est un résumé de ce qu'il va faire en fonctiondes options qu'on a choisi.Donc ici, quand je vais faire ret, quand je vaisfaire return, il va remplacer la sélection avecla réponse.Donc je fais RET.Ah, et là j'ai peur.The conversation must end with a user message.Qu'est-ce qu'on va faire ?On va rester, on ne sait jamais.C-x C-x C-u RET…Non, ce n'était pas ça.C-x u. C-x u. Voilà.C-x C-xC-u RETBen si, c'est ça.Non, C-u C-c RET.Il y a trop de raccourcis.Je fais la sélection, voilà.C-u C-c RET.J'ai mon menu, j'ai ma directive en haut qui est bonne.Je vais faire le i, Replace, la sélection duresponse, RET.Bon, j'ai l'erreur.Donc là, il faudrait que je regarde un peu plus.Je n'ai pas beaucoup utilisé ça.Et en fait, justement, sur le rewrite, il fautque j'utilise aussi un peu plus pour du texte etpour du code.Et je sais qu'il y a plusieurs possibilités après.
13:42diff
Fabrice: Il y a aussi moyen de dire, il va remplacer enthéorie par le nouveau texte, mais il y a moyende montrer aussi son changement sous forme dediff ou de ediff.Donc il y a moyen de dire accepte.J’ai vu qu’après on peut faire C-c C-apour accepte, C-c C-e pour ediff,C-c C-d pour diff.Et donc il peut rouvrir différents buffers pourbien visualiser la différence.Si c'est une traduction, tout serait changé.Et si on modifie du code, par exemple, en disantrajoute-moi un paramètre, on a envie de voirvraiment la différence et de sûr qu'il n'a pasfait plus de choses que ce qu'on a demandé.Donc il y a moyen de faire un diff ou un ediff après.Prot: C'est peut-être mieux comme ça avec Ediff.Fabrice: Oui, moi j'aime beaucoup Ediff.D'ailleurs, c'est quelque chose, je ne crois pasque je l'ai montré l'autre fois.Donc ici, moi j’ai C-<f9> chez moi, c’est
privé,pour lancer vc-dir sans me poser de questions surquel est le répertoire qu'on veut analyser.Donc ici, je vais faire C-<f9>.Donc il me lance vc-dir sur le répertoire danslequel je me trouvais.On voit que j'ai fait une modification dans lefichier readme.Si je fais égal, on voit un diff unifié avec lesplus et les moins en dessous.Il faut toujours relire avant de committer lesdiffs parce que là il y a une ligne qui n'étaitpas prévue d'être committée en imaginant que jedevais faire le commit.Je vais d'ailleurs effacer cette ligne.Ce qui est pratique ici dans tout ça, c'est quej'ai fait une modification, j'ai sauvé.Ce buffer-ci est le diff.Je n'ai pas besoin de retourner dans le vc-dir etde refaire un égal.Je peux faire g, faire refresh, il recalcule lecontenu.Donc ici, j’ai fait g et simplement, il arecalculé le diff.Et je ne vois plus que le paragraphe qu'on devaittraduire, qui a été effacé.Pour l'instant, c'est une possibilité.Et l'autre possibilité à partir de vc-dir.Pour l'instant, je n'utilise pas encore Magitparce que j'ai eu pendant des années desrépertoires, des repos qui étaient sous SVN,subversion.Certains étaient déjà sous Git.Je voulais une seule interface pour gérer tousmes repositories.Et donc, j'ai tout construit sur vc-dir qui marchetrès bien.Voilà, je suis sûr que Magit, pour Git, estplus sophistiquée, plus jolie à regarder, sans doute.Mais en fait, je fais tout mon boulot ici avec vc-dir.Donc, si j'avais égal, j'ai un diff unifié.Ça, c'est standard, je crois.Si je fais grand E, ça, c’est pas standard.Ça m'appelle ediff.Donc là, directement, j'ai ediff, puis j'ai plusqu'à faire next.Voilà, next.Bon, j'ai qu'un changement ici.Ce paragraphe là qui a disparu.Donc ça c'est très pratique, directement pouvoirlancer Ediff à partir d'un fichier modifié.Et alors une petite chose que j'ai récemmentrajoutée dans Git, c'est donc ici je viensde refaire un égal pour avoir un Ediff unifié.Donc ici j'ai leLe diff unifié où j'ai ce paragraphe-là qui est apparu.En fait, depuis n'importe quel diff, donc outputqui est sous forme de diff avec des plus et des moins,Ici, c'est à partir de vc-diff.Ça pourrait être le résultat d'un Git diff dansun buffer.Ça pourrait être dans Magit.Je peux lancer la touche avec w, ça marche.Ça m'a lancé la génération d'un message de commitvia gptel.J'ai écrit une petite fonction qui va envoyertout ce buffer-là.
17:52gptel-commit-message
Fabrice: Ça envoie le buffer en entier à gptel.En lui demandant de m'écrire un message de commit.C'est gptel-commit-message.Je l'ai mis sous GitHub.Il génère un message de commit à partir d'unfichier, d'un buffer diff.Il envoie tout ça et donc il a été générén'importe comment, donc ce n'est pas lié à Magit.Parce qu'il y a beaucoup, il y a plusieurssolutions qui existent déjà dans MELPA et c'estlié souvent à Magit.Ou à Git, en fait, il faut avoir stagé lesfichiers.Moi, je n'ai pas spécialement envie de mettre desfichiers, de les stager dans Git.J'ai envie de pouvoir simplement sélectionnermoi-même ici dans l'interface un, deux, troisfichiers et regarder la différence pour cesfichiers-là.Génère-moi un message pour ces fichiers.Ici, dans le package, qui est tout petit, il faitune fonction.Il y a juste une fonction.Il y a le nom du buffer et la fonction.Derrière, je l’ai mappé sur w. Quand je suis dansle diff-mode-map, dans vc-dir,Je tape w pour write.Ça me gênait le fichier de ce message.Donc, si je reviens sur ton changement, voilà.Donc, j'avais ici, j'avais fait W à partir d'unbuffer diff.Ça a envoyé via gptel, ça a envoyé le diff, ça arécupéré un message de commit, ça l'a mis dans lepresse-papier et ça m'a ouvert un buffer, donc çaa rajouté un buffer à côté où je pouvais le voir.Je vais le refaire ici.Donc ça envoie, voilà.Et donc on voit, ça ouvre un buffer avec lemessage en bas.Et en fait, je peux très bien me dire, tiens, enfait, ce message-là, il ne me plaît pas trop.Puisqu'en fait, on a chaque fois des réponsesdifférentes.Donc je peux très bien refaire W en haut.Il va réenvoyer le truc.Cette fois-ci, ça ressemble assez fort.Je vais le refaire.Il change un petit peu, il n'y a pas beaucoup dechangement ici parce que ce n'est pas le même texte.Donc, remove personal introduction from readme.Prot: Et il a copié aussi sur le kill ring.Fabrice: Oui, exactement.Donc, maintenant, en fait, je n'ai plus qu'àrefaire ctrl .Je suis déjà là-dessus.Je n'ai plus qu'à faire dans V, c'est dire, nextaction, c'est V. Pour dire, fais l'actionsuivante qui a du sens.Ici, puisque j'ai un fichier modifié, l'actionsuivante qui a du sens, c'est de committer.Donc, il m'ouvre un buffer pour pouvoir mettremon message de commit.Donc, je fais simplement C-y.Et éventuellement, je vais un peu le modifier.C-c C-c, c’est envoyé.Et donc, c'est committé.Ça, j'utilise depuis quelques mois quand même, jedirais depuis 4, 5, 6 mois.Depuis cette période-là, je fais tous mesmessages de commit avec gptel, l'intelligence artificielle.Et c'est merveilleux parce que ça a augmenté laqualité de mes messages de commit.C'est indéniable.Avant, je mettais souvent dans mes fichiers à moi « update ».Sacha: Moi aussi.Fabrice: Et en plus, non seulement j'ai un beau messagequi dit vraiment ce qu'il y a, et je relisencore, non seulement avant de committer, jerelis toujours.Au minimum, je fais égal pour voir la différence.Ou s'il y a beaucoup de différences, je vaisfaire un grand E pour pouvoir en Ediff comparerles versions avant et les versions après,systématiquement.Mais en plus, je fais générer mon message decommit et je lis aussi le message de commit.Voilà, pour voir qu'il a bien compris etqu'effectivement aussi, le résumé est correct parrapport à ce que j'ai fait.Parce que parfois, s'il y a eu des centaines dechangements, je pourrais voir aussi quelque chosequi me tient.Il me parle de ça, c'est bizarre qu'il me diseque... Je ne sais pas, que j'ai enlevé unparagraphe, ce n'est pas normal.Voilà, ça pourrait aussi m'attirer l'attentionsur, tiens, est-ce que je n'ai quand même pastrop vite lu le diff et donc d'aller voir destrucs.Donc, c'est vraiment...Ça change la qualité du commit sous Git.C'est vraiment quelque chose que je conseille àtout le monde.Et donc ici, pour vc-dir, il a fallu que j'écrivema fonction moi-même, parce que les autres Gitgptel commits, il faut avoir stagé les fichiers,donc ça regarde ce qu'on a stagé, et ça fait unmessage de commit par rapport à ce qu'on a stagé.Maintenant, on essaie de dire, on ne stage pas.On ne voit pas cette opération-là, ça se faittout seul.Donc on a les fichiers modifiés et puis on faitle commit directement.Donc ça fait le git commit -a, git commit
add…On ne passe pas par cette étape de staging.Et puis voilà, je trouve que c'est plus simple,simplement de sélectionner les fichiers, ceux queje veux, et puis de voir la différence, et puisde committer.Prot: Oui, c'est mieux comme ça.
23:30docstrings
Sacha: Tu utilises également gptel pour générerdes docstrings.Peux-tu montrer ?Fabrice: Alors, on pourrait aller dans un fichier.Par exemple, ce fichier de gptel-commit-message.Donc voilà.Donc, j'ai une seule fonction.
24:03hs-minor-mode - hideshow
Fabrice: Et donc, chez moi, quand je rouvre par défaut,j'utilise aussi H.S. hs-minor-mode, hideshow.Donc, tous les corps des fonctions, etc., il les réduit.J'ai rajouté qui m'indique le nombre de lignespour avoir quand même une petite idée, pas justetrois petits points, mais avoir une petite idéede ce qui se cache en dessous.Et donc après je fais metashifta chez moi, ce quiest visible mode.Donc ça m'ouvre.Il y a visible mode pour les fichiers Org avecles drawers et puis ça fait plusieurs chosesvisible mode et aussi HS show all.Donc ici, voilà, j'ai un message, j'ai unefonction, j'ai un message, un docstring et on vadire que j'ai écrit moi-même un docstring un peuplus bête.Je mets xxx à la place.Et donc, ce que je peux faire, c'est sélectionnertoute la fonction.Oui, les métas, parenthèses, accolades, fréquentes.Ils ne sont pas toujours faciles sur un clavierfrançais, d'ailleurs.Il faut faire le AltGr, tu vois.Le QWERTY est plus facile pour certains trucs.Donc ici, j'ai sélectionné, donc je vais faireC-u C-c RET.Donc je retombe sur le gptel menu.J'ai une directive traduite en français, donc ça,c'est pas très bon.Donc je vais changer la directive, donc je vaisfaire ici le s pour Set System Message.Encore une fois, le s, puisque on pourraitimaginer, si j'avais ça tout le temps, que j'aiwrite the string.Je n'ai pas de preset, de prompt avec ça, donc jevais essayer de le faire à la main.Je vais refaire S pour pouvoir moi-même éditer le message.Et donc ici, je vais taper à la place écrit undocstring correct montrant ce que la fonction fait.exactement.directive est correcte.Alors, je n’ai pas ?? le b, là, otherbuffer, ou je pourrais le mettre dans le kill ring aussi.On va le mettre dans le kill ring, par exemple,donc je vais faire k. Donc j’envoie la réponsedans le kill ring, k. Donc on lit bien, toujours enbas, ce qu'il va faire.Donc ici, quand je vais faire RET, il va envoyerles lignes sélectionnées avec la réponse dans lekill ring.Parfois, ce qui est dommage, c'est qu'on n'a pasun sablier comme des applications Windows.Voilà, ici, j'ai la réponse.C'est dommage qu'il n'y ait pas de sablier.Il y a tellement de choses qui passent surl'echo area qu'on ne voit pas nécessairement ce qui se passe.Je vais remonter ici et on va imaginer, je vaisdirectement effacer ça, faire C-y. Il a faitun peu trop.Alors, ben, voilà, ici en fait, c'est voilà,il faudrait je juste que j'édite un petit peu,voici un docstring correct. Il marque copié.Il faudrait voilà, il faudrait justement faire un prom pour lui direne me donne que le docstring sans répéter le nom de la fonctionet cetera et cetera. Mais en gros, ouais, bah tout çaa l'air bon en fait hein.Ici, il m'avait rajouté du texte et des infos.Après, il explique ce qu'il a fait.Voilà.En gros, ça c'est bon.C'est juste que je ne sais pas pourquoi il a mallié ceci.Sacha: [Je me suis] trompée, peut-être.Tu as une fonction boost-gptel-generate-docstringdans ta configuration.Fabrice: Ah ! Je ne l'ai pas encore testé, celle-là.Justement, avec la section de la fonction, jepense que ça ne marche pas toujours très bien.Il y a des choses qui ne marchent pas encore très bien.Il y a des choses qui sont en test un peu.Ça fait partie des tests.On peut réessayer.Je vais essayer de voir si elle marche.On voit ici qu'il a fait un docstring assez long,d'ailleurs.Un peu trop, à mon goût.En théorie, il doit détecter les bounds de lafonction et envoyer toute la fonction à gptel etavec un prompt.Ça a l'air de marcher assez bien.Sauf que de nouveau, il me fait plus, il merépète toute la fonction.Non, voilà, c'est ça, il me répète toute la fonction.Oui, jusque... Jusque l'interactif.Jusque là, oui, jusque l'interactif.Donc, c'est pas mal.Donc, oui, voilà, ça...Prot: Oui, pas mal.Fabrice: Voilà, hop.On peut voir, effectivement... Donc, ça, c'estnouveau, c'est... Enfin, c'est... Changez,changez, oui.Et donc l'interactive, je n'avais pasd'interactive apparemment.Si je devais l'avoir.Je ne sais pas si j'ai un interactive ou pas,donc je fais C-<f9>.Et on peut voir ce qu'il a modifié.On va le faire en vertical.Hop.Oui, si j'avais un interactif avant.Il considère, il n'est pas aligné au niveau du truc.Forcément, je devais avoir un interactif.Pour terminer ou pas avec tout ce qui estintelligence artificielle, j'ai un petit truc iciqui traîne sur l'écran.Ça fait une semaine que je l'essaye.J'ai déjà payé pour la version si il y avaittrois jours d'essai et ça me paraît très bien.
31:03Super-whisper
Fabrice: Donc c'est Super Whisper.Je vais le faire d'abord dans un Notepad parexemple pour vous montrer.Je l'ai mappé sur la touche Escape, ce qui n'estpas encore la meilleure touche.Je vais expliquer.Je voulais aussi pouvoir l'appeler à partir d'Emacs.J'aimerais bien que ce soit une touche assezfacile, donc un peu une extrémité du clavier.La touche Escape, c'est une extrémité du clavier.Voilà, il n'y en a pas beaucoup, et puis il fautqu'elle ne soit pas utilisée dans Emacs, parcequ'il faut que je la fasse arriver dans Emacs.Et donc ici, quand je suis dans un endroit où onpeut insérer du texte, donc ça pourrait êtreaussi en formulaire sur une page web, ici je suisdans Notepad, ça peut être dans Word, ça peutêtre n'importe où, je vais faire escape, et onvoit à ce moment-là qu'il écoute.Prot: On voit des ondulations là.Sacha: Je ne le vois pas.Prot: Je ne peux pas voir l'indication.Sacha: Parce que tu vas partager seulement lefenêtre Emacs.Fabrice: Je vais stopper la présentation etje vais faire un share.Sacha: Je suis très curieuse à la reconnaissance vocale.Fabrice: Voilà, screen 1.Voilà, ok.Prot: Est-ce que vous le voyez ?Maintenant, oui.Oui, oui.Ok.On va bien danser dans le pad, ici.Fabrice: Il y a une minute, j'ai fait Escape.Qui a lancé, donc, le Super Whisper.Et donc, on voit ici qu'il écoute, on voit despetites ondulations.Il y a une integration dans Notepad, aussi.Je le vois assez discret, d'ailleurs.Sacha: Je pense que parce que tu as aussi, danston séance de réunion virtuelle, lareconnaissance vocale a du mal avec l'audio.Fabrice: Je pense que ça va aller, mais enfait, c'est ça qui est bien.donc W-I-S-P-R.En fait, ça marchait bien, sauf qu'il y avait desconflits avec Emacs.Et que dans Emacs, de temps en temps, j'avaisdes caractères qui étaient générés de manière aléatoiredonc j'ai compris que c'était Wispr parce qu'enj'étais désactivée, quand j'aie désinstallé Wispr,je n'avais plus de problème avec Emacsde config. Et Wispr, lui, il écoutait ce que tu disaiset tu voyais afficher au fur et à mesure les mots.Prot: Peut-être tu peux désactiver Whisper,parce que on ne peut pas écouter bience que tu dit. Peut-être maintenant...Fabrice: Ok, excusez-moi, je n'avais pascompris, vous n'entendiez pas.Je n'avais pas compris, donc jevais réexpliquer.En fait, j'avais utilisé il y a quelques moisWispr, W-I-S-P-R, et cette application-là, elleavait des conflits avec Emacs, donc j'avais descaractères qui généraient dans Emacsaléatoirement à un certain moment.Et quand je l'ai désinstallé, je n'ai plus eu ça,donc je ne sais pas pourquoi, mais il y avait desproblèmes avec Emacs.Et l'application Wispr, elle écrivait les motsau fur et à mesure.SuperWhisper.com, je pense.Elle, elle écoute tout ce qu'on dit, donc je faisEscape pour l'activer, elle écoute tout ce que jedis, et puis quand je refais Escape, donc c'estun toggle, elle écrit tout d'un coup.Mais l'avantage, c'est qu'elle a du contexte.Et donc si je dis par exemple, je lui ai demandécomment ça va, en fait il va recopier, je lui aidemandé deux points, ouvrez les guillemets,comment ça va, point d'interrogation, fermez les guillemets.Parce qu'il interprète en fait, il reconstitue leparagraphe, le texte à la fin.Parfois, si je fais une faute et qu'il y a unautre orthographe, et puis je répète orthographe,il va comprendre que c'est le même mot.Il est assez malin, il remet le texte.de manière plus synthétique.Il met tous les mots, mais à la fin, au niveautypographie, etc., il met deux, trois petits points.Il met parfois des guillemets.Il comprend bien les questions.Je ne sais pas pourquoi ici, j'ai un caractère unpeu spécial.Il y a quand même de temps en temps des fautes aussi.Tout à l'heure, je parlais de aléatoire.Il a marqué aléatoire.Oui, j'espère que j'ai cité tout à l'heureDonc voilà, il fallait marquer W, I, S, P, R. Etpuis une fois, il a marqué c'est A, I, R. Bon, c'estpas parfait, mais en fait, ça marche et c'estbeaucoup plus rapide quand même.Et j'écris une petite fonction qui marche dansWSL, en tout cas pour ma config, de telle façonque dans... Donc si je vais dans un bufferscratch, par exemple.Que dans Emacs, ça fait la même chose aussi quandje fais escape.Parfois, j'ai des problèmes de clipboard, donc lekill ring.Quand je fais le deuxième escape, parfois il meremet un vieux truc qui n'est pas ce que je viensde dire, donc il y a encore des choses àcomprendre.Je vais essayer quand même juste une phrase,je fais ESC…Je sais pas... Ah, voilà. Il se lance. Mais...Il se lance. Est-ce que cela marche ?Ah, tu vois, j'ai un problème.J'ai dit est-ce que cela marche ?Il vient de remettre système, il faut que je comprenne.Il y a des petites configurationsdans Super Whisper à faire pour le presse-papier.C'est un peu plus compliqué avec le presse-papierWindows et le presse-papier Emacs.J'ai encore des petits réglages à faire, maisquand ça marchera bien, ce sera fantastique depouvoir directement aussi dans Emacs utiliser ça.Il y a peut-être d'autres solutions d'ailleurs.Ici, il marche assez bien et le fait qu'il insèrele truc une fois qu'on a fini de parler, ça faitquand même des phrases qui sont plus correctes.On hésite parfois, donc ça reconstitue un truc demeilleure qualité.Sacha: En fait, j'utilise aussi lareconnaissance vocale sur Emacs et j'utilise lesubstitute de Prot pour corriger facilement leserreurs et sauver les remplaçantes.Fabrice: Et ça marche bien, donc ?Sacha: Oui, oui, oui, ça marche.Fabrice: Tu es contente ?Sacha: Oui, je suis très contente.Je l'utilise pour les sous-titres, dicter mon texte.Ça marche bien.Fabrice: Mais il faudrait que je regarde taconfiguration, alors.Pour m'en inspirer, pour faire des tests aussi.Sacha: Si tu rediriges l'output de lareconnaissance vocale à Emacs, tu pourraistraiter avec Emacs Lisp pour faire des remplaçants.Prot: process sentinel, il y a les processus qui...Sacha: Oui, j'ai lancé un serveur de [Speaches] pour offrirun service de la reconnaissance vocale quiutilise aussi Whisper, le mode Whisper.Fabrice: Oui, il y a beaucoup de belleschoses qui vont arriver, c'est sûr.En fait, le problème maintenant, c'est que jen'aime plus quand je suis ici au bureau parce quej'ai des collègues, on est en open space, donc jene peux pas utiliser ça.C'est ça le problème.Quand je suis à la maison, je peux parler, maisquand on est au bureau, on ne peut pas.Sacha: Et dans ce temps, j'utilise avec lesenregistrements que j'ai fait en marche, à pied,et d'autres environnements qui sontProt: Et quand il y a les autres gens, ilpeut comprendre seulement toi ou il écrit toutesles phrases de les autres gens aussi?Fabrice: Je ne sais pas, en fait.Je ne l'utilise pas avec d'autres personnes.Et si je l'utilise, je vais utiliser le casque.Et je pense que le casque est assez... Le microest assez directif.En gros, il n'y a quand même que moi qu'onentend, je pense.Il faudrait vraiment qu'il y ait des bruits assezforts pour que ça puisse passer aussi.
41:21Dotfiler
Sacha: Je veux aussi consacrer du temps pourton outil dotfiler pour gérer les fichiersprofessionnels et les fichiers personnels.Comment tu l'utilises-tu?Fabrice: Oui, alors ça, ce n'est pas de l'Emacs.Par contre, c'est très utile, y compris pour Emacs.Et donc, j'ai écrit ici une petite note là-dessus.Je vais ouvrir l'HTML, par exemple.Donc, là, ici, vous voyez bien l'HTML, oui, OK.Donc, en fait, j'utilise le programme dotfiler,donc c'est son nom, c'est le nom du programme.Il y a un outil qui s'appelait Stow dans le tempsqui fait, je pense, à peu près la même chose.Je ne sais pas très bien les différences entreles deux, d'ailleurs.Mais donc l'idée en fait c'est, j'ai desrepositories, donc j'ai par exemple, et ça mesert notamment pour Org, donc justement j'ai unrepository avec des fichiers Org personnels, j'aiun repository avec des fichiers Orgprofessionnels, et en fait, dotfiler, enfait c'est plus clair ici dans mon explication,donc j'ai un repo A,Et sous le repo A, j'ai un répertoire Org, j'aiun répertoire bin, j'ai un répertoire Lisp.Repo B, je ne suis pas obligé d'avoir les trois.Donc, j'ai une autre structure, mais j'ai parexemple Org, bin, Lisp aussi, examples, peuimporte.Et donc, quand on utilise dotfiler,Il va créer, donc tout ça est mis par défaut dans.dotfiles sous le home directory.C'est là que tous les repos se trouvent, ceux quisont gérés par dotfiler.Et donc j'ai fait des clones, donc au début jevais faire, donc ici si je vais dans mes.dotfiles, c'est tous les repos que j'ai.Il y a des repos privés et des repos publics.Et je vais faire un dot add d'une URL Git.Par exemple, si je veux cloner un nouveaurepository.Et donc, il va mettre sous .dotfiles, il vamettre repo c, par exemple.Donc, si j'ai fait un dot add.Donc, la commande, c'est dot update, dot add.Donc je vais faire un add d'un repo C. Donc lerepo va être cloné localement sous dot files, unrepo C. Et puis je vais faire un dot update.Donc quand l'update, imaginons au départ quej'avais que le repo A, et puis je fais ça, doncj'ajoute un repo B. Comment on va faire ?Donc au départ, quand je n'ai que le repo A, cequ'il fait, c'est que tout ce qu'il trouve endessous de chaque repo,Il les met directement via des symlinks sous lehome directory.Donc le répertoire Org va se retrouver via unsymlink sous le home directory.Le répertoire bin va se retrouver sous le homedirectory.Et pareil pour l'isp.Donc si je n'ai qu'un repo au début d'ailleurs etque je fais un update, dot update, donc il vacréer des symlinks.Un dot update crée ou détruit des symlinks.Ça fait juste ça, en fonction de ce qui a évolué.Et donc la première fois, il va juste faire unsymlink du répertoire tilde slash Org verstilde.dotfiles repo a Org.Et il va faire la même chose avec bin, la mêmechose avec Lisp.Quand j'ai le deuxième repo, le repo b, que j'aicolonné, j'ai fait...dot add de ce repo-là.Puis je fais un update.À ce moment-là, évidemment, il y a deuxrépertoires Org qui doivent être symlinkés.Ils doivent avoir des liens symboliques dans letilde.Donc, il ne peut plus faire ça au niveau durépertoire.Et donc, ce qu'il fait, c'est qu'il supprime lesymlink qu'il avait sur le répertoire Org et ilscanne tous les fichiers qui sont là-dedans et ilva faire un symlink individuel, fichier parfichier.Donc si j'ai 10 fichiers ici, donc fichier 1,fichier 2, fichier 3, il va me faire des symlinksdans tilde Org fichier 1 vers le fichier 1 qui setrouve physiquement là.Et ainsi de suite pour tous les fichiers.Il va faire la même chose avec tous les fichiersqu'il va trouver ici.Et donc tous les fichiers qui se trouvaient dansrepo A Org et dans repo B Org, via des symlinks,se retrouvent en fait dans tilde Org.Et pareil ici dans mon exemple avec bin et avecLisp.Donc si je vais voir ici, si je vais dans monrépertoire bin, vous voyez, donc j'ai, je vaisfaire un peu plus petit, j'en ai des, enfin on neva pas, ici, voilà, celui-ci, il vient deArchibus Reports, de ce repo-là.Celui-ci, il vient de Archibus, OIL, MCD,Datamodel, Diagram.Donc de ce repo-là.Ici, il vient de gitboost.Donc j'avais dans ces différents répertoires,j'avais un sous-répertoire bin.Donc dans ces différents repos, j'avais unsous-répertoire bin.Et tous ces fichiers qui se trouvent directementdans bin, sous les repos,se retrouvent in fine via des symlinks dans monbin à moi, dans mon directory.Ce qui veut dire que dans ma configuration de monshell, j'ai juste à dire path égale tilde slashbin de point $path.Et donc il va connaître tous mes petits scriptsqui sont dans les bins d'un coup.Pour Org,Ça veut dire que, org-agenda-files, je faispointer simplement vers tout ce qui se trouvedans ~/org.Et vu que dans ~/org, je retrouve tous lesfichiers Org du repo A, du repo B, et ainsi desuite, j’ai une seule variable, un seul setq, etj'ai tous mes fichiers qui se retrouventdirectement dedans.Et donc, quelle que soit la machine, si j'aidifférentes machines, une machine pro, unemachine perso.Sur ma machine perso, j'aurai mon repo privé etmon repo pro.Sur la machine professionnelle, j'aurai que lerepo professionnel.En fait, mon fichier Emacs ne change pas.C'est le même fichier.Je dis juste scan tous les fichiers.Mon Emacs, en se lançant, il fait toujours unscan de tout ce qu'il y a. Donc, il va,org-agenda-files va regarder tout ce qu’il y a dans~/org et tout ça est rajouté à mon
org-agenda-files.Et donc, il va, à chaque lancement d'Emacs, il vavoir tous les fichiers qui sont dans laconfiguration.Mais c'est le même fichier Emacs.Pour ça, il ne change pas.C'est le même fichier Emacs d'un côté ou del'autre.Simplement d'aller voir tous les fichiers quisont en ~/org.Et en fait, c'est des symlinks vers les fichiersqui sont dans un repo privé, un repo public, etc.Pareil avec le bin.Donc ça vient de 36 repositories.Pareil avec le Lisp.Justement, et ça me force un peu aussi à faireune structure un peu plus standard.Donc ici, si je vais dans… dans ~/.dotfiles…gptel-commit-message.J'ai un répertoire.J'ai un répertoire Lisp.Et c'est dedans, donc c'est des lisps.J'ai ici le petit package, le petit fichier avecla fonction pour écrire le message de commit.Si je vais dans... Si je remonte...Je vais dans Emacs Leuven Lisp.Là, j'ai tous mes fichiers de configuration.Et tout ça se retrouve en fait dans ~/lisp.Grâce à symlink.Donc si je vais ici dans ~/lisp.Lisp.Voilà.Dotfiler, Emacs gptel, mon fichier deconfiguration, le nouveau avec gptel.Qu’il voit dans ~/lisp, en fait, physiquement,il est dans /home/fni/.dotfiles/emacs-leuven/lisp,il est physiquement là.Celui avec le gptel-commit-message,Physiquement, il est dans un autre repo, il estdans le repo gptel-commit-message, sous lerépertoire Lisp.Donc en fait, en dessous de chaque repository,tous les fichiers qui sont là vont directementtel quel dans le home et tous les répertoires quisont là vont directement tel quel dans le home.Donc en fait, il faut que je vois vraiment tousmes repos comme des puzzles, pièces de puzzle.Et tous les fichiers ou tous les répertoires à laracine de chaque repo vont se retrouver via dessymlinksdans mon ~Directement en tant que fichier.Par exemple, j’ai toujours un README.org danschacun de mes repos.Là, c'est une exception parce que on peut direcelui-là n’essaie pas parce que j’ai README.orgdans chacun de mes repos.Donc, il ne saurait pas créer des symlinks d'unREADME.org dans ~ vers tous les
repositories.Celui-là, c'est une exception.On me dit de ne pas le symlinker.Et tous les autres fichiers se retrouventdirectement dans mon ~ avec des symlinks.Et tous les répertoires, pareil.S'il y a des répertoires qui sont communs, doncj'ai chaque fois plusieurs fois des répertoiresbin dans les repositories, donc à ce moment-là,il fait des symlinks sur les fichiers qui sontdans les répertoires bin.Et donc pour le Lisp, pareil, dans Emacs, j'aijuste dit de scanner tous les fichiers que j'aidans ~ Lisp et de faire un load library.Donc je fais un dolist avec un find, etc.,Et donc en une seule commande, qui reste la mêmesur la machine professionnelle et la machineprivée, j'ai la même commande qui fait scannertout contenu mon ~/lisp et donc dynamiquement,sur une machine, j'aurai potentiellement plus defichiers Lisp ou pas, mais c'est la même commandeet je n'ai pas de config particulier à faire.C'est juste en fonction des repositories que jevais cloner sur telle ou telle machine quej'aurai plus ou moins de fichiers disponibles ettout ça est caché.Sacha: Il y a une question dans le chat.Ces liens sont gérés dans Git ?Fabrice: Non, non.Imaginons... Je peux le montrer en live si tu veux.Je vais aller dans Emacs... Donc, cd.Donc dans .dotfiles, dans emacs-chat-sacha-chua.Donc dans ce répertoire-là, j'ai une image, j'aiquelques fichiers, d'accord ?Et je vais rajouter un fichier.Fichier des mots, .Org.Voilà, donc ici, maintenant, j'ai un fichier deplus dans ce repository.Je pourrais l'ajouter dans Git, etc.Mais je n'ai même pas besoin de faire ça, en fait.Pour les symlinks, je peux... Donc, pourl'instant, ce fichier Org, imaginons que j'ai destâches dedans.Pour l'instant, il sera... Dans ce cas-là,d'ailleurs, je devrais... Enfin, je vais... Jevais créer un répertoire Org et je vais ledéplacer, fichier des mots, temps Org.Prot: Voilà.Fabrice: Donc, ici, dans...Dans ce répertoire-là, j'ai ce fichier de démo eton peut imaginer que j'ai des tâches.J'aimerais bien voir ce fichier-là dans mon Orgagenda files.Et pour l'instant, il n'est pas visible à partirde mon tilde.Il est physiquement là, mais il n'y a pas desymlink dessus.Pour que le symlink soit créé, je vais justefaire .dot files bin.Dans bin, il y a la commande dot.On peut voir son aide.Il y a dot update, dot status, dot add.Add, c'est pour ajouter un nouveau repository.Et update, c'est pour scanner le contenu desrepositories et mettre à jour les liens.Ici, je vais faire dot update.Et alors, je vais faire moins skip pool parce quesinon, si on ne fait pas moins skip pool, il vaen même temps pooler les repos.Donc, on va essayer que tout soit à jour.Donc ici, je vais faire le --skip-pull.Et on va voir qu'il va créer un lien, en fait,Vers ce fichier-là, voilà.Donc ici, il a créé, donc dans mon Org, donc dansmon tilde en fait, ça c'est mon tilde, donc dans~/org, fichier-demo.Org, il l’a lié vers laposition physique du chemin.Donc ça veut dire que maintenant, je peux dansEmacs, ici, donc si je fais C-x C-f, dans Org,donc c'est des fichiers, voilà.Il est directement là.Donc il est vide.Voilà, et donc ici, fichiers des mots, ça pointedans mon type d'org, il pointe vers sa positionphysique.Donc les symlinks, c'est lui qui gère ça, c'estlui qui met à jour en fonction des fichiersajoutés ou des fichiers supprimés.Il garde, il compare, voilà.Il y a tout ça dans les repos.Il y a tous ces symlinks qui existent à partir de~ et il met à jour.Il rajoute ou il retire en fonction.Prot: C'est Git qui garde toujours lefichier original.Fabrice: Et donc dans Git, tu as lesfichiers physiques.Git ne voit pas de symlinks, etc.Dans Git, tu as les fichiers physiques.Ici, si je vais... Imaginons que je mette Hello.Dans Git, je vais faire vc-dir, C-<f9>.C-<f9>.Qu'est-ce qui se passe ?Peut-être parce qu'il n'est pas... Ah oui, c'estparce que justement, je suis sur le Simlink, là,il faut que j'aille dans dotfiles.Donc c'était Emacs, chat, Sacha, voilà.Donc là, je fais C-<f9>.Et donc je vois que j'ai un nouveau fichier à ajouter.Et là, ce n'est pas un Simlink, ce sera lefichier physique.Donc au niveau Git, c'est les fichiers physiquesdans des repos.Simplement, conventionnellement, tous mes reposvont se retrouver en dessous de ~/.dotfiles.C'est là qu'ils scannent tout ce qu'il y a endessous de ~/.dotfiles.C'est là qu'ils scannent tous les repositories etqu'ils regardent le contenu de chaque repository.Si j'avais ton site web ou ta configuration, turajoutes un fichier, je vais faire dot update.Il va les pullerEt puis, si tu as rajouté des fichiers, si tu asfait des modifications, il ne se passe rien.Au niveau des symlinks, il ne se passe rien.Puisque simplement, j'aurai la nouvelle version àjour dans mon clone, dans le sandbox.Mais s'il y a des nouveaux fichiers ou desfichiers qui ont été enlevés, il va rajouter oueffacer des symlinks.Maintenant, si j'efface ce fichier-là, donc undelete, je vais l’effacer de .dotfiles. Emacs.Sacha Chua, le chat.Donc je vais effacer le répertoire ou le fichier ici.Donc hash, yes.Voilà, le fichier n'existe plus dans mon repository.Donc je vais relancer dot update avec le
--skip-pull.Il va montrer rm, voilà.Donc il vient d'effacer le symlink vers tilt Org.C'est génial parce qu'en fait, dans le temps,chaque fois que j'ai rajouté un repository, ilfallait voir où se trouvent les bins, puis faireadd dans le path de tous les bins.Ici, non, tout se retrouve, pour autant que cesoit directement dans le repositorysous ./bin, ça va se retrouver après le symlink dans~/bin.Et donc j'ai qu'à rajouter une fois le fait queTiltbin est dans mon path.Donc ça pourrait marcher pour les manpaths, pourles fichiers d'info, enfin pour les lists, pourl'org, donc c'est extrêmement pratique pour ça.
59:12Enseignement d'Emacs
Sacha: Pendant les dernières minutes,j'aimerais connaître ta point de vue surl'enseignement d'Emacs.Comment tu structures ton cours pour ne pas noyerles gens sur les fonctionnalités?Fabrice: Mon cours, je le vois plus commeétant un cours pour des gens qui connaissent déjàEmacs, donc c'est plutôt pour leur montrer deschoses qu'ils ne connaissent peut-être pas parcequ'ils n'ont pas lu des dizaines ou des centainesou des milliers de fichiers de configurationcomme moi.Donc voilà, c'est des gens qui a prioriconnaissent déjà Emacs, donc ils savent faireenregistrer, ouvrir un fichier,Et simplement, je vais leur montrer des choses enplus, comme Helm.Moi, j'utilise Helm pour tout ce qui est lagestion des fichiers et des buffers.Je vais leur montrer aussi des fonctionnalitésparfois qui sont peu utilisées ou parfois un peucachées, donc peu utilisées comme les macros oule multiple curseur qu'on ne connaît pasnécessairement et qui n'est pas standard, maisqui est facile à installer.ou l'édition rectangle, qui est standard, maisque pendant longtemps, je n'avais pas connu.Et donc voilà, je vais leur montrer des chosescomme ça.A priori, c'est des gens qui connaissent déjà,donc ça va un peu plus vite.Maintenant, j'avais quand même une fois dans unde mes cours, Martin, qui ne me connaissait pasdu tout Emacs.Son père connaissait Emacs, il l'avait amené deforce presque.Si on écoute et qu'on suit, je repassais aussi.En fait, je réexpliquais tout depuis le début.J’avais des slides sur C-a, C-e,
M-n,M-p, C-s, C-r.Je réexpliquais tout plus ou moins rapidement enfonction du niveau des gens, s'ils connaissent déjà.Je vais aller assez vite.En fait, c'est plus facile si les gensconnaissent déjà pour aller plus vite.Sinon, il faudrait plus que deux jours de formation.En fait, j'ai quand même fait quatre jours deformation Emacs.Donc, c'est deux jours sur ce que j'appelle lesfondamentaux.Tout ce qui est édition rectangle, macro,multiple curseur, donc avec des choses en plus.Et puis, deux jours plus orientés sur Org.Il y a tellement de choses dans Org.Il y a les Org agendas, le time blocking quej'utilise pour tout, pour mon temps, pour fairedes factures derrière.Et puis il y a évidemment Babel, Tangling.Donc tout mon fichier de configuration Emacs, ilest dans un fichier qui est tanglé.Un fichier de doc, la code littérée deprogramming de Knuth, où je décris pour un humain.Et puis j'ai des petits bouts de code qui vont se retrouver.dans des fichiers de code où il faut pour être exécuté.Mais l'objectif, c'est de faire un document quisoit lisible par un humain d'abord.Et donc, même en quatre jours, ça passe vite.Il y a trop à montrer.Sacha: C'est aussi le point lié à desincertitudes économiques.Quels sont les avantages du coaching ou del'enseignement humain par rapport à l'IA?Fabrice: Donc, répète les avantages ducoaching par rapport à l'IA.Sacha: Parce qu'il est facile de poser desquestions à l'IA gratuitement, mais je pense quetes formations, ton cours ont des avantages.Fabrice: Absolument.En fait, dans tous les domaines, c'est comme ça.C'est que l'IA, si j'ai une question précise ouune question claire, il va pouvoir me répondretrès facilement et de manière avec plein dedétails, donc extrêmement fantastique.Mais l'IA, le problème, c'est qu'on ne sait pasce qu'on ne sait pas.Et donc, dans la formation, je me doute que pleinde gens, même qui sont des utilisateurs d'Emacsdepuis 20 ans,Plein de gens ne connaissent pas l'éditionrectangle, utilisent peu ou pas les macros, neconnaissent pas Wdired, le Dired mode éditable, par exemple.Si je mets des photos sur mon disque, je veuxmettre la date de la photo dans le nom dufichier, je vais passer en mode direct éditableet je vais faire une petite macro qui va mecopier la date.Et donc voilà, en fait, ChatGPT, si je dis,ChatGPT, explique-moi comment marche l'éditionrectangle, il va l'expliquer.En fait, il ne va jamais répondre à dire, tiens,il y a l'édition rectangle que tu ne connais pas.C'est ça le problème.C'est justement de pouvoir juger ce que les genssavent et de ce qu'on pourrait leur apporter enplus.Et là, l'IA va être moins bonne parce qu'elle n'apas des propositions comme ça de choses que tu neconnais pas.C'est de ce point de vue-là que c'est beaucoupplus pratique en encadrement personnel ou pargroupe où on peut expliquer des choses qu'onpense que les gens ne savent pas et donc ils nevont jamais poser la question à l'IA là-dessus,par définition.Sacha: Merci beaucoup à vous deux, à tout lemonde.Fabrice: Merci beaucoup pour cetteorganisation, cette deuxième séance.Peut-être à une prochaine fois.Merci beaucoup en tout cas pour l'organisation, Sacha.Au revoir.Prot: À la prochaine, au revoir.Fabrice: Salut, je vais vite dans l'autre.
julienlambe78: Bonjour de Belgique 🙂 Sacha, le volume de votre micro est un peu bas. Nous n'entendons pas bien vos questions. Pensez-vous pouvoir augmenter le volume un petit peu?
SaMusz73: synlinks `ln -s` et hardlinks `ln ` vers des fichiers sont gérés dans git ? J'ai cru comprendre que git gére assez mal ces symlinks
Emacs ships with default settings intended for backward compatibility and convenience over strict security. The following settings enforce TLS certificate verification, reduce unintended ffap network lookups, restrict the evaluation of potentially unsafe local file, and directory variables, automated package review system, and protect authentication credentials stored by Emacs.
Network and communication security
By default, Emacs prompts the user interactively if a connection appears untrustworthy. You can additionally require certificate validation to fail at the TLS library level, causing invalid certificates to result in a connection error.
Requiring certificate validation at the TLS layer reduces the possibility of accidentally accepting an invalid certificate in response to an interactive warning.
Define these variables in your early-init.el to ensure the settings are established before startup code initiates network connections:
gnutls-verify-error: Controls GnuTLS certificate verification. Setting this to t makes any certificate validation failure fatal.
tls-checktrust: Controls external TLS binaries. Setting this to t ensures certificate validation is enforced if Emacs falls back to using external tools.
gnutls-min-prime-bits: Defines the minimum acceptable size for Diffie-Hellman key exchange primes. Setting this to 3072 rejects handshakes using primes smaller than 3072 bits.
The benefit of strictly enforcing connection security at the TLS layer is preventing the accidental acceptance of invalid certificates or weak cryptography.
Find file at point network requests
The ffap (Find File At Point) package scans text around point to identify file paths, URLs, and hostnames. This triggers when you invoke an ffap command, most commonly find-file-at-point, which replaces standard C-x C-f if you have (ffap-bindings) enabled, while your cursor is resting on or near a string of text that resembles a domain name.
Setting ffap-machine-p-known to 'reject prevents FFAP from probing hostnames with known domains:
(setq ffap-machine-p-known 'reject)
The tradeoff is that ffap will no longer automatically recognize hostnames as remote connection targets, meaning you must enter remote host paths explicitly.
Encrypting auth sources
Emacs' auth-source uses files such as ~/.authinfo and ~/.netrc to store authentication credentials that other Emacs packages can retrieve when connecting to remote services. For example:
Tramp: Passwords for remote server access.
Gnus and Message: Credentials for mail retrieval (IMAP, POP3) and transmission (SMTP).
An auth-source entry can contain information such as a hostname, username, password, port, or protocol. Because these files may contain sensitive credentials, storing them as plain text exposes them to any process or user that can read the file.
To enforce the exclusive use of an encrypted file and prevent accidental plain-text credential storage, redefine the authentication sources list:
(setq auth-sources '("~/.authinfo.gpg"))
Additional security configurations:
To encrypt the file using specific GPG public keys, define the recipients using auth-source-gpg-encrypt-to:
Emacs can cache passwords to minimize prompt interruptions. The cache expiration can be configured using auth-source-cache-expiry:
(setq auth-source-cache-expiry 3600)
Clear auth-source cache when the user is idle
The auth-source library uses password cache to store authentication data in memory. By default, auth-source-do-cache is enabled (t) and auth-source-cache-expiry is set to 7200 seconds (2 hours). You can restrict this credential exposure using idle timers, targeted cache clearing, or by disabling the cache entirely.
Use an idle timer to purge credentials after a period of inactivity instead of waiting for the absolute cache expiry limit:
(defun my-security-clear-caches ()
"Clear all cached authentication data managed by auth-source."
(when (fboundp 'auth-source-forget-all-cached)
(auth-source-forget-all-cached)))
(run-with-idle-timer900t #'my-security-clear-caches)
The benefit is that it limits memory exposure to 15 minutes of idle time. The tradeoff is that background operations requiring authentication may fail or block while awaiting a passphrase when returning to the editor.
Symbol shorthand code execution
Emacs 28.1 added a feature called symbol shorthands (read-symbol-shorthands). A vulnerability exists in Emacs versions 28.1 through 31 pretest where this feature can be abused to trigger arbitrary code execution simply by opening a specially crafted file. The malicious execution occurs immediately, even before the file contents are displayed.
The exploit relies on Emacs evaluating symbol shorthands during unsafe intern calls in vc-find-backend-function and c-compose-keywords-list. Emacs 31.1 mitigates the vulnerability. Earlier versions, including affected Emacs 31 pretest versions, remain vulnerable unless the mitigation is applied manually.
For users on Emacs 31 pretest or older, you can implement a mitigation by defining an advice function that locally disables symbol shorthands and applying it around the vulnerable operations.
(defun my-suppress-shorthands (orig&rest args)
"Call ORIG function with ARGS while binding `read-symbol-shorthands' to nil.
ORIG is the original function being advised.
ARGS is the list of arguments passed to the original function.
This acts as advice to prevent arbitrary code execution via symbol shorthands
during unsafe operations like interning symbols on file open."
(let (read-symbol-shorthands)
(apply orig args)))
;; A workaround patch (Commit 8466eb44) was applied to the emacs-31 release;; branch on August 5, 2026. Early pretest versions of Emacs 31 do not;; include this mitigation.
(when (< emacs-major-version 32)
(advice-add 'vc-find-backend-function :around #'my-suppress-shorthands)
(with-eval-after-load 'cc-fonts
(advice-add 'c-compose-keywords-list :around #'my-suppress-shorthands)))
This protects the editor from silent code execution attacks embedded in untrusted source files without requiring an upgrade to an unreleased Emacs version. The tradeoff is that it modifies core function behavior via advice, though the impact on version control and C/C++ fontification performance is negligible.
Package Management
Emacs 31.1 added package-review-policy, an automated package review system, allowing inspection of upstream code changes before installation:
(setq package-review-policy t)
Setting package-review-policy to t causes Emacs to require a review of packages before installation or upgrade. The package is unpacked into a temporary review directory, where the source can be compared with the previous installation (diff), and reviewed. The package is installed only if the review succeeds.
The main downside is that updates can no longer run in the background. Having to review diffs by hand gets tedious when upgrading multiple packages at once.
Securing .dir-locals.el and local variables
Emacs automatically applies project-specific configurations through file-local and directory-local (.dir-locals.el) variables when opening a file or directory. While this feature ensures consistent settings across environments, it can cause security risks and persistent prompt fatigue when editing source code. Malicious .dir-locals.el files or file-local variables containing eval forms can execute arbitrary Lisp code if Emacs is configured to evaluate them, or if the user explicitly approves the relevant prompt. This article outlines configurations for securing file-local and .dir-locals.el variables: Emacs .dir-locals.el and Local Variables - Securing and Reducing Prompts.
I do a lot of writing, almost all of it in Emacs so I’m always happy to see how other people use our editor of choice for writing. The latest example I’ve found is from JTR over at The Art Of Not Asking Why. His post considers some simple expansion functions in Emacs.
He starts off with dictionaries. As many of you know, I’ve been a Webster 1913 dictionary user ever since James Somers explained why I (and you) should be. I have gone through many iterations of integrating it into Emacs but have converged on the same solution that JTR uses: the built in dictionary-search function built into Emacs that we both have configured to use the online collection dict.org, although I still have a local copy from my previous iterations.
The dict.org collection is nice because it contains several dictionaries besides Webster’s, and dictionary-search will return entries from all of them that have the word you’re looking for. The best part is that it’s trivial to set up You don’t have to download dictionaries or packages. You simply set a variable saying you want to use dict.org and that’s it. There are some nuances with that so be sure to read JTR’s post or the documentation.
JTR’s post is worth reading just for his dictionary advice but there’s more.
Next he briefly discusses Hippie Expand, a way of completing words based on a list of functions that you can specify. I’ve been using it for a long time, but his list of completion functions was more complete than mine so I stole it.
Finally, he considers ispell. If you’ve been following along, you’ll know that I recently started using Jinx instead of ispell but JTR showed me a couple of things I didn’t know about. I’ve always used ispell with flyspell so I never learned some of its extra tricks. One of those is a sort of fuzzy search for a word you’re not sure how to spell. That happens to me a lot so I was glad to find out about ispell’s ability to handle it.
The other thing you can do with ispell is to use it as a quick way of doing a replace. Take a look at JTR’s post for more on both those functions.
JTR’s post is from last week so Sacha will doubtless cover it before you see this but I had a lot to say about it so I decided to write about it anyway. It’s definitely worth a careful reading.
In late April of this year, I’ve began experiencing a heightened anxiety. I
started therapy then later taking anti-anxiety medication. I worked to change
some of my behaviors. And started writing even more personal journal entries.
I found a list of feelings and responses that resonated with me:
Angry: Lift weights
Stressed: Go for a walk
Procrastinating: Set a 10-minute timer
Sad: Get sunlight
Can’t focus: Clean your workspace
Negative thoughts: Write 3 gratitudes
Stuck: Change your environment
Financial stress: Build an emergency fund
Low energy: Fix your sleep
Overthinking: Journal it out
Lonely: Call someone
No motivation: Start with 2 minutes
Anxiety: Slow your breathing
Brain fog: Drink water and move
Low confidence: Keep small promises
Lost: Define one clear goal
Twice I’ve looked to that list and found the emotions I was feeling and took
action from the responses. The second time I found myself digging back through
my journal to find the entry and pick how I was feeling.
And I realized, that I could encode this as an Emacs📖
function and when I found
myself needing to interrogate my “feels” I could type M-x how-im-feeling and
get a set of responses I could then use to take action.
Here’s the how-im-feeling command and supporting variable:
(defvar how-im-feeling-responses
'(("Angry" . "Lift weights")
("Stressed" . "Go for a walk")
("Procrastinating" . "Set a 10-minute timer")
("Sad" . "Get sunlight")
("Can't focus" . "Clean your workspace")
("Negative thoughts" . "Write 3 gratitudes")
("Stuck" . "Change your environment")
("Financial stress" . "Build an emergency fund")
("Low energy" . "Fix your sleep")
("Overthinking" . "Journal it out")
("Lonely" . "Call someone")
("No motivation" . "Start with 2 minutes")
("Anxiety" . "Slow your breathing")
("Brain fog" . "Drink water and move")
("Low confidence" . "Keep small promises")
("Lost" . "Define one clear goal"))
"How I'm feeling and a simple response to move with that feeling")
(defun how-im-feeling ()
"Prompt for how I'm feeling and respond with what to do."
(interactive)
(let ((feels
(completing-read-multiple "I'm Feeling: "
how-im-feeling-responses
#'completing-read-omit-p
t))
(concatter
(lambda (feel)
(format "- *%s:* %s"
feel
(alist-get feel
how-im-feeling-responses
nil nil #'string=)))))
(insert
(format "I'm feeling:\n\n%s\n"
(mapconcat concatter feels "\n")))))
(defun completing-read-omit-p (thusfar)
"Omit completions THUSFAR given."
(let ((input
(butlast (split-string
(minibuffer-contents-no-properties)
crm-separator)))
(test
(car (last (split-string thusfar crm-separator)))))
(not (member test input))))
Wrapping Up
Now, when I’m feeling the feels, I have another strategy for approaching and
addressing those feels. The prompts don’t provide direct answers but instead
nudge me towards moving with and through those feels. All in service of helping
me cope with anxiety.
I don't play games within Emacs. Instead I enjoy learning about the
many functions and modes it has. I even derived a Tip of the Day
that displays help on a randomly-chosen function. I invoke it at
startup; there's a non-zero probability that I'll spend 15 minutes
or so exploring the function it provides.
I get the impression that I'm not the only one who enjoys Emacs this
way. When I read others' postings, I see references to "customize,"
"configure," "init file," "rabbit hole." I'm pretty sure I also came
across a few posts that contained this: "my ex-spouse made me chose
between him/her and Emacs." Often, reading these posts is enjoyable
in itself.
I enjoy writing and blogging, too. I started my first journal when
I was 13 years old. My answer to "What book would you choose if
stranded on an uninhabited island?1" would be "a blank notebook that
comes with a pencil." I also enjoy interacting with the Emacs
community. It's as if this "Ray on Emacs" blog provides
entertainment squared. This can be expressed mathematically as E2,
where E is entertainment and 2 is two.
While researching this topic, I was surprised at how many games we
can choose from, even in plain vanilla. Naive folks might scoff at
programmers who write games for Emacs. But they don't understand
that games demonstrate Emacs' capabilities. I suspect the
underlying motivation is that writing games for Emacs is fun. It's
a challenge that leads to accomplishment.2
I'm looking forward to reading submissions to Emacs Carnival September 2026 and trying
some games. participate is Humor not to necessary.3
Lexical binding: If you've recently updated to Emacs 31, you might have gotten warnings about lexical-binding in your packages or personal Emacs Lisp configuration. You can check your packages to see if the package authors have updated them. For your personal Emacs Lisp configuration, consider adding
;;; -*- lexical-binding: t -*-
to the top line. If that makes your code stop working correctly when you restart Emacs, set it to nil instead with:
Emacs automatically applies project-specific configurations through file-local and directory-local (.dir-locals.el) variables when opening a file or directory. While this feature ensures consistent settings across environments, it can cause security risks and persistent prompt fatigue when editing source code. Malicious .dir-locals.el files or file-local variables containing eval forms can execute arbitrary Lisp code if Emacs is configured to evaluate them, or if the user approves the relevant prompt by mistake.
This article outlines configurations for securing file-local and directory-local variables while reducing prompts.
Allowing safe local variables
The enable-local-variables variable controls how Emacs handles file-local variables. File-local variables are configurations embedded directly within a file. Here is an example of a file-local variables:
;; Local variables:;; fill-column: 100;; byte-compile-warnings: (not free-vars);; End:
To mitigate security risks and prevent interactive prompts, set enable-local-variables to :safe by adding the following to your init file:
(setq enable-local-variables :safe)
The benefit is that Emacs applies only verified safe values while silently dropping risky ones, reducing prompt fatigue without requiring manual confirmation. The tradeoff is that non-standard but safe project variables will be ignored unless their values are explicitly recognized as safe, for example through safe-local-variable-values or the safe-local-variable property.
Note: Setting enable-local-variables to nil disables normal processing of local variables entirely. (Specific settings, such as lexical-binding, remain active via the permanently-enabled-local-variables list.)
Whitelisting specific local variables
When you set enable-local-variables to :safe, Emacs silently ignores unverified variables. To prevent custom project variables from being ignored, you can explicitly mark specific variable/value pairs as safe using safe-local-variable-values, or define a predicate with the safe-local-variable property:
;; Whitelist a specific variable and value: Add the pair directly to the list.
(add-to-list 'safe-local-variable-values '(my-custom-variable . "expected-value"))
;; Whitelist a variable based on a predicate type: Define a property that;; validates the variable type to accept any matching value.
(put 'my-custom-variable 'safe-local-variable #'stringp)
;; Whitelist a variable with a predicate: Only allow "hello" or "world";; as safe values.
(put 'my-custom-variable 'safe-local-variable
(lambda (val)
"Return t if VAL is 'hello' or 'world'."
(member val '("hello""world"))))
Whitelisting specific directories
The safe-local-variable-directories variable allows you to designate specific paths where all directory-local variables are automatically considered safe. When a directory is added to this list, Emacs trusts every variable defined in its .dir-locals.el file without prompting. Note that this setting applies exclusively to directory-local variables and completely ignores file-local variables.
The enable-local-eval variable controls whether Emacs processes eval forms in file-local and directory-local variable definitions. Unlike ordinary local variable assignments, eval can cause Emacs to evaluate arbitrary Lisp code when processing the local variables.
If your workflow does not require dynamic project configurations, you should disable this feature by setting this variable to nil:
(setq enable-local-eval nil)
The tradeoff is that useful evaluation forms, such as project-specific configurations dynamically generated through eval, will not execute.
Keeping default values for security
Emacs comes with the following defaults for local environment handling. Sticking with these defaults is already a good choice:
enable-dir-local-variables: This enables the use of directory-local variables (.dir-locals.el). Note that Emacs also reads .dir-locals-2.el if present, and non-file buffers like Dired can inherit these settings.
enable-remote-dir-locals: Leaving this as nil prevents Emacs from loading directory-local variables from remote filesystems. This avoids applying potentially untrusted remote .dir-locals.el settings and avoids the additional work required to search for them.
I've loved the plugins that Obsidian has, but those plugins are also it's downfall. I used the Long Form plugin for a while, but then stopped and move that writing to Emacs because the plugin was loosing my data any time I tried to use it. But despite my love for Emacs, I haven't moved my notes over to it and I have no plans on moving my notes over to Emacs.
Currently Obsidian has around 9,000 individual files across my notes and writing. It's saved in a portable markdown format. Obsidian's sync service works flawlessly all the time and the mobile apps are good.
On the Emacs side, every few weeks I have some issue with Syncthing that means I need to sort out some type of sync conflict and because of this I really only add content from my phone via Beorg to my inbox.org file. I don't edit other files if I can help it, the risk is simply to high that I'll have some huge mess to sort out when I get back to my desktop.
The main reason I don't switch is that I don't think that migratingtools is the bigproductivity boost that most people think it is. Yes the Emacs vim key bindings are better than what Obsidian provides. Yes I can very easily hack Emacs to do whatever I want, but Obsidian does everything I need already. So moving my notes to Emacs would most likely be me finding Emacs more fun right now than Obsidian because it's new.
I've wondered about accessing Obsidian from Emacs so I have Emacs goodness in my Obisidian, but even that feels like me searching for a reason to use Emacs where I don't really need it.
Chris Maiorana has a useful post on directory local variables. The idea is that rather than putting a lot of context specific configuration in your init.el file, you can put those configurations in a .dir-locals.el file and they will be set only when you open a file in that directory.
I really like the idea of directory local variables but it has never worked well for me. That’s mainly because I usually wanted to execute some code using the eval keyword. That always causes some problems involving a stack overflow, no matter how simple the code is. Lately, I’ve been using them a lot in conjunction with Jinx, which has the nice feature of adding words to the local file or local directory in addition to the other usual options.
I find that really useful for my blogging. I can add post specific words to the post source file itself or I can add them to all posts in my blog directory. Maiorana uses then to set things like compilation settings, fill and wrap settings, and indentation styles.
If you have different workflows with different setting needs, directory local variables is a very nice way of dealing with them. Of course, not everything should go in a directory local variable. Some things apply to all your buffers and should go in your init.el but for those things that are specific to particular workflows, directory local variables are just what you need.
This article presents a set of Emacs Dired configuration changes to enhance file management. The configurations sort directories before files for easier navigation, hide dotfiles, reduce unnecessary confirmation prompts, prevent UI lockups caused by network latency, among other improvements. Each configuration explains the behavior of the relevant setting, its benefits, and its tradeoffs, making it easier to determine which changes are appropriate.
Keeping Dired clean by hiding dotfiles
Using dired-omit-mode helps hide hidden files and directories, such as .git or .DS_Store, by applying a specific regular expression. The tradeoff is that required configuration files are hidden by default, forcing you to toggle dired-omit-mode off to see them. Disabling dired-omit-verbose prevents the mode from displaying status messages when omitting files.
The following code snippet configures Dired to sort directories before files. The benefit is improved navigation, as folders are grouped at the top of the buffer, matching standard file managers. The tradeoff is a deviation from strict alphabetical sorting, which can disorient users accustomed to raw UNIX or Linux ls output.
Killing the current Dired buffer upon navigating into a different directory
Setting dired-kill-when-opening-new-dired-buffer to t (Emacs >= 28.1) causes Dired to automatically kill the current Dired buffer upon navigating into a different directory. The benefit is a cleaner buffer list, as it prevents leaving behind a trail of open buffers when traversing deep directory hierarchies. The tradeoff is the loss of buffer state. Navigating back to a parent directory requires generating a new buffer, which erases previous cursor positions, active file marks, and modified subdirectory visibility.
(setq dired-kill-when-opening-new-dired-buffer t)
Version control integration
Enabling dired-vc-rename-file causes Dired to perform file renames through the underlying version control system when supported, using the vc-rename-file function. The benefit is accurate version control history during Dired rename operations. The tradeoff is that VC renames can introduce slight latency on very large repositories.
(setq dired-vc-rename-file t)
Simplifying deletion confirmations
This configuration accelerates file cleanup by reducing the number of manual prompts required to delete files and directories.
Setting dired-deletion-confirmer to 'y-or-n-p: Changes the deletion prompt to accept a single 'y' or 'n' keypress instead of requiring you to type the full word 'yes'.
Setting dired-recursive-deletes to 'top: By default, deleting a non-empty directory in Dired can be a tedious process because Emacs will prompt you to confirm the deletion of every single nested subdirectory as it traverses down the tree. Setting this variable to 'top establishes a practical balance between speed and safety. Emacs will ask for your confirmation exactly once for the top-level directory you have marked. Once you confirm that initial prompt, it silently and recursively deletes all nested files and folders inside that tree without any further interruptions.
Setting dired-clean-confirm-killing-deleted-buffers to nil: Suppresses the prompt that asks for permission to close Dired buffers associated with directories you just deleted, closing them automatically instead. This makes sense because once a directory is removed from the disk, any open buffer pointing to it becomes orphaned and unusable. Automatically killing these dead buffers keeps your buffer list clean, as deleting the directory already establishes your intent to remove it from your workflow entirely.
The benefit is faster file management and fewer repetitive keystrokes. The tradeoff is an elevated risk of accidental data loss, as the safety barrier to executing destructive operations is significantly lowered.
Removing disk space indicator
Disabling dired-free-space (Emacs >= 29.1) removes disk space indicator from Dired buffers. The tradeoff is that you lose quick visibility into remaining disk capacity.
(setq dired-free-space nil)
Restricting vertical cursor movement
Setting dired-movement-style to 'bounded-files (Emacs >= 29.1) restricts vertical movement to file lines in the Dired buffer. When point reaches the first or last file line, further vertical movement stops instead of moving to other non-file lines. The benefit is increased navigation efficiency. When scrolling rapidly to the top or bottom of a directory, the cursor stops exactly on the first or last file or directory, avoiding empty space or metadata lines. The tradeoff is that placing the cursor on the directory header to copy its path requires alternative navigation commands, as standard vertical movement keys will no longer reach it.
(setq dired-movement-style 'bounded-files)
Managing recursive copies and destination directories
These variables force Dired to copy directories recursively by default without prompting, while explicitly asking for confirmation before creating new, non-existent destination directories. The benefit is bulk copying combined with a safety check against typos in destination paths. The tradeoff is that large, deeply nested directory structures might be copied unintentionally since the recursive copying happens automatically.
Configuring dired-auto-revert-buffer to use dired-directory-changed-p causes Dired to automatically revert the buffer when the listed directory has changed. The benefit is improved efficiency because Dired does not unconditionally rebuild the listing. The tradeoff is that changes that do not modify the directory itself, such as in-place edits to existing files, might not trigger an automatic refresh.
Note: On Unix-like filesystems, a directory's mtime (modification time) is updated when its directory entries change. This occurs when a file or subdirectory is created, deleted, or renamed within the directory. Modifying the data inside an existing file updates the file's mtime, but does not update the mtime of its parent directory.
Reverting destination buffers after file operations
This configuration automatically updates destination Dired buffers after file operations like copying or renaming, but uses a custom predicate to skip remote directories. The benefit is automatic synchronization of local Dired buffers without performing the same operation on remote directories, avoiding unnecessary TRAMP network activity and associated latency. The tradeoff is that remote directory buffers require a manual refresh to reflect recent file transfers.
Setting dired-mouse-drag-files to t (Emacs >= 29.1) allows you to click and drag files directly from a Dired buffer into external desktop applications, such as graphical file managers or web browsers.
(setq dired-mouse-drag-files t)
Cross-platform file associations
The following snippet configures Dired to open specific file extensions using the default application handler of the host operating system. It identifies the environment (macOS, Linux, or Windows) and assigns the corresponding system command (open, xdg-open, or start) to the dired-guess-shell-alist-user variable for documents, images, and media files. The main benefit is that it delegates file association management to the operating system, removing the need to configure individual applications within Emacs for every file type.
How to use it: To use this configuration in practice, navigate to a directory using Dired and place your cursor over a file. For example, if your cursor is on a video file, press ! (which invokes dired-do-shell-command) or & (for asynchronous execution). Dired will prompt you in the minibuffer with the appropriate system command already populated based on your operating system, such as xdg-open. Press RET to confirm, and the video will open in your operating system's default media player.
If you prefer forcing all file types to open asynchronously via the operating system handler (e.g., xdg-open), the following snippet routes every file through the system default application.
Emacs configurations are like well shuffled card decks: no two are alike. But, as Arlo Guthrie famously sang, that’s “not what I came to tell you about”. Rather, this post is about differing organizations for those Emacs configurations.
The idea was suggested to me by a post from Andros Fenollosa on his own Emacs configuration. Fenollosa says that he doesn’t use any of the all-in-one configurations like Spacemacs or Doom because they include too much and are hard to understand and make changes to. He also doesn’t use the popular literate configuration method using Org mode because it’s just another layer of complexity that can go wrong.
His idea is to to have a separate file for each “area of responsibility” and to use init.el as a sort of index that loads them in order. I like that idea but don’t use separate files. I keep everything in init.el and just organize it in sections.
Still, Fenollosa has a good argument for the separate files but I think I get the same benefits with a single file broken into sections. Actually, it’s not quite true that everything is in a single file. I run Emacs on several type of systems and keep the system/OS specific information in separate files that I load conditionally based on the system and OS type1. That way I don’t have a lot of conditionals dealing with system/OS type in init.el itself.
There aren’t as many ways of organizing your Emacs configuration as there are configurations but there are a lot of them. What principle do you use?
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; Pull in system and platform specific configurations ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; Just keep on going if the requisite file isn't there.
;; Manipulations in second load is for "gnu/linux" → "linux"
(load (car (split-string (system-name) "\\.")) t)
(load (car (reverse (split-string (symbol-name system-type) "/"))) t)
Cet article est inspiré par le Carnaval d'Emacs sur la recherche de connaissances. Merci à Charlie Holland pour son accueil ! C'est une bonne occasion de réfléchir à la gestion de mes notes, qui est un de mes intérêts forts.
Saisir : Je souhaite saisir et publier ce que j'apprends aussitôt que possible parce que ma mémoire n'est pas fiable. Je préfère le format texte brut car il est le plus consultable et le plus archivable à long terme. Je suis loin de mon ordinateur la plupart du temps, donc j'utilise Orgzly Revived sur mon téléphone pour saisir des notes courtes dans ma boîte de réception inbox.org ou mon fichier de brouillons posts.org. Une fois que je suis sur mon ordinateur, je développe mes notes en utilisant Org Mode sur Emacs. J'utilise org-refile pour déplacer des notes vers d'autres fichiers comme organizer.org. J'utilise quelques grands fichiers Org. Pour les tâches répétitives comme mes flux de travail, j'y ajoute des détails autant que possible.
Chercher : Pour chercher mes notes publiques, j'utilise souvent Google. Les notes publiques me permettent de les récupérer facilement. Si je publie mes notes, d'autres personnes peuvent en profiter et les enrichir. Forcément, j'ai aussi des notes personnelles. J'utilise org-refile pour chercher par titres ou consult-ripgrep pour naviguer dans mes notes privées. J'utilise aussi consult-line et isearch si je veux chercher par mots dans le corps. Ils reposent sur des comparaisons exactes, mais si j'essaie des mots similaires, je ne peux pas trouver ce que je cherche. C'est la raison pour laquelle je m'intéresse au paquet p-search et aux plongements de phrases (embeddings) pour la recherche approximative, mais je n'ai pas encore mis en place un bon flux de travail. Je rêve d'un système pour suggérer automatiquement des liens vers mes autres articles, ma configuration et mes notes privées, ce qui peut me rafraîchir la mémoire sur des choses oubliées. Une vraie évaluation doit attendre d'avoir plus de temps libre.
Naviguer : J'utilise C-u org-refile pour naviguer dans mes sous-titres n'importe où dans mes fichiers org-refile-targets. Je relis aussi ma boîte de réception et mes brouillons de temps en temps. J'ai une fonction sacha-blog-edit-org qui ouvre le code source Org à partir d'un lien, même s'il est dans mon posts.org ou seulement dans la copie publiée.
Lier : Beaucoup de mes idées sont inspirées par des articles d'autres personnes, donc quand je trouve un article intéressant sur mon téléphone, je l'envoie à Orgzly Revived pour l'ajouter à ma boîte de réception. Ça me permet d'inclure le lien ou les liens dans l'article une fois que je réussis finalement à l'écrire et à le publier.
J'ai une petite fonction sacha-org-contacts-suggest-mentions qui m'aide à notifier l'auteur de l'article précédent et peut-être d'autres personnes qui sont potentiellement intéressées en utilisant mon fichier people.org, où j'ai spécifié des expressions régulières à comparer au texte.
Pour m'aider à lier l'article aux autres ressources, j'ai des fonctions pour lier :
(Hmm, je peux automatiser les liens vers les autres parties de ma configuration qui définissent les autres fonctions…)
Mes pensées sont souvent déconnectées à cause d'un cerveau qui aime sauter d'un sujet à l'autre. (Bien évidemment, même avec cet article…) En écrivant un article, j'ajoute de nombreuses idées dans ma boîte de réception. À bien y penser, je pourrais ajouter à Org Mode un genre de lien qui résoudra un vrai lien une fois que l'article lié sera publié, comme les WikiWords, ce qui peut m'aider à les connecter. Je sais qu'il y a des paquets qui offrent cette fonctionnalité.
J'ai toujours envie de graphes de connaissances comme celui dans le cerveau de Jerry. Je pense que l'affichage du voisinage autour de l'article actuel est plus utile qu'un aperçu global (comme ceux de ) qui est impressionnant mais un peu trop difficile à utiliser.
J'adore également les grands jardins publics de connaissances comme celui d'Andy Matuschak même si on n'a pas une véritable carte ou un graphe. La richesse des liens m'encourage à les explorer. Je préfère les liens intégrés plutôt que les listes ou les connexions sans explications. Je suis curieuse aussi du projet Anagora, qui essaye de créer un grand réseau de graphes de connaissances personnels et de faciliter le saut de l'un à l'autre par sujet.
Mais je ne veux pas investir trop d'efforts pour le faire moi-même. La majorité de mes articles (sauf le bulletin d'information Emacs News et les revues) contiennent peu de liens. Si je reste au niveau des catégories, il y a trop de points près d'Emacs. Je pourrais convertir les sous-catégories dans mon catalogue en données pour la visualisation… Quand même, le catalogue aurait bien besoin d'une mise à jour.
Il y a d'autres genres de graphes que j'utilise fréquemment. Je dessine souvent pendant que j'écris. Je fais des cartes mentales et des notes dessinées. En fait, c'est plutôt que je jette plusieurs mots sur une page et puis que je les déplace, les connecte, et les range graduellement. Je le fais pour découvrir ce que je veux dire et comment je peux les organiser. De temps en temps, j'inclus des dessins dans les articles ou les publie dans mon carnet de croquis public, et c'est une agréable surprise quand un de ces dessins intéresse d'autres personnes.
J'accumule beaucoup de brouillons dans mon fichier posts.org, que je synchronise avec mon téléphone via Syncthing pour éditer sur Orgzly Revived. De temps en temps, j'utilise une carte arborescente (ou treemap) pour visualiser les tailles des brouillons oubliés (et qui sont peut-être presque finalisés, il leur manque seulement une petite révision ou un ajout). (Hmm, je me demande comment je peux ajouter la taille du sous-arbre aux titres d'Org…)
Figure 1: Une capture d'écran de ma carte arborescente pour mon fichier actuel posts.org
En plus, j'apprécie l'analyse des tendances saisonnières ou annuelles dans ma fréquence de publication (qui me rassure sur le fait que les journées chargées de l'été sont normales et je retrouverai bientôt un peu plus de temps libre):
Tendances mensuelles
import json
import seaborn as sns
import re
from collections import defaultdict
import requests
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
withopen('/home/sacha/proj/static-blog/_site/blog/all/index.json') as jsonfile:
posts= json.load(jsonfile)
jsonfile.close()
monthly_counts= defaultdict(lambda: defaultdict(int))
for post in posts:
title= post.get("title", "")
date_str= post.get("date", "")
if re.search(r'emacs news', title, re.IGNORECASE):
continueif date_str andlen(date_str) >= 7:
year= date_str[:4]
month=int(date_str[5:7])
monthly_counts[year][month] += 1
months_labels= ['Jan', 'Fév', 'Mar', 'Avr', 'Mai', 'Juin', 'Juil', 'Août', 'Sept', 'Oct', 'Nov', 'Déc']
years= ['2022', '2023', '2024', '2025', '2026']
df= pd.DataFrame(index=range(1, 13), data={'Mois': months_labels})
for year in years:
df[year] = [monthly_counts[year][m] for m inrange(1, 13)]
df.iloc[8:, -1] =Nonedf_long= df.melt(id_vars=['Mois'], value_vars=years, var_name='Année', value_name='Compte')
num_old_years=len(years) - 1
gray_shades= np.linspace(0.8, 0.3, num_old_years) # 0.8 is lighter, 0.3 is darkercustom_colors= {year: str(shade) for year, shade inzip(years[:-1], gray_shades)}
custom_colors['2026'] ="#000000"# Force the current year to pure blackwidths= {year: 1 for year in years}
widths['2026'] = 3
plt.figure(figsize=(10, 5))
ax= sns.lineplot(data=df_long, hue='Année', x='Mois', y='Compte', size='Année', palette=custom_colors, sizes=widths, sort=False)
ax.set_xticks(range(12))
ax.set_xticklabels(months_labels)
plt.title("Fréquence mensuelle des articles sur sachachua.com\n(hors Emacs News)", fontsize=12, fontweight='bold')
plt.xlabel("Mois", fontsize=10)
plt.ylabel("Nombre d'articles publiés", fontsize=10)
plt.grid(True, linestyle='--', alpha=0.5)
plt.legend(loc='upper right')
plt.tight_layout()
plt.savefig('frequence.svg')
return df
Je suis plus intéressée par le réseau de savoir collectif plutôt que par mes notes personnelles. Depuis plus de dix ans, je rassemble et catégorise de nombreux liens pour Emacs News, ce qui permet de les parcourir. J'adore rencontrer beaucoup d'idées et de nombreuses personnes en chemin. C'est facile à faire et cela ne demande pas trop de temps, donc j'ai pu continuer ainsi malgré les interruptions de ma vie de maman. En parlant de graphes de connaissances, le bulletin d'information sert à connecter des nœuds et des personnes. De temps en temps, je suis très contente d'entendre qu'un article en inspire un autre, et encore un autre, puis une collaboration… C'est le pouvoir de créer des liens entre les personnes qui apprécient des choses similaires. Si je deviens incapable de faire ce bulletin hebdomadaire, j'espère que quelqu'un le continuera.
Grâce à mon assemblage de liens pour Emacs News, je rencontre souvent des occasions de recommander un article ou une personne en réponse à une question ou un message. Si je peux trouver l'article dans mon fichier d'archive d'Emacs News avec consult-line ou isearch, je peux coller le lien exact. De temps en temps, je veux présenter une personne, ce qui est un peu difficile sur Mastodon parce que les noms de Mastodon sont souvent très différents de leurs vrais noms. Je note les noms de Mastodon dans mon fichier people.org, et j'ai une petite fonction sacha-mastodon-insert-handle-from-contacts pour compléter les noms de Mastodon. J'ai aussi une autre fonction sacha-mastodon-insert-interested-handles qui ajoute des personnes qui peuvent être intéressées sur la base des correspondances d'expressions régulières dans le message, comme quand j'écris mes articles.
En plus des connexions entre des articles et des personnes, je suis également intéressée par les connexions entre des sujets. Avant d'apprendre une chose, quels sujets doivent la précéder ? Quand on apprend une chose, qu'est-ce qui est proche et plus facile à apprendre par la suite ? C'est peut-être utile pour donner des conseils ou des suggestions en tant que coach, ou pour apprendre en mode autodidacte. Les besoins et les chemins diffèrent selon la personne, donc c'est impossible de tracer un parcours qui convient à tout le monde. J'ai commencé à faire une carte des ressources pour les débutants, mais je pense que c'est toujours un peu intimidant, même pour moi. La place naturelle de ces liens est peut-être EmacsWiki pour permettre aux autres personnes de les trouver et de les enrichir, si je trouve une façon de sauvegarder les données et si je résous le problème de l'édition de quelques pages. En plus, si j'étudie les flux de travail et que je les classifie par aspect, je pourrais aider quelqu'un à trouver des ressources qui sont similaires à ses idées.
Je veux aussi identifier des lacunes sur lesquelles me focaliser quand j'ai du temps libre. Il y a beaucoup de ressources pour les débutants, mais grâce à l'immense capacité de personnaliser Emacs, il y a une explosion combinatoire de possibilités au niveau intermédiaire. J'ai hâte d'explorer mes questions et d'autres. C'est difficile de trouver et de naviguer dans les informations, donc si je commence avec des améliorations personnelles ou des questions concrètes de personnes spécifiques, c'est mieux que d'écrire isolément. Je suis sûre qu'en répondant aux questions individuelles, les points communs émergeront.
Je suis aussi fascinée par mes études sur d'autres sujets. J'apprends le français et je m'amuse tellement à bricoler mon environnement Emacs et mon processus d'apprentissage. Un des conseils pour les débutants est de développer des îlots linguistiques : les mots autour d'un sujet passionnant. (Comme cet article-ci que j'écris en français pour me forcer à enrichir mon vocabulaire.) Je me demande comment visualiser ces choses… Par exemple, si je récupère des listes de mots triés par fréquence (peut-être en analysant la base de données lexique), je peux créer une matrice de carrés… Hmm…
Mes obligations de maman diminuent au fur et à mesure que ma fille grandit. J'espère pouvoir consacrer plus de temps à explorer, à saisir et à partager des choses intéressantes.
Being a text editor (among other things), Emacs has quite a few tricks up its sleeve for making writing not just easier but also fun and seemingly magical. I call this section in my config “Wordcraft.”
Emacs Dictionary
When we need a dictionary, Emacs got us covered and then some. After version 28, Emacs connects to a dictionary as a two-step process by default:
It tries to connect to a local host.
If it fails, it next connects to an online dictionary at www.dict.org,
This workflow remains in place as long as dictionary-server is nil, which is the default. It’s a bit confusing (to me anyway), since “nil” might indicate that no dictionary is set up, but here it means we simply didn’t introduce any configuration to it.
The online version at dict.org contains the Webster dictionary from 19131, which (if you follow the footnote) will start sounding like an excellent resource. What I didn’t know up until now however is that dict.org contains more than just this wonderful dictionary.
When you search for something like “US” with Emacs’ built-in dictionary, it checks this database and gives you results from:
The Collaborative International Dictionary of English
WordNet
V.E.R.A. – Virtual Entity of Relevant Acronyms
The Free Online Dictionary of Computing (it has an entry as an ASCII character)
There are others, like the CIA Factbook and the Bible, among others. All of those are at your fingertips, inside Emacs, from a simple quick search2.
The problem is that it’s an online dictionary. No internet, no dictionary. On my MacBook this can happen as I take it with me when I travel. Meanwhile, on Linux, I’m always online (desktop, wired connection).
Follow the footnote, and you will learn how to get that Webster dictionary (but just that, not the database at dict.org) working offline. As for me, I don’t want to set up an offline dictionary because macOS’s dictionary app3 (which works fine offline) is good enough — a quick copy-paste and I’m done. So I configured my dictionary-server to point directly to dict.org, so Emacs doesn’t even search for a local one first.
Here is the setting itself, including the localhost option as a reference and a reminder:
;; (set dictionary-server nil) default, search localhost and then dict.org;; (setq dictionary-server "localhost") local only, no online search (setq dictionary-server "dict.org")
Hippie Expand
I learned about hippie-expand through Mastering Emacs. It’s one of those things that make Emacs magic. It uses different expansion functions (not to be confused with completion functions, which we will learn more about in a minute) to guess which word should come next in our text as we write, depending on which buffer we are in.
Following Mickey’s advice (linked above), I rebound this to M-/, which is bound to dabbrev-expand by default. It’s similar, but not as powerful (essentially it’s more basic; hippie-expand takes the functions that dabbrev-expand uses and builds on top of those with additional ones). Keep pressing M-/ to flip through suggestions.
The catch is that you need to open the buffers you want it to look through. For example, if I want it to pull words from my personal journal while writing this post, I need to have my journal buffer open somewhere. If you’re one of the people who don’t turn off Emacs often, this is not a problem — just keep that in mind when you use it.
Turning it on (by replacing the command, using the same binding):
We also want to change the order in which it searches for things so it works better for us. For me, as a person who does less coding and more writing, I prefer it to look at Lisp-related functions toward the end. The same logic applies to global (all-buffers) lists vs those that operate on the visible buffer only. So our first M-/ will give us suggestions from our current buffer, and as we keep pressing M-/ and exhaust those, it will search all the other buffers we have open. for other suggestions4.
ispell is one of those things I set around the time I started using Emacs and never looked at again. I’ve made a mistake: I bound ispell-region to M-$. Those of you who use ispell often are probably perking an eyebrow: “huh? Why would you do that for? ispell does it by default!”
But somehow I didn’t know this. For years I highlighted whatever I needed to spell check, even single words. The default keybinding attributes M-$ to ispell-word, which checks the word at the marker, but if you have a region selected, it spellchecks the region. I should have given Emacs more credit; today I know better.
Another really handy thing I did not know about ispell is its ability to complete words I’m unsure of.
But before we go on, please note that word completion (and completion functions) in Emacs is a big topic that I just spent a whole week on, and what I’m going to discuss here is just the tip of the iceberg.
First, the useful built-in completion built into ispell itself: l for lookup. Say you forgot how to write “psychiatry”. You type what you know — “psy” — and then call ispell, and as the suggestions show up, press l. ispell will ask you what to complete, along with a wildcard, so add * and then y so you have “psy*y”. ispell will show you what words start with psy and have whatever letters after but end with y. Useful!
Another thing I didn’t know: use R for query replace. This is good when you make the same mistake across several words. For example, when I wrote this post, I originally capitalized every ispell as Ispell. No problem. Stand on one of those, call ispell, press R to replace, and change it to ispell. Now you are prompted if you want to change all of them one by one, or, just hit ! and get it over with in one swoop.
With those quick tips, now back to completion functions.
Years ago, when I started using Emacs, I installed company with its minimal default settings and called it a day. It’s been working OK, and up until I started writing this post, I didn’t realize how much about Emacs completion-at-point functions (capf). Prot has a good video about this, which I’m still chewing through, but basically for ispell the magic happens when you stand on a word you don’t know how to finish and ask Emacs to suggest completions.
To understand how this works, Let’s say we don’t know if water is written as watter or watar, as a simple example.
We write “wate” and we ask Emacs to complete the word by calling completion with ispell with M-Ci. Because the only word that exists (in English anyway) that starts with “wate” is water, Emacs will complete it for us to “water” automatically.
Now, while standing on the word “water,” we call completion again. Emacs informs us: complete, but not unique. In other words: the word water is real and complete (we have it spelled right) but there are other words that start with water.
While still standing on “water”, we call complete yet again. Emacs goes “OK. So you know it’s a word that exists, and you also know there are other words that start with water, so if you’re calling complete again, you probably want to see the other words that start with water.”
Bam, we have a buffer full of words that start with water, like waterboard or watermelon. We can select a word from this menu, even though it’s a bit weird — by going down or up with M-⇡ and M-⇣, then insert the word we want with M-Ret5.
This is nice, but usually you want to complete words that you already know, or kind of know. If I type “psych,” I don’t need a whole buffer of words if I’m looking for “psychology”; I just slowly add letters until I find what I need — and this is exactly what packages like company or corfu6 do.
Harper, languagetool and abbrevs
Let me just remind you of those for now and finish up this post. We will dig into those more next time.
This post from irreal is a good place to get a bit more understanding of how to use a local dictionary, a copy of the online one, and why. If you read it, you may find your way into Irreal’s older post and then this wonderful gem he found: You’re probably using the wrong dictionary. The way these posts are written, I feel it makes more sense to read them in the reverse order I presented here, inviting more exploration as you go — however, if you’re short on time or patience, I recommend you read at least that last one, especially if you enjoy writing. ↩︎
As I was learning this, this next question popped into my mind: can we then look for a definition in Wikipedia and the Urban Dictionary as well? The straight answer: yes (it’s Emacs, duh). The full answer: it’s complicated. We can’t integrate it into the results buffer from dictionary-search directly, but what I can do is have it inside a “wrapper” buffer that will contain the dict.org definition with the Wikipedia and Urban Dictionary below it. For now, I’m fine with just using eww-search-words which initiates a search with Duck Duck Go, which already pulls Wikipedia search results near the top (Wikipedia itself forces JavaScript usage, which blocks Eww) ↩︎
By the way, something fun I discovered while researching: we can call the macOS dictionary using dict:// (as a URL link), and we can build a function in Emacs for it if we want. Further — we could create something like this for offline situations only, but this is not needed at least in my case. Worth mentioning: there’s also the osx-dictionary package, which bridges Emacs and the built-in dictionary app on macOS. But why, when you have a different (and arguably better) version built in? ↩︎
hippie-expand doesn’t really have a help file that lists all the functions, though I think I’ve listed most of those if not all above. Check out the package directly if you want to read more about the functions; they are described in comments at the top of the code. Online, it’s here, or you can navigate your way to the package inside Emacs with C-hkM-/, and then go from there to the package. ↩︎
Turns out you don’t have to do it this way: you can switch to the completion buffer, and then you can move around with arrows, including up and down, and select what you want with Enter like a normal human being. You could also add (setq completion-auto-select t) to your init for this switch to happen automatically whenever you press C-M-i. ↩︎
As mentioned earlier, these packages and their functions go beyond what I’ll write in this post, but a few things are worth mentioning: company is much older then corfu, which came into existence after Emacs implemented completion-at-point functions; as a result, corfu is based on capf, and doesn’t come with anything “extra” out of the box besides a few option to display completion in a dynamic matter. Company, on the other hand, is a much bigger package that has a whole backend to it, complete with a lot of additional functions. I ended up installing and trying corfu, with Emacs’ native functions, and then installed cape (completion at point extensions) makes more functions available in a buffer than you’d usually have. What does that mean? Well, say you’re writing something in org-mode. the cap functions you have available do not include, for example, file names and paths, which you might want; these functions will be available to you when you are in a shell buffer (usually) — it’s Emacs being Emacs and trying to be useful. So with cape, you can ask for that function to be available in org-mode as well. ↩︎
This post is my participation in this month’s Emacs Carnival, on the theme of The Search for Knowledge.
About my current workflow
My way of handling notes and knowledge has evolved a lot over time. I’ve written about my note-taking journey in this blog post. As of today, my knowledge management sauce is composed of Emacs and Vakana.mg. I’ve been using Logseq less and less over the past few months.
Indoor
In my day-to-day computer usage, I have the Emacs client open all day long with a scratch buffer ready for anything. I dump everything into the scratch buffer, remove or add paragraphs as needed and use vundo for time traveling. I use no structure except some dashes as separation between topics. Whether it’s to capture fleeting insights or mold prompts for LLMs, the scratch buffer acts as a buffer between my mind and the computer (notes, prompts, etc.). If something interesting is worth saving, then I save it. I use Denote for that. If I’m not using my computer, I use pen and paper to capture thoughts and ideas, which I extract later on using a VLM if it’s worth saving. I talk about the process in this post.
Outdoor
For external capture I use Vakana.mg, my own offline mobile app. It solves the problem of taking photos, writing notes and commenting on them at the same time. Very practical when I’m in the wild documenting plants. When I’m back home, I can export to an Org file from my phone and send it to my computer using LocalSend.
The Search for Knowledge
Retrieving
I don’t bother too much about retrieving. In Emacs, I either just open Denote or use project-find-regexp to reach whatever I want. Vakana.mg’s search also does fuzzy finding inside note contents.
On knowledge graphs
From what I’ve learned and in my own experience, retention of knowledge works better when the owner defines the visual representation of the information to retain, so it matches the way their mind works, and not the other way around (by letting the patterns emerge). Putting direct intent into the graph’s shape makes the retention of the knowledge stronger. It’s the same mechanism as “charging an image” for better retention in the Memory Palace technique. Vakana.mg’s visual design works better for me because it’s easier to traverse nodes in a linear way. And custom threads representing specific topics make for clean reading and understanding.
Emacs Packages for Knowledge Management
I like Hyperbole and org-roam but I don’t use them at all. I use Denote and plain org files. The simpler, the better for me. Even the information structure is very simple: one file = one topic. The length may vary.
Where do all my notes come from?
I don’t copy-paste much from other sources. Most of what I write comes from my brain. Either by digesting some books or through insights and observations. The exceptions are pictures I take with my phone.
What’s next? What I wish to have
Resurfacing random notes. I write more notes than I can review. A lot of notes are buried forever. In my spare time, I want to be able to go through my old collected notes. I added the feature in Vakana.mg. I don’t have the system in place yet in Emacs.
Whiteboard inside Emacs. Artist mode is really good but it’s not that great for sharing because not everyone is a fan of ASCII art. Especially in a professional context. For that, I rely heavily on Excalidraw for diagrams. Now that we have the Emacs canvas, I really hope someone will soon build something similar to Excalidraw within Emacs.
A couple of days ago I wrote that smarter form targeting was coming to CIDER,
and I ended that post by asking whether I’d got the resolution rules right and
whether anything still surprised people. I got an answer. CIDER 2.1 will ship
with the classic behaviour intact.
This is not a sad story, though. The detour turned up a bug that was quietly
mangling people’s comments, and CIDER came out of it better than it went in.
What I was actually after
The targeting change wasn’t really about cursor positions. What I wanted was
for every CIDER command that operates on a form to behave the same way,
without adding yet another command to get there. CIDER has an enormous surface
of evaluation commands, and every “at point” variant I could have added would
have made that worse. If the existing commands simply resolved the form you
meant, newcomers would have had fewer commands to learn, not more.
That was the bet: consistency by redefinition rather than by addition. In
hindsight it was the wrong bet for a project this old. Fifteen years in, the
existing behaviour isn’t an implementation detail I get to tidy up - it’s
the contract. And as it turned out, the redefinition wasn’t nearly as
transparent as I’d convinced myself it was.
The feedback
Several users, including CIDER’s co-maintainer Sashko Yakushev, voiced
concerns and flagged issues I’d overlooked while playing with this initially.
The most important one: inspection is just another flavour of evaluation, and
people inspect bare symbols constantly. If it got the same treatment, the
disruption would be real enough to run a fork over.
My instinct was that this was an edge case, so I measured it instead of
arguing. Across every cursor position in a buffer, only three rules actually
differ, and the flagship flow - type a form, hit C-x C-e - is identical
under both. Two of those three were fine. The third was this one:
The cursor doesn’t move between those two evaluations. That’s the same
position, twice, and the answers differ - "b" under the classic rules,
"ab" under the new ones.
I’d filed “cursor on a closing paren” as an oddity nobody hits deliberately.
But think about when you land there: you finish typing the last thing inside
a form, and the cursor is now sitting on the ). For someone inspecting
symbols all day, that fires constantly. And the classic answer isn’t
arbitrary either - as Sashko put it, the rule of thumb is “whatever
paredit-backward jumps back to”, which is a better description of the
tradition than anything I’d written down.
Why the tradition exists in the first place
Here’s the part I under-weighted, and it’s worth spelling out for anyone who
finds “the form before the cursor” arbitrary.
Emacs form navigation overwhelmingly leaves the cursor after a form.
C-M-f (forward-sexp) moves over the next form and stops just past its
closing delimiter. C-M-e (end-of-defun) leaves you after the whole
top-level form. C-M-n (forward-list) does the same for the enclosing
list. Paredit’s paredit-forward behaves the same way, and so does typing:
finish a form and the cursor is, by definition, right after it.
So “evaluate the preceding form” isn’t a quirk - it composes with how you
already move around. Navigate forward over a form, evaluate it. Type a form,
evaluate it. The cursor is already in the right place, every time.
Getting onto a form instead takes deliberate effort: C-M-b
(backward-sexp), C-M-a (beginning-of-defun), paredit-backward, or a
jump package like avy. All perfectly good tools, but you have to reach for
them - unless you’re clicking around with a mouse, in which case the cursor
lands wherever you pointed and “the form before the cursor” genuinely is
useless. Which, I suspect, is exactly the workflow difference behind this
whole argument.
What CIDER got instead
The half of the idea that was never controversial is still there, as commands
you opt into rather than a new meaning for keys you already use. Every
operation now has an “at point” variant, not just eval and tap:
cider-inspect-sexp-at-point, cider-pprint-eval-sexp-at-point,
cider-macroexpand-1-at-point, cider-macroexpand-all-at-point,
cider-format-edn-sexp-at-point, cider-insert-sexp-at-point-in-repl.
Three ways to say which form you mean, and now every command supports all of
them:
The at-point commands fall back to the preceding form when there’s nothing to
point at, so they’re drop-in replacements rather than a separate mode of
working. Which means anyone who wanted smart targeting can simply have it:
Two lines, no hidden mode, and everyone else’s fingers keep working. This is
what I should have shipped in the first place, and it’s what the
manual now
recommends. Yes, it’s more commands than I wanted. It’s also the version that
doesn’t break anyone.
Macroexpansion is the one place a bit of cleverness survived on its own
merits, because an expansion needs a call form. Stand on a bare symbol and
cider-macroexpand-1-at-point widens to the call around it, since expanding a
lone symbol is never what anyone meant.
The bug at the bottom of the hole
While unifying the plumbing I found this, which is much worse than anything
form targeting was ever guilty of. Put the cursor at the end of a comment:
(defnfoo[]);; a comment|
CIDER answered comment. Not the (comment ...) form - the word, lifted
out of your prose, because sexp motion has no notion of comments once the
cursor is inside one and happily reads the words as symbols.
For evaluation that produced a puzzling error. For the in-place macroexpansion
commands, which replace the region they resolved, it did this:
;; a comment -> ;; a EXPANDED<comment>(+12); hey -> (+ 1 2) ; EXPANDED<hey>
It rewrote the comment. That bug has been in CIDER for years, and I only found
it because I went looking at the primitives while cleaning up after myself.
Then I checked the neighbours, and this is my favourite part of the whole
episode: SLIME, SLY and Emacs Lisp itself all still do this. Both Lisp
environments use a bare backward-sexp, and if you put the cursor after
(+ 1 2) ; hey in any Emacs Lisp buffer and ask for the preceding sexp, you
get hey. CIDER now steps out of the comment first, which as far as I can
tell makes it the only one of the family that gets this right.1
One idea worth stealing
The same survey turned up something CIDER was missing. SLY briefly flashes the
region it compiled, so you see what it acted on. That’s a direct answer to
the confusion this whole saga was about - “which form did it just take?” -
and it changes nothing about what gets taken.
(setqcider-flash-evaluated-regiont)
Off by default, because I’ve learned my lesson about switching things on for
everyone. But if the targeting rules ever puzzle you, turn it on for a day.
The moral
In the original post I described cider-form-targeting, the escape hatch back
to the classic rules, as “living on borrowed time” - clutter left over from a
plan I’d since reversed, which I was itching to delete.
That option is the only reason the conversation stayed a conversation. Its
existence made the objection “please don’t remove the fallback” rather than
“I’m forking CIDER”, and I very nearly removed it before anyone had tried the
change.
Every mistake is a learning experience for me, and this one was cheap: nothing
had shipped, so the whole thing cost a few days and some rewriting. The
testing and feedback cycle worked exactly as it should have - people tried
something on master, told me plainly what was wrong with it, and the result
is better than either what I proposed or what we had before. Everyone gets
the behaviour they want, and CIDER lost a text-eating bug on the way.
Thanks to everyone who took the time to tell me I was wrong.
That’s all I have for you today. Keep hacking!
If you’re an Emacs maintainer reading this: elisp--preceding-sexp has the same behaviour, and I’d be happy to be told why it’s intentional. ↩
The built-in completion-preview-mode lets us expand what we type in
a minimalist, distraction-free way. It can also be combined with the
default Completions buffer to show all the available options. The
Completions buffer has several nice features, starting with Emacs 31,
which make this an excellent combination that does not require any
third party packages (though packages can still augment the overall
experience). I also mention the Emacs 31 ‘newcomers-presets’ theme,
which configures some of these options as well: it is a great way to
start your Emacs configuration.
Sample configuration
(use-packagecompletion-preview:ensurenil:demandt:bind(:mapcompletion-preview-active-mode-map("M-i".completion-preview-insert-word)("M-n".completion-preview-next-candidate)("M-p".completion-preview-prev-candidate)("M-<return>".completion-preview-insert);; With TAB we effectively defer to the *Completions* buffer to;; show more completion candidates at once.("<tab>".completion-preview-complete)):config(setqcompletion-preview-minimum-symbol-length2)(with-eval-after-load'org(add-to-list'completion-preview-commands#'org-self-insert-command))(global-completion-preview-mode1));; I have a more detailed configuration (and explanation) that works;; with Emacs 31 built-in completion capabilities:;; https://protesilaos.com/codelog/2026-07-29-emacs-default-minibuffer-completion-overview/.(use-packageminibuffer:ensurenil:demandt:bind(:mapcompletion-in-region-mode-map("M-i".minibuffer-choose-completion)("M-n".minibuffer-next-completion)("M-p".minibuffer-previous-completion)):config(setqcompletions-format'one-column)(setqcompletions-max-height12)(setqcompletion-auto-helpt)(setqcompletion-auto-selectnil)(setqminibuffer-visible-completionst)(setqcompletion-eager-updatet))
Finally, the newcomers-presets theme is an excellent way to start
your configuration. It is available as part of Emacs 31. Put this at
the top of your init.el:
As an Emacs configuration grows, startup time can gradually increase. Measuring that increase accurately makes it easier to identify regressions. The majority of Emacs users use emacs-init-time. However, this built-in function does not measure all the work performed during startup.
Emacs startup proceeds through several stages: it loads the early init and regular init files, processes command-line options, sets after-init-time, runs emacs-startup-hook, performs additional initial-frame setup such as applying frame parameters from the configuration, and then runs window-setup-hook after the initial frame parameters have been configured. This makes window-setup-hook a useful place to record a broader startup-time measurement.
For a more accurate startup-time measurement, add the following Emacs Lisp code to early-init.el or init.el:
;; Emacs startup proceeds through several stages: it loads the early init and;; regular init files, processes command-line options, sets after-init-time,;; runs emacs-startup-hook, performs additional initial-frame setup such as;; applying frame parameters from the configuration, and then runs;; window-setup-hook after the initial frame parameters have been configured.;; This makes window-setup-hook a useful place to record a broader startup-time;; measurement.;;;; URL: https://www.jamescherti.com/measuring-emacs-startup-time/
(defvar my-recorded-startup-time-message nil"Stores the formatted string of the Emacs startup metrics.")
(defun my-record-startup-time ()
"Calculate and record the elapsed startup time.
This function records the time when `window-setup-hook' runs."
(setq my-recorded-startup-time-message
(format"Emacs loaded in %.2f seconds (Init time: %.2fs) with %d garbage collections."
(float-time (time-since before-init-time))
(float-time (time-subtract after-init-time before-init-time))
gcs-done))
;; Output to the *Messages* buffer during the initial launch
(message"%s" my-recorded-startup-time-message))
(defun my-display-startup-time ()
"Display the previously recorded Emacs startup time in the echo area."
(interactive)
(if my-recorded-startup-time-message
(message"%s" my-recorded-startup-time-message)
(message"Startup time was not recorded.")))
;; Read startup summary:;; https://www.gnu.org/software/emacs/manual/html_node/elisp/Startup-Summary.html
(add-hook 'window-setup-hook #'my-record-startup-time 99)
This provides a broader measurement than M-x emacs-init-time because it continues measuring after after-init-time has been set, including subsequent startup processing through window-setup-hook.
A depth of 99 places the function near the end of window-setup-hook. This makes the measurement less likely to omit work performed by other functions on the same hook.
The variable gcs-done tracks the total number of garbage collections during the session. Because this function formats and saves the message string during window-setup-hook, it locks in the exact number of garbage collections triggered during the startup phase, giving you a metric of memory allocation bottlenecks during initialization. Ideally, it should be 0 or 1.
Capturing the time the UI finishes rendering provides a reliable metric to track down performance regressions. Once Emacs has finished loading, you can retrieve the recorded startup time at any point during your session. Execute M-x my-display-startup-time to print the exact initialization metrics in the echo area. This interactive command ensures the startup data remains accessible even if the initial message is cleared from the screen by subsequent buffer activity.
I am not going to share my Emacs configuration, but the way I keep it organized. I do not think I am discovering anything new, but it might be useful for someone who is starting to organize their configuration and does not know where to begin.
I do not use any framework (neither Doom nor Spacemacs) nor the literate approach of Org mode. Everything is vanilla Emacs and each file is an area of responsibility, while init.el acts as the index.
Why vanilla and not a framework? Frameworks decide for you. They lend you a huge configuration that you do not understand and that is hard to tweak. For me, the literate approach of Org is one more layer where something can break. I prefer the simplicity of the file I edit being the file that runs. The less magic, the fewer things that break.
So, inside my configuration folder (~/.emacs.d/), I have the following files:
"init.el": the index of my configuration. It loads the rest of the files in order.
"core.el": base Emacs behavior, such as handling backups, paths, clipboard, etc.
"functions.el": my personal scripts and macros.
"packages.el": repository configuration, such as MELPA (among others).
"ui.el": everything related to visual aspects: theme, modeline, typography, visual behavior, etc.
"ide.el": LSP, debugger, linters and programming configurations.
"plugins/init.el": third-party packages, each with its own configuration.
An example of a block you might find inside plugins/init.el:
;; ===
;; vterm-editor
;; Compose text in a full Emacs buffer and send it to vterm with a single keystroke.
;; Docs: https://git.andros.dev/andros/vterm-editor.el
;; ===
(use-package vterm-editor
:ensure t
:vc (:url "https://git.andros.dev/andros/vterm-editor.el")
:after vterm
:bind (:map vterm-mode-map
("C-c e" . vterm-editor-open)))
Each element is isolated in its own context. Using use-package lets me have an isolated and organized configuration block for each package.
I also have other satellite files for third-party packages, such as "plugins/feeds.el" for my RSS subscriptions, "plugins/erc.el" for IRC, "plugins/mu4e-config.el" for email, "plugins/gnus.el", etc. When they grow large and specific, I split them into individual files.
And why break it into areas instead of a single giant init.el? Because I want to know where to look without stopping to think. The brain appreciates each thing having its place, especially when you come back to it after months. Besides, each file is a piece I understand fully, because I wrote it myself. Over time your configuration stops being an editor and becomes a workstation tailored to you. It is my interface with the machine, and it should be as finely tuned as possible.
I am not saying this is the best way, but it makes it easy for me to know where to look and where to put things. Simplicity is also a feature.
Help me keep writing Every coffee gives me a push toward the next article. Sure, it's on me!
I saw an amusing Reddit post recently with an Emacs user thinking about declaring “Emacs bankruptcy” with a config file falling in the 800-900 line range.
Many users in the replies were happy to point out that even 10-12K lines is nothing if you understand what’s going on, and it’s well-documented.
This reminded me of something I’ve been experimenting with lately. Instead of updating my config, I’m trying out directory-local variables per project.
Having local configurations in a project avoids the extra cogitation of worrying about conflicts with other file types or projects down the line. I like this. Also makes settings easier to transfer to a similar project in the future.
The bedrock of my init file has not changed much over the years. I don’t even use a theme for very long before succumbing to the default colors. Most of the time the editing I do in the config involves tweaking wrap and fill settings or adjusting export functions for different writing projects.
I’ve made use of various hooks that trigger slightly different settings for writing and programming modes. But now I’m keeping it local.
Creating a dir-locals.el file right in your project’s working directory lets you set local variables and custom functions that will apply to all files in that directory tree but nowhere else in Emacs.
I’ve found this incredibly helpful for managing compilation settings for different writing projects with slightly different deliverable formats. Instead of configuring multiple export options as needed, I can just edit that build file once per project.
Evaluate on entry – juggle settings with ease
The “dir-locals” file evaluates when you open that project directory. So if you don’t visit that directory today, none of those settings will take effect.
I like to keep to keep my init file pretty lean anyway. I have a habit of throwing things in as a trial and then letting them linger past usefulness.
The nice thing about having these directory-local functions is that they stay with the project forever, and I can tweak and adjust without having to touch my minimalist config.
Over the years I’ve had to oscillate needlessly between hard vs soft line wrapping. In some cases I needed soft wrapping in org documents used for my blog, but preferred hard wrapping in plain .txt files. Likewise, I might prefer to have Olivetti mode ON for a fiction project, but OFF for a technical writing project.
These minor problems no longer vex me. With directory local, I can easily have one project with auto-fill-mode set at 60 characters per line and jump right to a project with visual-line-mode soft-wrapping with no sweat.
Put the right stuff where it needs to go
In most cases the big, unmovable landmasses of your editor configuration will go in your init file. But directory-local is there for settings that should only apply to that particular project.
The separation might look something like this:
Init file
Directory local
Theme, font, style
Compilation/build settings
Modeline
Linters
Editor defaults (auto save, backups)
Fill-column or wrap settings
Org agenda stuff
Indentation styles
I like having this directory-local option in the back pocket so I can focus on one project at a time. Also, I think it’s nice to have a quick and dirty way to try a certain variable on before blasting it out globally.
Update on the hacker novel beta reading call: big BIG THANK YOU to everyone who reached out and offered their time and talent. I have received some great feedback and still waiting on some more. Then it’s off to a final draft run.
Here are some other nice items for you to check out:
HyRolo, the full-text search layer of GNU Hyperbole, is a grep-like retrieval tool for records in your knowledge base instead of lines. Rather than retrieving single lines, it retrieves the full hierarchical record surrounding your match, descendants included.
HyRolo can search your existing knowledge bases in place (org-roam, Denote, Obsidian, Logseq, etc.), even if you use more than one of these. You can point hyrolo-file-list at any container of files (directories and wildcards) containing text in any of the following formats: Org, Markdown, Koutline (Hyperbole's own outline format), or Emacs outline format. Commands like hyrolo-grep will then display every matching record for your search query in a single navigable *HyRolo* buffer, which you can treat like an outline.
HyRolo uses full-text search to find any matching terms across your whole knowledge graph, largely eliminating the need for any graphical views of the connections. Similarly, it eliminates or reduces the need for backlink commands since it finds all references in each search.
In this post we give a better idea of what HyRolo really is and explain why it is a uniquely powerful search and retrieval tool in Emacs. We then demonstrate its use on my own highly fragmented, format-agnostic knowledge base. You will see Hyperbole's philosophy in action throughout this post: HyRolo adheres to the same principles of incremental adoption and in-place data management that make its implicit buttons and HyWiki so useful.
2. About emacshyperbolehyroloknowledgeManagement
This is the third post in my series on Hyperbole. The first post introduced HyWiki, the zero-markup personal wiki, and the second covered implicit buttons, the pattern recognizers that turn your Emacs into a navigable hyperverse.
Those two posts were about traversal, using Hyperbole to navigate the connective tissue that already exists in your free text. This post is about retrieval, the complementary capability of a knowledge/information management system.
I should contextualize my requirements for a retrieval system with an explanation of what my knowledge base looks like:
My notes are highly fragmented. My idea of a 'knowledge base' is more abstract than most, as I don't have a single directory into which I deposit my notes. Instead, I consider any information-bearing piece of text on my file system as a potential component of my overall knowledge base. In that way, my notes are spread across local files, a synced notes directory, Dropbox directories, my blog, and the various READMEs within repositories and projects that I might install on my machine. A knowledge management system must be able to search, edit and retrieve information from all of these sources.
The notes I write myself and the notes I read from others are ubiquitously organized in hierarchical outline format, and oftentimes, the length of a line in these documents is quite small (typically <80 characters) as most writers wrap their text for ergonomics. Oftentimes, the 'thing' I'm searching for appears in an awkward place in these ~80 character chunks. A knowledge management system needs to analyze the associated context here to retrieve the proper records.
My notes are also written in a variety of file formats, including Org, Markdown, and Koutline. Based on the above requirement, I can't control the format in which the 'records' in my notes get stored. I mostly use Org, but most people write README documents in Markdown form, and I share files with folks who like to use Koutline. I need a largely file-format agnostic retrieval mechanism that recognizes the hierarchical structure across all these outline formats.
I don't manually link notes together, with extremely rare exceptions. I mostly write in free text, and I avoid the bookkeeping of connecting notes that other PKMSs (like org-roam, Logseq, Obsidian, etc.) typically prescribe. A good knowledge management system should let me dynamically see the associations I want.
So when I'm searching and retrieving information for a given 'thing' from this multi-format, fragmented expanse of hierarchical notes files, how can I collect everything my system knows about it into one navigable place? This is what I use HyRolo for; it satisfies the search and retrieval requirements of my PKMS-agnostic setup: there is no notes silo, no prescribed file or data format, and no bookkeeping asked of me to enable traversal, or in HyRolo's case, retrieval.
Before HyRolo, I was using consult-ripgrep and embark-export to accomplish some form of record retrieval, but in this Consult+Embark use case the 'records' lack context. They are too fragmented, given the line-matching nature of grep. HyRolo goes one step further and retrieves the tree context around the text I'm searching for, and can retrieve from any location in-place that I specify in my hyrolo-file-list. That's the special trick for me. In this way, HyRolo lets me see the forest for the trees.
3. Point It at Anything hyroloknowledgeManagement
Like HyWiki, HyRolo refuses to lock you into a siloed notes vault or directory.
Retrieval scope is defined by one variable, hyrolo-file-list, which accepts individual files, whole directories, and wildcard patterns:
(setq hyrolo-file-list
'("~/.rolo.org";; the classic: personal contacts
"~/org/";; all standard Org files
"~/org-roam/";; all standard org-roam files
"~/Documents/notes/";; all standard denote files
"~/vault/*.md";; an Obsidian vault, via wildcard
"~/projects/*.kotl"));; Koutliner project files
Although this list seems simple, I hope you recognize HyRolo's two game-changers:
Look at the middle entries: those are the stock locations for org-roam, Denote, and an Obsidian vault, and each of those tools ships its own siloed search over its own directory. You can continue using those if you like, and HyRolo gives you a single, unified retrieval scope across all of them. That is a major deal for me, as I not only use multiple PKMSs, but also want to include the READMEs scattered through my file system in my retrieval scope.
HyRolo does not prescribe a file type. It searches any hierarchically organized text it recognizes, currently the four formats: Org, Markdown1, Hyperbole's own Koutline, and classic Emacs outline files (.otl).
Beyond files, HyRolo can also fold your Google Contacts and BBDB records into the same searches2. There is also a standalone Python-based command-line version, hyrolo.py, that lets you specify both the search term and the paths to search on the command line. It ships with Hyperbole for retrieval outside Emacs3.
I want to belabor the point here because it is really important. Your Obsidian vault, your org-roam directory, a Denote folder, and a stray project README are already valid HyRolo sources, simultaneously, with zero migration or symlinking required to unify them under a search and retrieval interface. Where other tools ask you to migrate your notes into their containers before they will search them, HyRolo adds a retrieval layer over the containers you already have. This is the same philosophy that makes HyWiki a tool for sharing and traversing knowledge, applied here to the search and retrieval use case. There is an 'anywhere, anything' mindset to these tools that makes them unique in the knowledge management space.
4. Records as Trees, Not Lines hyroloretrieval
HyRolo's retrieval features are unique in their treatment of what your query matches.
Think about a plain grep. When grep (or consult-grep, or any of their descendants) finds a match for your query, it outputs the matching line, perhaps with a few lines of mechanical context via -A and -B flags. That context is measured in lines, but lines are not the unit in which you store meaning.
HyRolo instead matches on the entity that does store the meaning, treating your files as sets of hierarchical records when it retrieves them. As the HyRolo documentation mentions:
Any search done on a [HyRolo file] scans the full text of each entry. Whenever an entry matches, it and all of its descendant entries are retrieved.
This Hyperbole manual example makes the consequence clear. Given these entries:
* Company
** Manager
*** Staffer
searching for Company retrieves all listed employees, and searching for Manager turns up all Staffer entries. Context is important, and the hierarchy of data beneath the record is the context. When your query matches data in a record, it displays the record's heading, its body, and its descendants in the *HyRolo* match buffer with each match term highlighted.
Here's a clear distinction between the line-matching and tree-matching approaches. With simple grep (here consult-ripgrep for "mitochondria" followed by embark-collect), we get single line matches for each term.
Figure 1: grep returns matching lines
With HyRolo, we see the full context (the tree) around each match.
Figure 2: hyrolo-grep returns the full hierarchical records around each match
The *HyRolo* match buffer is itself a first-class, navigable outline. You can jump between matches, fold and unfold entries, and get an overview of just the headings. And because this is Hyperbole, the Action Key works in the *HyRolo* buffer. Press M-RET on any retrieved entry and it is displayed for editing in its source file buffer.
5. The Search Commands hyroloretrieval
All of the retrieval commands live on the Rolo menu (C-h h r), or can be called directly:
Command
Key
Finds entries containing…
hyrolo-fgrep
C-h h r s
a string, or a boolean match expression
hyrolo-grep
C-h h r r
a regular expression
hyrolo-word
C-h h r w
whole-word matches only
hyrolo-tags-view
C-h h r t
matching Org tags across your rolo files
The last letter of each key selects the item from the Rolo menu by its label's initial: StringFind, RegexFind, WordFind, and TagFind.
hyrolo-word matches whole words, so searching for product won't drown you in mentions of production. And hyrolo-fgrep accepts full Lisp-like boolean expressions with and, or, xor, and not:
(and postgres (not migration))
The above query with hyrolo-fgrep retrieves every record that mentions postgres but not migration, across every file in your list. This can be a useful query language over your entire note corpus.
6. A Knowledge Graph Without the Graph hyroloknowledgeManagement
This is how I use HyRolo most. This is also how I think about HyRolo abstractly: as a more practical alternative for knowledge-graph-style tasks.
Some quick terminology first: knowledge graphs are graphs in the data-structure sense, and they show up as ball-and-stick models. The balls are the 'nodes' and the sticks are the 'edges'. It's a simple yet powerful model, and most PKMSs use it.
The canonical knowledge-graph question: "show me everything connected to X". What's the answer?
Graph-oriented PKMSs provide this answer with a backlinks pane or a graph view, but only if you did the up-front manual labor of hand-authoring every 'node' as a hard-coded link, inside the tool's own vault. This answer is also quite sparse, usually just a list of node names or file names, stripped of the surrounding prose that actually defines the connection. I think of that surrounding prose as the 'edge type', an indispensable part of the knowledge graph.
hyrolo-grep answers the question with more detail and fewer requirements. Searching for a term retrieves every record it occurs in, across every file in hyrolo-file-list, into one *HyRolo* buffer, with the meaningful surrounding context right there. In this way, I get every piece of the knowledge graph I need (the node is the ad hoc input query, the connected nodes are the records in the *HyRolo* buffer, and the edge type is the surrounding prose in the record).
You can think of the *HyRolo* buffer as a sophisticated (but more practical) backlinks buffer, with the added benefit that you never need to hardcode the nodes or edges, as HyRolo displays these on the fly based on your input query. To emphasize, HyRolo can materialize "backlinks" for anything that can be matched with a regular expression.
This is the Hyperbole philosophy showing up again with HyRolo, and I'm glad to keep highlighting it in this series. Other systems make you author relationships before you can use them, whereas Hyperbole infers relationships when you ask for them. Implicit buttons infer them at the point of traversal, and HyRolo infers them at the point of retrieval. With HyRolo, mentioning a term in a note is the linking.
Figure 3: The *HyRolo* buffer behaves like a knowledge graph
7. Yes, the Name Means Rolodexhyrolo
The name HyRolo is important to address because it can be somewhat misleading.
HyRolo was originally written as a digital Rolodex™ for managing contact records, and it still supports that quite well. The default ~/.rolo.org file, automatic datestamps, alphabetized insertion, and a hyrolo-mail-to command all remain as features from when HyRolo functioned primarily as a digital contact management tool.
But it's important to emphasize here that contact management is just one application of HyRolo. It is a generic tool. The Hyperbole manual makes this clear:
Hyperbole includes HyRolo, a complete, advanced system for convenient management of hierarchical, record-oriented information. Most often this is used for contact management but it can quickly be adapted to most any record-oriented lookup task requiring fast retrieval. For example, you can look up glossary entries listed in Org file headlines or legal-numbered requirements from a Koutline file.
If you replace the word "contact" with "record", you get a better sense of HyRolo's reach. It is a generic hierarchical record retriever for any outline-structured text. A record is simply a heading plus its descendants (body text and recursive subsections).
Consider what instances of data can fit into this flexible type 'record':
Notes
READMEs
Project logs
Architecture Decision Records (ADRs)
Glossary entries
Recipes
And, of course, contacts
These are all naturally hierarchical records. The vast majority of the Emacs community uses org-mode, and so the culture of writing everything down in record form is baked deeply into the philosophy of the modern Emacs user. You may use Markdown over org-mode, and HyRolo supports that as well. In a flexible, generic way, it can retrieve records from any hierarchical format.
Usefully, HyRolo integrates with consult. hyrolo-consult-grep (and hyrolo-consult-fgrep) run your search through consult's live-updating minibuffer, so you can preview matches in place and refine your query before displaying a *HyRolo* buffer. Although the consult buffer highlights one selected candidate, all remaining candidates and their associated records show up in the Hyperbole *HyRolo* match buffer when you press RET.
Figure 4: hyrolo-consult-grep: previewing matches through consult's live minibuffer before committing to a materialized *HyRolo* buffer
For embark users, here is the mental model I'd offer: hyrolo-grep feels like a consult-grep followed by an embark-export, giving you a persistent, navigable buffer of results. The key difference is that HyRolo is aware of hierarchical context. An embark export gives you a grep-mode buffer of matching lines, while HyRolo materializes the full records around the matches, nested structure intact, foldable as an outline, in their original formats. This doesn't obviate embark-export or embark-collect, by the way. I still use both frequently (for example, I use Embark for interactive 'find and replace' via consult-ripgrep -> embark-export -> wgrep-change-to-wgrep-mode).
9. Try It hyperbole
The install snippet from the first post is all the setup you need. Then tell HyRolo where your records live:
Run M-x hyrolo-grep (or explore the Rolo menu with C-h h r ?) and search for a term you know is scattered across your notes (I bet "Emacs" would turn up a ton of matching records!). Fold the results, jump to a source with M-RET, then try a boolean query with hyrolo-fgrep.
10. Further Reading
The HyRolo chapter of the Hyperbole manual covers the concepts, menu, search commands, keys, and settings in full. The first two posts in this series, on HyWiki and implicit buttons, cover the traversal side of the hyperverse that HyRolo's retrieval completes.
Google Contacts are searched on each query when the google-contacts package is loaded (controlled by hyrolo-google-contacts-flag), and BBDB databases are searchable via hyrolo-bbdb-grep and hyrolo-bbdb-fgrep.
hyrolo.py ships in the Hyperbole package directory and provides a command-line version of HyRolo search, useful for scripting retrieval outside of Emacs.
Eglot provides built-in LSP support in modern Emacs, giving developers a native interface for autocompletion, linting, and formatting. When paired with python-lsp-server (pylsp), creating a Python development environment comes down to managing the LSP server configuration properly.
Historically, setting up this toolchain required wiring together a stack of individual utilities such as flake8 and isort. Now, ruff stands entirely apart from that legacy ecosystem by consolidating all of those separate checks into a single, high-performance Rust binary. This article demonstrates how to configure Eglot to detect and use Ruff dynamically when it is present on your system, while retaining a clean fallback to the older stack of tools like Flake8, pycodestyle, and pydocstyle when Ruff is missing.
The configuration strategy
The configuration provided in this article begins by checking for the presence of ruff and flake8 in your system path. Based on these checks:
If Ruff is installed, the configuration relies on the python-lsp-ruff plugin. Ruff consolidates linting, formatting, and import sorting.
If Ruff is missing but Flake8 is present, Flake8 takes over linting duties alongside Pylint.
If neither is installed, the system falls back to the individual pylsp plugins (pyflakes, pycodestyle, mccabe, etc.).
The dependencies can be installed locally pipor directly via your system's package manager.
The complete configuration
Below is the complete Emacs Lisp code to achieve this dynamic configuration. You can place this snippet in one of your init files:
;; Use ruff when it is available because it is fast (written in Rust). When ruff;; is not available, fall back to flake8 and its individual underlying tools.;;;; URL: https://www.jamescherti.com/emacs-python-dev-using-eglot-pylsp-ruff-pylint-flake8/;;;; To sup up:;; - When ruff is available: Ruff, and Pylint.;; - When Ruff is not available: Flake8, isort, and Pylint.;;;; Documentation:;; https://github.com/python-lsp/python-lsp-server/blob/develop/CONFIGURATION.md;; https://github.com/python-lsp/python-lsp-ruff;; https://github.com/chantera/python-lsp-isort
(with-eval-after-load 'eglot
(let* ((has-ruff (executable-find"ruff"))
(has-flake8 (executable-find"flake8")))
;; Target ONLY the 'pylsp key in the global configuration alist safely
(setf (alist-get 'pylsp (default-value 'eglot-workspace-configuration))
`(:pylsp
(:plugins
(;; Plugin: https://github.com/python-lsp/python-lsp-ruff
:ruff (;; Ruff configuration
:enabled ,(if has-ruff t :json-false)
:formatEnabled ,(if has-ruff t :json-false)
;; Add 'W' (pycodestyle warnings), 'UP' (pyupgrade),
;; and 'D' (pydocstyle).
:extendSelect ["W""UP""D"]
;; Ignore specific rules
;; D213: Multi-line docstring summary should start on
;; the second line.
;; D202: No blank lines allowed after function
;; docstring.
;; :ignore ["D213""D202"]
)
;; Pylint remains enabled regardless of whether Ruff
;; or Flake8 is active because it serves
;; complementary role.
:pylint (:enabled t)
;; Flake8 is a wrapper tool that bundles pyflakes,
;; pycodestyle, and mccabe.
:flake8 (:enabled ,(if (and (not has-ruff) has-flake8)
t
:json-false))
;; When Flake8 or Ruff runs, they execute these under
;; the hood. If we enable either, we must explicitly
;; disable the individual pylsp plugins for them,
;; otherwise the language server will run the exact
;; same checks twice and duplicate all editor
;; diagnostics.
:mccabe (:enabled ,(if (or has-ruff has-flake8)
:json-false
t))
:pyflakes (;; pyflakes catches logical errors
;; (unused imports, undefined names...)
:enabled ,(if (or has-ruff has-flake8)
:json-false
t))
:pycodestyle (;; pycodestyle catches style/formatting
;; violations (PEP 8)
:enabled ,(if (or has-ruff has-flake8)
:json-false
t)
;; Ignore specific rules
;; :ignore ["W293"]
)
:pydocstyle (;; pydocstyle enforces PEP 257 docstring
;; conventions
:enabled ,(if (or has-ruff has-flake8)
;; Use flake8-docstrings
;; https://github.com/pycqa/flake8-docstrings
:json-false
t)
;; Ignore specific rules
;; D213 Multi-line docstring summary should start on
;; the second line.
;; D202 No blank lines allowed after function
;; docstring.
;; :ignore ["D213""D202"]
)
;; Formatting: isort
;; https://github.com/chantera/python-lsp-isort
:isort (:enabled ,(if has-ruff :json-false t))
;; Formatting: autopep8
:autopep8 (:enabled ,(if has-ruff :json-false t))
:yapf (:enabled :json-false)
;; Code completion
:jedi_completion (;; jedi configuration
:enabled t
;; Disable resolving documentation details eagerly
;; :eager t
;; Add class objects as a separate completion item
;; :include_class_objects t
;; Add function objects as a separate completion item
;; :include_function_objects t
;; Auto-complete methods and classes for each parameter
;; :include_params t
;; Fuzzy matching for typos/abbreviations
;; :fuzzy t
;; Modules for which labels and snippets should be cached.
;; :cache_for ["pandas", "numpy", "tensorflow", "matplotlib"]
;; How many labels and snippets should be resolved?
;; :resolve_at_most 25
)))))))
This Eglot configuration prioritizes Ruff to handle linting and formatting when it is available in the environment. If Ruff is missing, it falls back to Flake8 and the default pylsp plugins. This keeps Emacs Eglot adaptable across different machines, guaranteeing consistent diagnostics and autocompletion without demanding a strict set of dependencies on every system you use.
For me, one of the most difficult Emacs behaviors—or at least “normal” Emacs behaviors—to understand is scrolling. The default behavior is so unintuitive and jarring that it kept me from embracing Emacs for a long time. It was only after I took the plunge that I discovered that that behavior, like everything else in Emacs, is configurable.
That was almost 20 years ago when I was just starting and knew next to nothing about Emacs configuration but I was able find the solution on the Internet. Here’s what I’ve had in my init.el almost from the beginning:
Back then, I had no idea what any of it meant, only that it made Emacs scrolling rational by my standards. It’s been sitting there undisturbed ever since.
Now James Cherti has a really excellent post that expains all the fine points of configuring scrolling. He explains the above lines, which most people probably have in their configuration. If you want Emacs scrolling to behave as it does in every other application, you can leave them as is but you can tweak the behavior a bit if you like. Cherti has all the details.
But wait. There’s more. There are similar settings for horizontal scrolling that you might like. I’ve never had any problem with horizontal scrolling—probably because I’ve always wrapped lines in one way or another—but they’re there if you need them.
There are some other tweaks that you can apply to delay fontification during scrolling to speed things up a bit. Again, this is not something that I use but if you deal with large files, you may find it useful.
In Emacs 29 and later you can set pixel-precise scrolling to make scrolling over images more pleasant. There are a few other tweaks in Cherti’s post so you should definitely take a look at it. There’s a lot of good content and even if you use only a bit of it, you will find his post worthwhile.
This is now fixed: GCC drive error on macOS: This seems to show up with almost every new Emacs package I’ve been trying recently on my Mac. I remember trying to dig into it, and I went in circles trying to install something from Homebrew. Has anyone fixed this? I’m on GNU Emacs plus 30.2.
Emacs produces a backup of each file by default. It does this by
appending a tilde to the file's name. For example, Myfile.txt is backed up to
Myfile.txt~. It's a reasonable strategy, assuming that
a Version Control system is in place.
But in the absence of version control, you're limited to restoring
just the previous version; older versions need to be restored from
backup.
I use git for coding projects that I share to the public. But I
didn't create a repository for my home directory. So when
C-c a m TAG1 generated an error instead of a list of
TAGged items, I hoped fervently that the latest init file change was
to blame.
After finding and fixing the problem quickly, (and realizing I was
lucky this time), I resolved to be more diligent about disaster
recovery. I looked into what backup schemes Emacs already
provides.
Emacs can create unique backups for a file rather than overwriting
the previous backup. When Emacs sees a numbered backup for a given
file, it will make a new, sequentially-named file for each backup of
that file, while continuing to use the "append tilde" method for
other files.
Once I renamed (or copied) .emacs~
to .emacs.~1~, Emacs created backups
named .emacs.~2~, .emacs.~3~ (and so on).
I didn't even need to change a variable -- when Emacs
detected .emacs.~1~, it knew that backups
of .emacs needed to be numeric, so it acted
accordingly. Emacs continues to use bills.org~ as the backup
for bills.org. Until I tell it otherwise.
The topic of this month's Emacs Carnival interests me, even though I
"dump ... notes into a directory and simply grep for information
...."0 Org does not work the way my brain
works,1 or rather, my brain doesn't work the way Org works.
I do use Org. At first I created a few Org files (work.org
and home.org, and maybe todo.org) thinking I could rigidly dedicate
all my topics to them.
Over time I had about dozen files, one for nearly every major facet of my
life until I gave up. Why? Because it really doesn't matter where I
put Org TODOs and their notes. When I need to locate something, Org
provides a way to search entries for a regular expression, which is
what grep does, sans all that Unixy CLI stuff.
So faced with the task of paying one of my wife's medical bills, I
could agonize over whether to put the task into wife.org or
bills.org or medical.org.2 Or I
could just:
establish the task somewhere,
act on it
link to a receipt
mark it done3
I created a special template for org-capture, which places these
one-off tasks into todo.org. And it clocks into the task, too.
A topic with even greater complexity is "education." I'm a lifelong
learner; my edu.org file has (only) 29 headings, all of
which have at least a few subheadings. If I learn about an Emacs
feature, I might want to write about it, so I'll put a task
into blog.org that links to it.
Occasionally a post will span multiple topics in edu.org,
topics such as Emacs, Vivaldi, LibreOffice, HTML or LaTeX. In a year
from now, which file should I look for it in? The keystroke C-c
a s comes to the rescue.4 I might search for "Sacha"
because all I remember is that I discovered an Emacs feature on
Sacha's website.5 Incidentally, I get 13 hits when I
search for "Sacha," one of which was the one I had in mind.
The question I think all Emacs users face is, "At what point does
tweaking Emacs (and self-quantifying tools in general) tip the scales from
aiding our progress to impeding it?" I dread going down
a Knowledge Management rabbit hole,6 but I'm sure I'll enjoy
the leap.
6 Can one gain knowledge of Knowledge Management Systems
without having a Knowledge Management System in which to record that
knowledge? That may be a Zen koan for someone.
I installed Donovan Ratefison's brand-new Vakana app1 on my Android
device this morning. After following the tutorial, I exported the
thread as an Org-mode-compatible package in a ZIP and emailed it to
myself in order to transfer it to my desktop.
After opening the Org file that Vakana generated, I exported the
entire buffer to HTML. The result is shown below. The only change
I wanted to make was to scale the bead of each heading 25%.2
It's fortuitous that Donovan Ratefison released Vakana in a month
whose Emacs Carnival theme is "The Search for Knowledge," because
Vakana provides a unique way to manage one's knowledge. I'll need
to become proficient in it before adding it to my workflow. So my
eventual Emacs Carnival submission will be more pedestrian.
2 The images used for beads are SVG files. However this
blogging platform does not support that file type, so I converted
them to GIF for this post. This should explain why they don't look awesome.
Your first thread
Your first thread
A 90-second tour. Walk it bead by bead, then make it yours — delete it, or reload it anytime from Settings → Pro Tips.
Welcome — and if this is all new, that’s perfectly fine. We’ll go gently.
Vakana is the Malagasy word for beads — there’s no other word for them in the language, and it’s where this app takes its name. The idea is old and simple: you turn a moment into a small bead — a little coloured token — and beads you keep together form a thread you can walk back along, like beads on a string. (These six notes are already a thread — “Your first thread.”)
That’s the whole shape of it. What a bead means is always yours — the app never decides that for you.
This is a short, hands-on tour: five steps, each one a thing you try yourself.
Next → 1/5 · Tap the bead.
3. 1/5 · Tap the bead
[2026-08-08 Sat]
,*Tap the bead* at the top of this note — the composer opens. That’s where you pick a bead’s shape, pattern, mood-colour and a tiny glyph (this one’s is the number 1️⃣).
Eight shapes, eight patterns, eight moods — small enough to learn, wide enough for anything. There’s no legend: a circle isn’t “an encounter” unless you decide it is.
How does the user navigate the beads?
Now I see "< Prev" and "Next >" so it's obvious.
Next → 2/5 · Open the thread.
4. 2/5 · Open the thread
[2026-08-07 Fri]
You’re on a thread right now — a chain of beads strung together on purpose. Open “Your first thread” (tap its chip here, or find it in the Threads view) and you’ll see all six beads in a row you can walk, bead by bead.
The app also keeps a quiet thread for every day you write, under Days in the Threads view — built for you, no effort.
Next → 3/5 · Comment.
5. 3/5 · Comment — swipe me right
[2026-08-06 Thu]
Some thoughts arrive after the moment. Instead of rewriting a note, leave a comment in its margin.
Try it: tap the 💬 at the top of this note — or, back in your list, swipe this row to the right. Either opens the margin, where you can jot a thought or even set it to remind you later.
(The next bead, 4️⃣, already has a comment — spot the 💬 on its row.)
Next → 4/5 · Long-press the bead.
5.1. Comments
5.1.1. Comment — [2026-08-09 Sun 18:50]
The tutorial shows an icon for comment as a white-filled rectangular balloon with three dots. However the actual icon is a balloon that's an empty circle
6. 4/5 · Long-press a bead in the list
[2026-08-05 Wed]
Beads repeat, and you’ll want to find a bead’s kin. This is a thing you do from your list, not from inside a note — so head back to the list first.
There, press and hold any bead: the list instantly filters to every bead like it. (Long-press is the move most people miss.)
Want finer control? Tap the funnel at the top of the list to filter by shape, mood, pattern or glyph, then gather the results into a brand-new thread.
Next → 5/5 · Make it yours.
6.1. Comments
6.1.1. Comment — [2026-08-09 Sun 18:22]
I keep coming back to this one — long-pressing a bead is the move most people miss.
6.1.2. Comment — [2026-08-09 Sun 19:14]
The alarm clock icon next to the note field is for a reminder, not a timestamp. It puts it on the next business day at 9:00 a.m., perhaps. I pressed it at 3:00 p.m. on Sunday August 9th, and then entered the reminder for the next day at 9:00 a.m..
7. 5/5 · Close it, then make it yours
[2026-08-04 Tue]
This thread is closed — notice the clasp. Closing marks a loop that finished; you can reopen it anytime.
A finished thread can leave the app: open it and Export to turn it into a web page or a slide deck.
That’s the whole grammar — bead, thread, comment, close, export. Now make it yours:
• Delete this tour — swipe any note left, or remove it in one tap from Settings → Pro Tips → Interactive tutorial.
• Want it back later? Reload from the same place.
Now write your first real moment, and bead it however you like.
Before I start writing in Org, I dedicate the window. This "protects"
the content, preventing it from being replaced by something else. It
ensures I can stay focused on my writing no matter what other buffers
Emacs opens.
My usual workflow is to display help on the functions I write about.
Emacs opens the help buffer in another window. That's good. But then
if I click on the link to the source code that help references,
that file will open in my writing window, and I lose focus. Dedicating
the window prevents this.
I could dedicate a window manually by invoking
toggle-window-dedicated (bound to C-x w d)
every time I start writing. But because I clock in to anything I'm
serious about, I let org-clock-in-hook to do it for me as
shown below. Note that the hook calls set-window-dedicated-p rather than
toggle-window-dedicated because the latter will
un-dedicate a window that's already dedicated. I could
configure org-clock-out-hook to un-dedicate the window;
I'm debating whether it's necessary.
Interestingly, while I was writing this, JTR published his
"Gems" article in which he writes about setting
help-window-keep-selected to true "to keep help in its
own dedicated window, so it won’t open in a separate window once we
follow a link."1 Perhaps that’s his approach to the same issue.
HTML enforces the use of a single space to separate sentences. If
you want to add another space, you need to use
. I still remember how annoyed I was about
that. Back in the century in which I took my technical writing
course, two spaces were required. Of course, we weren't using
computers very often; hand-written sentences needed to appear more
separated than the words within.
I'm definitely not here to convince anyone to switch. Instead, I
just want to share that I paid close attention to this detail when I
was formulating my style guide for this blog. Should I continue with
my out-of-date two-space habit? Or should I adopt the current
convention?
I turned to Emacs itself for guidance. It turns out that the
individuals who wrote Emacs comments, docstrings and help text used
two spaces. Although many times they started each sentence on a new
line.
It turns out that the content here mostly is one-space. I start
out using two spaces in Org where most of this originates. But when
I export to HTML, that second space vaporizes. That would seem like a bug to me.
The only place where you'll see evidence of my two-space style is in the comments
and docstrings of my source code.
Am I the only person who nit picks about such details?
If you've ever written an Emacs function only to find out that it
was there all along, masquerading as something else, this post is
for you.
I just finished writing an Emacs function that copies the link at
point to the kill ring in EWW. It will remain unpublished. Why?
Because I discovered that shr-maybe-probe-and-copy-url
(bound to u and w
in eww-mode) does the same thing. Just position point
on a link and invoke it.
It's not obvious that pressing w will do this. In
fact the major mode help for eww-mode suggests
that w will not copy the link because it's
bound to eww-copy-page-url. But it turns out that if
point is on a link, w will
invoke shr-maybe-probe-and-copy-url, "which copies this
link's URL to the kill ring." The manual goes on to say, "If point is not
on a link, pressing w
calls eww-copy-page-url, which will copy the current
page's URL to the kill ring instead."1
Before writing my function I had been looking through the mode help
(C-h m from within a EWW buffer) and the EWW source code for any
function that looked like copy-link-at-point.
So how did I find out about this? I wanted to ensure that I was
following the standard for capitalizing EWW, so I searched the web.
This took me to the GNU Emacs manual, which has a section on EWW and
shows it in all caps. But then my distractibility encouraged me to
read a great deal more.
I've known that keybindings can change depending on the major mode.
But it seems a keybinding can change based on context
(i.e. the position of point). This idea is new to me. Maybe it's
common for a function to "hand off" to another based on context.
But in this case it seems irresponsible to omit documentation for
the alternate behavior of a w key press. Suppose a
user expects to copy the page URL and gets the URL of a link, instead?
While this was a frustrating experience, it is yet another reminder
to RTFM. WDYT?
If there were an underappreciated web browser feature, it would be
this: history.
On many browsers, history is invoked with C-h
(Ctrl+h). Vivaldi shows history as a calendar. The user
can choose from time intervals of Day, Week or Month and search for
a keyword within that time interval.1
I tend to write about a topic after I've forgotten where I read
about it. Suppose I recall reading about "nslookup" but can't
remember when or where. Simple. Vivaldi remembered that I
visited a page with "nslookup" on Windows Central on July 3 at
21:26. Being able to find such references helps.
History helped me track my hours when my employer required
bi-weekly timesheets. I would combine browser history with Org's
clock report to get a result I was actually proud to submit.
I still track my time with Org, which stores start and stop times
in a drawer called LOGBOOK. But I tend to forget to start or stop
the clock, so the times are unreliable. Sometimes a file's
timestamp can help me reconstruct the LOGBOOK. But at times like
these I'd wish that Emacs had a history feature.
Actually Emacs does have a few history-like features. One example
is savehist-mode,2 which "save[s] the values of
minibuffer history variables." Unfortunately, timestamps are not
included.
So I configured Emacs to write a timestamp to the *Messages* buffer
every time it loads or saves a file. This frees me from switching
to a file manager to check a file's timestamp. Using Customize, I
added functions to find-file-hook
and after-save-hook:3
Then I clock in and change the start time to 12:33.4
Similarly, if I've been away from the computer and realize I'm still
clocked in, I'll look for the last occurrence of the message, clock
out, and change the time accordingly.
The *Messages* buffer is temporary; timestamps from an
earlier Emacs session are lost. But this simple system does what I need.
What methods do you use to track project time?
1 The user also can view history as a simple list in
reverse chronological order. And the search can span the entire
history, if desired.
3 Note that these expressions are arguments to
custom-set-variables.
4 Timesheets required time entries in units of hours
rounded to one decimal. A tenth hour is 6 minutes. So I configured
Org to round clock times to 3 minutes by setting
org-clock-rounding-minutes to 3. This affords more granularity.
With several clock entries per heading, an even number of 0:03 clock entries would
produce the desired tenth hour accuracy.
I've been exploring Jasspa MicroEmacs lately - a lightweight Emacs like editor with a surprisingly thoughtful set of defaults, its footprint is surprisingly small, 1-2MB on your disk, useful for a portable Emacs or in my case allows me to run up an Emacs like editor on my work laptop (as it blocks GNU Emacs). No installation, just a single very small executable and it comes in many different varieties, windows, linux and GUI or terminal based.
For those unfamiliar with the lineage, MicroEmacs originally dates back to 1985 when Dave Conroy built a tiny, portable Emacs clone, later maintained for decades by Daniel M. Lawrence (and famously still used by Linus Torvalds today!). Around 1988, Jon Green and Steven Phillips forked MicroEmacs 3.8 to build Jasspa, it uses a subset of full emacs command set and it can emulate the keybindings from Microemacs, GNU Emacs, and even Windows CUA keybindings. It's a much enhanced version of the Daniel Lawrence's original MicroEmacs 3.8 of 1988 and they have grown it into a remarkably feature-packed environment with syntax highlighting, windowing, and a macro language and its still regularly updated!
One thing that immediately caught my eye in Jasspa was its file manager (a dired equivalent that, while not a direct clone of GNU Dired is surprisingly functional). Specifically: single numeric keystrokes to instantly re-sort directory listings by size, time, name, or extension.
Back in GNU Emacs, I usually reach for s in Dired to cycle the sort, or C-u s to manually type ls switches. It works, but it's the kind of subtle friction you don't notice until you've tried a direct alternative. Once I'd spent an afternoon in Jasspa, the muscle memory was already forming:
3 for size when cleaning up disk-hungry directories
4 for date when looking for recent modifications
5 to reset back to alphabetical
6 to group by extension
Bringing this to GNU Emacs took less than thirty lines of Elisp - four small wrapper functions around dired-sort-other and a few keybindings:
The Emacs commands that create a new frame disappoint me. They
create a frame that shares the same set of buffers as the main
frame. I don't think a Tip Of The Day (TOTD) popup should have
access to all the buffers in a session. I want a popup to display
some help and be easily dismissed.1
That's why I designed my Tip Of The Day to run in a
separate Emacs instance. Instead of invoking the TOTD function
directly, Emacs asks the operating system to launch a second Emacs
process. One of the command line switches loads the Lisp file in
which the TOTD function is defined, and another runs the function.
This is achieved by invoking call-process-shell-command
to run a one-line batch file2 that contains this:
The -l switch loads totd.el, defining
both totd and ztotd. The
--eval switch then invokes ztotd. (The
other function, totd is not used; it's retained for reference.)
The -Q switch starts Emacs "quietly." -g
defines window geometries:3 WxH+X+Y,
where
W and H specify the frame's width
and height in character units.
X and Y, if ≥0, specify the pixel
coordinates of the upper left corner of the window. If <0, specify
the distance from the right and bottom of the screen.
This is the statement in my init file that runs the batch file:
Here's the listing for totd.el. The first
function, totd, is based on code that Dave Pearson
posted on EmacsWiki.4 To make it work with modern Emacs,
I replaced (require 'cl) with (require
'cl-lib) and (loop for s...) with
(cl-loop for s...).
;; Time-stamp: "2026-07-19 17:00:17 RayZ";; totd.el by DavePearson;; This small code snippet displays a “tip of the day”;;;; 2026-07-14 Downloaded from https://www.emacswiki.org/emacs/TipOfTheDay;; 2026-07-19 Derive ztotd to run in separate session
(require 'cl-lib)
(defuntotd ()
(interactive)
(with-output-to-temp-buffer"*Tip of the day*"
(let* ((commands (cl-loop for s being the symbols
when (commandp s) collect s))
(command (nth (random (length commands)) commands)))
(princ
(concat "Your tip for the day is:\n========================\n\n"
(describe-function command)
"\n\nInvoke with:\n\n"
(with-temp-buffer
(where-is command t)
(buffer-string)))))))
(defunztotd ()
"Display documentation of a random Emacs function to providea Tip Of The Day. It is intended to run in a separatesession of Emacs where it modifies the frame to resemble aminimalist popup. Invoke at the command line with (for example): emacs.exe -Q -g 80x42-2+2 -l totd.el --eval (ztotd)"
(interactive)
(set-buffer (generate-new-buffer "Tip of the day"))
(let* ((commands (cl-loop for s being the symbols
when (commandp s) collect s))
(command (nth (random (length commands)) commands)))
(insert
(concat (describe-function command)
"\n\nInvoke with:\n\n"
(with-temp-buffer
(where-is command t)
(buffer-string))))
(tool-bar-mode 0)
(menu-bar-mode 0)
(setq frame-title-format '("Gnu Emacs Tip of the day"))
(pop-to-buffer (current-buffer))
(goto-char (point-min))
(delete-other-windows)
(message "Press C-x C-c to dismiss")))
totd is very intrusive by design -- it's supposed to
put a help page "in your face." But I wanted a "clean" look. So I
eliminated the toolbar and menu from the frame, moved "Tip of the
Day" from the buffer to the title, and got rid of other windows.
Plus I call it with
call-process-shell-command instead
of shell-command to prevent a shell output window from
appearing in the main frame.5
The batch file allows the OS itself to initiate the display of TOTD.
It can be set up to run right after logon, or combined with Tea
Timer or Scheduler for a periodic display.
I like how this performs. However, I hardly ever close and
restart Emacs. As a result, I may go a week or two without seeing a
tip. Is that why Emacs has never included a built-in Tip Of The
Day? Or is it just so easy to implement that "the proof has been
left as an exercise to the user?" Or maybe the help that's shown is
too specific. For example, "What does
image-dired-delete-tag actually do?" you might
wonder. I think that's the whole point of a TOTD -- to inspire
wonder!
If you try this on a non-Windows system, please let me know whether
it works.
1 I tried display-buffer-pop-up-frame.
Somehow the popup lost focus and got buried behind the main frame.
When I closed the main frame, I thought that all my files were saved
and Emacs was closed. Only then did I realize the popup was still
running. When I tried to close it, Emacs warned about open files. So
nervously I maximized the small popup frame to review the unsaved
buffers.
2 In fact the batch file contains several lines of comments, too, as
well as the cherished @echo off statement at the very beginning.
@echo off prevents the many lines of comments from getting dumped to
standard output; otherwise those comments would end up in a shell buffer.
Please note that org‑get‑cust‑priority has been updated. The code that was published on 2026-07-10 causes org‑tags‑list to crash.
In the previous post,1 I was disappointed to learn that valid priority characters are limited to 0–9 and A–Z. I
thought it would be neat if the exclamation mark ("!") could represent
the ultimate priority, a sort of "drop everything and do this
immediately" kind of priority.
It turns out that you can use "!" for this purpose, regardless of
which priority characters you've configured. The trick is to write a
function and assign it to org-priority-get-priority-function.
The function (which I named org-get-cust-priority) simply
looks for "!" in the priority cookie and returns one hundred times the
value that would be returned by the highest valid priority. If it
doesn't find "!" it simply calls a modified version of the usual
org-get-property function,2 which I
named org-get-std-priority. The two functions are shown below.3
(defunorg-get-cust-priority (s)
"Return 100 times the highest priority when S containsa priority cookie with `!'. Otherwise call the usual`org-get-property' function. Intended to be the function`org-priority-get-priority-function' is set to."
(interactive)
(if (not (functionp org-priority-get-priority-function))
(org-get-std-priority s)
(string-match ".*?\\(\\[#\\(!\\)\\] ?\\)" s)
(if (and (string-match ".*?\\(\\[#\\(!\\)\\] ?\\)" s) (string-equal (match-string 2 s) "!"))
(* 100000 (abs (- org-priority-lowest org-priority-highest)))
(org-get-std-priority s))))
(defunorg-get-std-priority (s)
"Find priority cookie and return priority.S is a string against which you can match `org-priority-regexp'.Same function as `org-get-priority' sans test for a customfunction in `org-priority-get-priority-function'."
(save-match-data
(if (not (string-match org-priority-regexp s))
(* 1000 (- org-priority-lowest org-priority-default))
(* 1000 (- org-priority-lowest
(org-priority-to-value (match-string 2 s)))))))
The number that Org uses to sort on priority depends on the values of
org-priority-lowest
and org-priority-highest,4 which are "C" and "A" by
default. In the default configuration, the highest priority has a
numeric value of 2000, while the lowest has a value of 0. If the
highest and lowest priority characters
are 1 and 9, respectively, the highest priority has a value of 8000.
You know you need this if you have about a dozen items on your agenda
and half of them are prioritized A.
2026-08-13 Update org-get-cust-priority. Previous version breaks org-tags-list.
1https://ray-on-emacs.blogspot.com/2026/07/numeric-priorities-in-org-mode.html 2 Modified just to prevent recursion. 3 I know this code isn't particuarly elegant. I could have made
custom variables for "!" and the priority multiplier, for example.
I'd like to use it for a while before deciding whether it's worth investing any more effort.
I do welcome suggestions on how to improve. 4 Or rather, the difference between the highest and lowest priorities.
Org mode is configured "out-of-the-box" with these three levels of
priority: A, B, C, where A is the highest priority. However, this can
be changed. You can use numbers to indicate priority, such as 1
(highest) to 10 (lowest) as shown in the documentation.1
If you specify priorities from 65 to 90, Org will interpret the
numbers as ordinal values of ASCII characters and give you "A" to "Z."
If you try to be clever and specify priorities as 65 to 91 (91 is the
decimal ASCII code for left square bracket character) you'll be disappointed
that when you lower the priority from "Z," you don't get "[." (I was
disappointed, anyway.)
You can define priorities from 10 to 99, for example. But if you try
to get to a priority higher than 10 using org-priority-up, Org will
clear the priority cookie. However, you can manually enter a priority
of 0 to ensure the item appears above all other items. (Please see
Figure 1, below.) Thus, a priority value of 0 can be your "stat"
item. It would be super neat if Org could understand that ASCII
characters below 0 (such as the exclamation mark) are an even higher
priority than 0. Unfortunately, it doesn't even recognize [#!] as a
priority cookie.
A large priority span seems like it would be useful. For example, you
might have all utility bills set from 50 to 59, and shopping for
birthday or anniversary gifts from 10 to 19. However, in order to go
from a priority of 50 to 10, you'd have to press the Up Arrow key forty times.
If you think you can invoke M-4 0 M-x org-priority-up to go quickly
from 50 to 10, you'll be disappointed (again). The function
org-priority-up doesn't accept a prefix argument.2
Emacs, LibreOffice Writer and Vivaldi are the productivity tools I use
most often. So I like to incorporate a feature from one into the
others. One example is the ability to initiate a search from selected
text. I added this feature to LibreOffice as described here. Another nice
feature is the ability to switch between the current buffer and the next (or
previous) buffer with C-<TAB> and C-S-<TAB>
that I added to Emacs1.
But I've also configured Vivaldi with a useful Emacs feature. Now it
responds to C-S-k by invoking "Close tabs to the Right."
Emacs devotees know that C-k is bound
to kill-line, which "Kill[s] the rest of the current
line...."2 If you imagine that the row of tabs is a line of
text, it's natural to expect C-k (or C-S-k)
would remove all the tabs to the right of the current tab.
Why did I choose the key combination C-S-k instead
of C-k? I think at the time C-k performed another
function. Or perhaps I felt that it was more appropriate to use two
modifier keys for such a significant operation. I rarely need to
delete tabs to the left; if I did, I'd bind C-M-S-k to "Close tabs to
the Left."
You might utter the following complaint, "But C-S-k doesn't work for
Gmail; it brings up a virtual keyboard instead!" Correct. So while
we're on the topic of Vivaldi key configurations, I'd like to point
out that it's possible to tell Vivaldi to ignore a website's key
assignment, allowing the key to pass to Vivaldi. Just search for the
setting "Browser Priority Shortcuts," click on one of the existing key
combinations and press C-S-k to add it to the others.
I rely on the web browser's tab history a great deal, too. I'm
working on implementing something vaguely similar for Emacs."
1 To get the C-<TAB>
and C-S-<TAB> to switch buffers, just add
this to your init file:
I work on an air-gapped VM, which is to say, a virtual machine with no network access at all. You copy code in via USB or a shared folder, develop, and then you need to get that code back out to the host machine. Git is running inside the VM, of course, it's the sensible thing to do for version control, but git push wants a remote, and there is no remote. What you actually need is a file copy operation that shuttles the repo data to a shared folder, where the host can pick it up.
So I have a pair of bash scripts, unimaginatively called in and out. out copies the source tree to the shared folder so the host sees my commits, and in pulls changes back from the host into the VM. Works fine from the terminal, but I live in Emacs, and I want to trigger this from within Emacs rather than jumping to a shell.
The obvious answer is to bind a key to async-shell-command and call it done. But then I remembered, Emacs already has this infrastructure - the VC mode's push command. When you press P in a vc-dir buffer, or invoke vc-push from the mode-line popup, Emacs runs vc-git-push which calls git push. I don't want that, I want my out script. The pull side is the same problem. So the question becomes: can I make Emacs's VC commands do something else?
Yes, obviously, this is Emacs. advice-add lets you replace any function, and vc-git-push is just a function. The :override strategy means my function runs entirely in place of the original, which is what I want - there's no useful fallback to git push on this machine, it would just fail.
These two functions live in my coding.el starter snippet, here's the push side:
(defunmy/vc-git-push-shuttle (_file-list &rest _args)
"Override vc-push to run a local sync script instead of git push."
(interactive)
(let ((script-path "/home/jdyer/bin/out"))
(if (file-executable-p script-path)
(progn
(message "Syncing source TO shared folder...")
(async-shell-command script-path "*out*"))
(error"Sync script not found or not executable at %s"
script-path))))
The _file-list and _rest _args soak up whatever VC passes to the push function, I don't care about the arguments since my script doesn't need them. The function checks the script is executable, shows a message, and runs it asynchronously into the *out* buffer so I can inspect the output if something goes wrong. Pull is the same thing pointing at /home/jdyer/bin/in into *in*, so:
(defunmy/vc-git-pull-shuttle (_file-list &rest _args)
"Override vc-pull to run a local sync script instead of git pull."
(interactive)
(let ((script-path "/home/jdyer/bin/in"))
(if (file-executable-p script-path)
(progn
(message "Syncing source FROM shared folder...")
(async-shell-command script-path "*in*"))
(error"Sync script not found or not executable at %s"
script-path))))
And that's it. Now every Emacs command that would have called git push or git pull runs my shuttle scripts instead. When I press P in vc-dir the code ends up in the shared folder ready for the host to collect. No context switch to a terminal.
The same pattern would work for any situation where you want VC's push/pull to do something domain-specific - copying to a local archive, rsyncing to a staging server, or in my case, bridging an air gap.
I was working on my diff-minimap package, deep in the weeds of fringe indicators and search highlights, and I did the thing. You know the thing. I was in Dired, accidentally, for some reason hit D to delete a filename (probably some malfunctioning muscle memory) and it was gone!
Except it was my own source file. diff-minimap.el. The one I had been hacking on for the last hour. The one with all those uncommitted changes.
Now, in my head, I have a trash setup. I am sure I configured that somewhere, probably years ago, in some init file on some machine. But the machine I was sitting at? Nope. Emacs just did a straight delete, no safety net, no way back.
The file was tracked in git, so I could git checkout HEAD -- diff-minimap.el to get the committed version back. But the uncommitted work - the inline diff preview, the vc-diff positioning, the side-window hunk viewer - that was gone.
Emacs has a built-in variable, delete-by-moving-to-trash. When it is set to t, Dired (and other delete operations) move files to the system trash instead of permanently deleting them. On Linux that is ~/.local/share/Trash/files/, on macOS it is ~/.Trash/, on Windows it is the Recycle Bin.
It is one of those settings that seems obvious in hindsight but is easy to overlook because Emacs defaults it to nil. The reasoning is probably that Emacs is a serious tool for serious people who know what they are doing, so deleting means deleting?!
So I added it to my config:
(setq delete-by-moving-to-trash t)
That is it. One line. From now on, D in Dired moves to trash instead of deleting permanently and I will not have that sinking feeling again!
Anyway, back to diff-minimap.
This post is a reflection on over ten years of Emacs usage. Why did I even start using it? How has my usage changed over the years? What's the future looking like? I'll discuss this and a whole lot more. It's a spiritual successor to my post.
As I wrote earlier, I consider diary to be an underappreciated
Emacs built-in.
I got interested in it again when I wanted to schedule a recurring
meeting in Org for the third Tuesday of every month. I saw on
Reddit that some folks use diary-float in place of the usual active
timestamp to achieve this. So I added this after the meeting header
to make it work:
* Third Tuesday of the Month Club
SCHEDULED: <(%%diary-float t 2 3) 19:00-20:00>
I have five meetings that use this method of automatic scheduling.
I like this approach because of its simplicity and elegance.
Unfortunately, it doesn't behave the same way as a recurring task.
When I mark a recurring task as complete, Org reschedules it for the
next date on which it should occur. And it changes DONE to TODO to
ensure the event shows up on the agenda for the next occurrence.
This doesn't happen for a task that's scheduled with diary-float.
When marked complete, the headline remains at DONE; the next event
won't show up on the agenda.
And so Fengyuan Chen wrote next-day-spec1
to solve this issue. Unfortunately it doesn't work with the latest
versions of Emacs.
One of the many recognized cognitive biases is called the Sunk Cost
Fallacy2, which impels an individual to favor an inferior (or
incorrect) solution because a significant amount time, money, and/or
effort has been invested in the solution. Well, I've invested time
and effort in both diary-float
and next-day-spec, so I've continued to
endorse it as a solution.
But even if rescheduling did work, I think it's better to have each
meeting scheduled as a subheading of the overall meeting topic,
like this:
* Third Tuesday of the Month Club
** DONE May 2026: <2026-05-19 Tue>
:LOGBOOK:
CLOCK: [2026-05-19 Tue 18:54]--[2026-05-19 Tue 20:12] => 1:18
:END:
- Note taken on [2026-05-19 Tue 21:20] \\
We ate pizza. Again.
** TODO June 2026: <2026-06-23 Tue>
** TODO July 2026: <2026-07-21 Tue>
The advantage to this is that clocked time and notes are organized
neatly within each individual subtopic, not aggregated into one
logbook and a series of notes. If the aggregated time is desired,
one could insert the log report. I'll review this in another post.
But it's fun to schedule events that occur on irregular dates, such
as National Engineers Week (US), which is the week in which George
Washington's birth anniversary occurs3. Here's how you
can add it to an Org file:
When I adopted Emacs in July of 2000, I hunkered down to learn the
keybindings. But after I learned the basics1 I started
to RTFM (C-h r), and I was instantly drawn to the
Calendar/Diary2 node, which I consider to be an
underappreciated Emacs built-in, and, therefore, the topic of Emacs
Carnival for June 2026.
Back then I mistook "diary" to mean a blank-page-confidant into
which we write our thoughts, fears, ambitions or even just what we
had for dinner last night. At that time I'd been writing in
a journal for about 24 years (yes, since 1976, on and off), so my head
swelled with ideas of using Emacs for a revamped computerized
version. I imagined being able to forward-search-regexp
in order
to find, for example, that weird dream I wrote about in which I
caught a toad that was hopping around the kitchen and then turned
into a hot coal sizzling in my hand...
I used diary-block functions to date my entries and
organize the content. I ended up making four such entries from 2003
to 2004. But each day following the entry, M-x diary
would display a nearly blank buffer, showing only the day's date at
the top followed by a line of equal signs and perhaps a holiday.
Where did yesterday's entry go?! It was disconcerting that those
words "vanished" into thin air magnetic film or
silicon.
Today, however, I appreciate the simplicity of that approach -- a
clean slate sans distractions. But back then I was accustomed to
seeing a vast amount of my writing, which fed my ego.
But that's not all there is to appreciate. There are several
functions that can be used for "an entry" in addition to
diary-block, such as diary-anniversary, diary-cyclic, diary-float or even just a simple line that begins with a date and a brief note. This is not an all-inclusive list.
Every fancy diary buffer can show local time of sunrise and sunset;
to do this, include diary-sunrise-sunset in the diary file.
Include diary-lunar-phases to show one of the four
phases of the moon when one is active on that day; the
local time of that moon phase will be included.
What follows is an abbreviated and edited listing of my diary
files. Note how I've structured diary into a hierarchy
using include statements.
2026-06-17 02:25 GMT Important updates: First, you'll need to modify two hook variables in order
for the include statements to work. Please see
the help
for Fancy Diary Display. I also "use the normal hook
diary-list-entries-hook to sort each day's diary
entries by their time of day," which is described at the top of that
page. Second, specify the full path to the included files.
I've modified the two include statements to add the "~/" path. This
is needed only if you want to press TAB on the item in the Agenda in
order to focus the diary entry.
The beauty of this is that these events will show up in Org Agenda
at the appropriate times when org-agenda-include-diary
is non-nil.
Four examples are shown below. Here are some things to note:
The content that's derived from the diary file has "Diary" for value of CATEGORY.
The Sunrise / Sunset lines are supposed to show the time of
each occurance. However, the time for Sunrise isn't displayed;
rather, it's indicated by its position on the time grid. The times for both the New Moon and the Solar Eclipse are indicated by the time grid, as well, and are expected to peak at 13:38 in my area. That's something to look faroward to!
Sometimes I include links in my headings. In this case I can
follow a link to the credit card website to pay the Visa
card.
Even a link in the dairy file will be rendered
in the Agenda properly, as shown in the reference for Richard
Stallman's birthday.
file listing: diary
# -*- mode: diary -*-
#include "~/diary-anniversaries-property"
#include "~/diary-birthdays-friends"
%%(diary-sunrise-sunset)
%%(diary-lunar-phases)
%%(diary-remind '(diary-anniversary 3 16 1953) 14) [[https://html.duckduckgo.com/html/?q=Richard+Stallman+birthday][Richard Stallman's birthday]] in 14 days
%%(diary-block 6 21 2004 6 21 2004) The summer solstice. I
always take time to observe the position of the sun on this
day, especially in the morning and evening. Sunrise was at
5:16am and sunset will be at 8:29pm. From today onward, the
days will be getting shorter. At first it will be
imperceptible, but in September, it'll be quite noticeable.
file listing: diary-anniversaries-property
# -*- mode: diary -*-
# Recurring Property events or records
%%(diary-remind '(diary-date 7 3 '(2023 2026 2029)) 30) Car Registration is due
%%(diary-anniversary 8 12 2022) Kitchen Cabinets Painted %d Years Ago
file listing: diary-birthdays-friends
# -*- mode: diary -*-
# Birthdays of friends
%%(diary-anniversary 8 13) Bucky Thorndike Miller's Birthday (Does Dave still have his guitar amp?)
%%(diary-anniversary 2 13 1966) Ben's %d%s Birthday
Day-agenda (W11):Tuesday 16 March 2027 Diary: 7:03 ┄┄┄┄┄ Sunrise (EDT), sunset 18:58 (EDT) at Home (11:54 hrs daylight) 8:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ BILLS: 9:00 ┄┄┄┄┄ Deadline: TODO Visa Card LINK:Bills::Credit: 10:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 12:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 14:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 16:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 18:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 20:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Diary: Richard Stallman's birthday in 14 days
Day-agenda (W23):Wednesday 3 June 2026 Diary: 5:19 ┄┄┄┄┄ Sunrise (EDT), sunset 20:20 (EDT) at Home (15:00 hrs daylight) 8:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 10:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 12:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 14:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 16:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 18:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 20:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Diary: Reminder: Only 30 days until Car Registration is due
Day-agenda (W33):Wednesday 12 August 2026 Diary: 5:58 ┄┄┄┄┄ Sunrise (EDT), sunset 19:54 (EDT) at Home (13:56 hrs daylight) 8:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 9:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 10:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 12:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Diary: 13:38 ┄┄┄┄┄ New Moon (EDT) ** Solar Eclipse ** 14:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 16:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 18:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 20:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Diary: Kitchen Cabinets Painted 4 Years Ago
Day-agenda (W32):Saturday 13 August 2022 Diary: 5:59 ┄┄┄┄┄ Sunrise (EDT), sunset 19:53 (EDT) at Home (13:53 hrs daylight) 8:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 10:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 12:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 14:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 16:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 18:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 20:00 ┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Diary: Bucky Thorndike Miller's Birthday (Does Dave still have his guitar amp?)
1My requirements (the basics) for a text editor were:
I was asked recently, "Do you use Org Mode protocol ... for browser
to Emacs interaction? If so, were there any complications to set it up
on Windows? Is there like bookmarklets for capturing or doing
things?"1
At the time I wasn't doing anything too fancy to incorporate Web
Content into Org. I'd copy the URL from Vivaldi's address bar to the
clipboard and (if applicable) the Web Content I'm interested
in.2 Then
in Org, I'd yank the Web Content (if applicable), mark it and then do
org-insert-link (C-c C-l) to turn it into a hyperlink. The marked
text becomes what you see in the document. But I usually just specify
the link Description simply as "LINK." In fact I created a macro
(bound to the Insert key) to do this.
But I've started to use eww as my web browser for this. If I write
something that I need to verify, I'll mark it and invoke
eww-search-words (M-s M-w), which brings me to a DuckDuckGo page of
search results. If I follow a search result that I like, I'll mark a
small section and invoke org-store-link (C-c l) and create the link
with org-insert-link, with the marked content as the Description.
Thus org-store-link captures both the location and the context in one
magical swoop. Here's an animated GIF that illustrates the process.
Note that the video shows me capturing and referring to part of Sacha
Chua's website that shows some neat solutions to launching a web
browser from Emacs.3
My ongoing journey with vc-dir and vc-mode led me into wanting to delete a stash (not to pop). There are "z" keybindings in vc-dir:
z c vc-git-stash
z p vc-git-stash-pop
z s vc-git-stash-snapshot
but what about delete?, and are there any others?
In vc-dir you get a rather nice little Stash section tacked on at the bottom listing all your stashes:
I genuinely assumed I was missing something obvious, so I did the sensible Emacs thing and hit C-h m to describe the mode, scrolled through the whole keymap, and, nothing about stashes at all.
Well how about C-h b (describe-bindings)?, initially nothing!, however I stumbled on a solution, if the point is on a stash name, the following is exposed!
C-k vc-git-stash-delete-at-point
RET vc-git-stash-show-at-point
= vc-git-stash-show-at-point
A vc-git-stash-apply-at-point
C vc-git-stash
P vc-git-stash-pop-at-point
S vc-git-stash-snapshot
Surprisingly difficult to find.
and C-k does obviously seem a natural fit to delete a stash, but what wasn't obvious was the fact that the point had to be on the stash name.
I couldn't quite understand why these keybindings were so difficult to find, so I went digging into vc-git.el and there it was, the bindings are not part of vc-dir-mode-map at all, they live in their own little keymap:
And the crucial detail, this keymap is not installed as the major mode map, it is slapped directly onto the stash lines as a keymap text property when those lines are rendered. So the bindings are only live when point is literally sitting on a stash, which is rather elegant actually, but it does mean describe-mode never sees them, because C-h m only reports the major mode's own keymap, and a text-property keymap is invisible to it. That explained my confusion completely, I was not going mad after all!
And the Emacs manual, well I did have a little look but found nothing. It seems these keybindings are almost impossible to find, but now I have found them this blog post should help me to remember.
So, lesson learned, and a genuinely useful one I think, C-h m is not the whole story. When a mode renders interactive regions, buttons, clickable lines, embedded widgets, those often carry their own text-property keymaps that describe-mode will never show you, and the move in those situations is C-h k or C-h b with point actually on the thing.
Anyway, C-k on a stash line to drop it, that is the headline, write it on a sticky note, you will want it eventually!, if you are a vc-mode user of course, I suspect that magit has this very discoverable already through the stash menu.
If you have a numeric keypad1, try this in the *scratch* buffer:
enable numlock and press the zero on the numeric keyboard
(Num-0). Then press the zero located between the alpha keys and the
function keys (0). Okay, you see 00, no big deal.
Why do I bring this up? Emacs interprets Num-0 keypress as <kp-0> while it interprets 0 as
0. This means that you could bind Num-0 to a function without
affecting the other 0. You could configure Num-0 to insert "zero" by
entering and evaluating this in the *scratch* buffer:
That's a trivial example, of course. But it implies that you can
have ten more keys to play with. Similarly, the arithmetic
operator keys on the numeric keyboard differ from the "regular
ones" that are grouped with the alpha keys. They can be bound
to functions, referenced
as <kp-add>, <kp-subtract>, <kp-multiply>, <kp-divide>. But
these also accept the standard C- M- S- modifier keys, which
gives you twelve more possibilities. This is true also for the
Insert key <kp-insert> and the Delete key <kp-delete> that
double as Num-0 and Num-., respectively.
I have a "Calc" key above the numeric keypad. Unfortunately I
can't seem to use it for anything in Emacs -- it opens the OS's
calc.exe program regardless of what modifiers I use with it. This is
a shame because it would be the ideal mapping for M-x calc. Perhaps
this can be altered in BIOS.
But I do use my Win key as a modifier (sometimes). And I've heard
that the Caps Lock key can be repurposed. These are topics for
other posts.
In my last post I wrote about finally caving in and adding Magit to my config after years of being a contented vc-mode loyalist (VC-Mode Meets Magit - or Why I Finally Gave In!). My conclusion then was that the two complement each other nicely, Magit for repo-level operations, vc-mode for the everyday file-level stuff, commit a single file, a blame, a quick diff, C-x v and away you go. Well, I have just had a small but instructive lesson in not assuming where the limitations actually lie, because I caught myself about to reach for Magit for something vc-mode handles perfectly well!
The scenario: I had a pile of changes sitting in my working tree across a few files, and I wanted to set aside the changes in just one of them, temporarily, without touching anything else. Stash a single file, in other words.
I knew that vc-mode had some stash commands through the "z" keybinding when in vc-dir, and of course you can find them through M-x by completing on vc-git-stash*, but I had only ever used it for stashing everything, straight from the worktree. So I did a quick test: in vc-mode, I moved my point over a single modified file and selected vc-git-stash, but this of course would stash everything! Well, how about a single file? I assumed that vc-mode couldn't do it and reached for magit instead!
However a little niggling doubt surfaced and I decided to have a little rummage around the vc-git code:
(defunvc-git-stash (name)
"Create a stash given the name NAME."
(interactive"sStash name: ")
(let ((root (vc-git-root default-directory)))
(when root
(apply #'vc-git--call nil "stash""push""-m" name
(when (derived-mode-p 'vc-dir-mode)
(vc-dir-marked-files)))
(vc-resynch-buffer root t t))))
It runs git stash push -m NAME and, if you are inside a vc-dir buffer, it appends the list of marked files as a pathspec. Which means a single-file stash is built right in, no Magit required!
Open vc-dir with C-x v d (or C-x p v for the project-wide view)
Move point to the file you want to set aside and press m to mark it
Press z c, type a stash name, hit return
And that is it, git stash push -m "your name" -- the/marked/file runs under the hood, that one file's changes are tucked away, and everything else in your working tree is left exactly as it was. Mark two files and it stashes two, mark none and it stashes the lot, the marking is the whole mechanism.
To bring it back, z p pops a stash (it prompts you with a completing-read list of what is there)
This is almost the inverse of my last git post. Last time I went looking for vc-rebase and there genuinely was not one, a real git-specific gap that vc-mode does not fill, and off to Magit I rightly went. This time I assumed the same shape of limitation and assumed wrong, the feature was sitting there in vc-dir the whole time (and actually was pretty obvious really),
So, vc-mode really is quite capable for file-level git work, single-file stashing very much included, and I should be slower to assume "git-specific" means "Magit only". Magit is still there for the repo-level heavy lifting, but for tucking one file out of the way, C-x v d, m, z c, done, no need to leave the comfort of the built-in tooling at all.
I do enjoy these little corrections, each one teaches me a bit more about where the seams in Emacs version control actually are, and occasionally, that the seam I thought I had found was never there in the first place!
I have been a VC-mode loyalist for years, it is built in, it is simple, it covers the basics - commit, push, pull, diff, all there with C-x v bindings. I never really felt the need for Magit, honestly. VC-mode just works, right?
Until it does not. Or rather, until you need something that is not quite available. As vc-mode is source code control agnostic then there were always going to be some limitations and my git usage is now starting to become a little more advanced.
Recently I hit a divergent branch situation, remote had moved on, I had local commits, and git refused to push, I reached for C-x v m (vc-merge) to pull in the upstream changes, and that worked fine as a merge. But what if I wanted a rebase instead? Clean linear history, no merge commits?
Turns out there is no vc-rebase at all in Emacs! You can use vc-pull and it will rebase if pull.rebase is set in your git config (I think, although I haven't tried it), but there is no interactive vc-rebase command to invoke directly. And vc-merge only merges, If you want a rebase, you have to drop to shell or configure git to default to it. I wanted something more cohesive, more discoverable, and #sigh, here enters magit, I used it a few years ago so I still had a residue of muscle memory, but that said, I am not abandoning VC-mode entirely. For quick operations, a single file commit, a blame, a quick diff – C-x v I still like vc-mode. Magit is great for repo-level operations; VC-mode is great for file-level ones. They complement each other nicely, and there is no reason to pick one over the other, this is Emacs!
I added use-package magit to my config with a C-x g binding, full-frame status display, word-granular diff refining, and also released C-w in the magit status map so my window-management prefix still works (small things, but they matter):
This post is simply about some more yak shaving and there was this one niggle that I just kept putting off, because it never really annoyed me enough, but recently I started watching a lot more video content from within dired and it finally got to me.
The problem. I press W in dired, which is bound to dired-do-async-shell-command, type mpv (or accept the default guess that my dired-guess-shell-alist-user provides for video files), and the video starts playing. Great. But when I close Emacs, the video dies too. The whole point of an async command is that it runs in the background, right?, so why does it disappear when Emacs goes away?, this is most perplexing!
Now I should say, the answer is obvious in retrospect, but it took me a little while to actually stop and think about it properly. The command dired-do-async-shell-command adds a trailing & to the shell command and passes it to async-shell-command, which creates a shell process - Emacs is the parent of that shell, and the shell is the parent of mpv. When Emacs exits, it kills its child processes, the shell goes down, and mpv gets orphaned, which is fine actually, the problem is not being orphaned, the problem is that it receives a SIGHUP from the process group cleanup and dies before it can be reparented to init. So the media stops.
The solution is a single, tiny piece of advice. I just prepend nohup to the command string before dired-do-async-shell-command does anything with it. nohup makes the process ignore SIGHUP, so when the shell gets killed by Emacs, mpv is orphaned cleanly and keeps running.
(defunmy/dired-async-shell-command-nohup (orig-fun command &optional arg file-list)
"Wrap COMMAND with `nohup' so the process survives Emacs exit."
(funcall orig-fun (concat "nohup " command) arg file-list))
(with-eval-after-load 'dired
(advice-add 'dired-do-async-shell-command
:around #'my/dired-async-shell-command-nohup))
That is it. Six lines of Elisp, five of which are boilerplate.
And yes, this works for any command you run through W, not just mpv. gimp, firefox, libreoffice - they all get the nohup treatment now. Which is fine, those programs usually detach themselves anyway, but it does not hurt to have it.
So that is the tweak. A tiny, one-function, one-advice change that finally stopped my videos from dying every time I closed Emacs. I suspect someone will tell me that there is a simply variable flag that I can set that enables this anyway, but hey ho!
Configuration bankruptcy is someting of a meme in the Emacs community, to the point where there is a dedicated wiki page describing the phenomenon. As a fully-programmable text editor, Emacs configuration tends toward spaghetti over time, at least for many users (including myself). And for many of us, the only remedy for an unintelligible, slow, and tangled Emacs configuration is wiping the whole file and starting again.
I admit that I can be a perfectionist in some things. I also, for some reason, enjoy configuring Emacs, so I tend to tinker with it in my spare time. As a result, I’ve declared .emacs bankruptcy somewhat more than the average bear: when I finally stopped counting, I’d made it up to 12 or 13.
A few years ago—either because I got busier with work, got less interested in perfecting my setup, had a child, or whatever—my Emacs configuration more-or-less calcified. I had a custom macro for installing packages and grouping their configuration; I had plenty of custom functions and macros to mold Emacs perfectly to my needs; I had a system whereby I didn’t overwhelm my configuration with too many packages du jour.
As of April 2026, I have declared .emacs bankruptcy yet again.
As is usual with my bankruptcy declarations, the reasons are many, unfocused, and inchoate, but basically boil down to “I was bored.” Not that I have any reason to be bored: I have a toddler and a one-month-old at home, and I’m feverishly looking for work in one of the worst economies in generations. So maybe bored isn’t the right word.
My ~/.emacs (it’s actually in ~/.config/emacs, but who’s counting) was, despite my best efforts, getting brittle again. I found myself, once again, with the itch to explore and install new packages. I kept having to look up the source of functions I’d written years ago to remember how they worked. So I made a ~/emacs2 directory and ran emacs –init-dir=~/emacs2 in my shell, and was off to the races.
I’m really not changing all that much, though. The main theme of this go-round is that I’m not afraid of using external packages if that means I can have the functionality I want without having to think about it too hard. Thus:
I’m using use-package to declare packages.
I’m using super-save-mode to save buffers without having to think about it.
I’m using snap-indent instead of a complicated auto-indentation/whitespace-cleanup thing (though I might change it back; I think it might be causing a subtle bug in my workflow).
In addition, I’ve refrained from copying and pasting all my old options, which I do every bankruptcy. It’s a good chance to reëvaluate the features I really need while pruning those I don’t (or that I’ve been working around without realizing it for a while). Plus—and maybe I’m just strange for this one—it’s fun!
The theme for this month’s Emacs Carnival is May I recommend…, and my recommendation is tearing out your configuration from time to time and starting over. You’ll never really know who you’ve become until you do.
I almost always reach for project-vc-dir when I want a VC status overview, and most of the time this is exactly what I want, the whole project laid out in one buffer, every modified, added and unregistered file in the repo sitting right there, ready to be diffed or committed. But every so often, particularly when I am deep inside a big repository and I only really care about a single subdirectory's worth of changes, that project-wide view is, frankly, a bit too much. Too many rows, too much scrolling, too much noise.
So, what am I actually after?, I want the same vc-dir buffer, but scoped to whatever directory I happen to be looking at, most commonly the directory I have open in dired. And it turns out this is almost trivially easy in vanilla Emacs.
The bit I had not initially appreciated is that vc-dir itself already accepts a directory argument, it is the interactive prompt that steers you towards the repo root, because it defaults to (vc-root-dir) rather than default-directory. If you call it non-interactively with a subdirectory instead, the Git backend quite happily scopes the status listing to files underneath that path, even though the overall VC root is still the same project root.
So the fix is a tiny wrapper that just hands vc-dir the current default-directory and skips the prompt entirely:
(defunmy/vc-dir-here ()
"Run vc-dir on the current directory (dired's dir when called from dired)."
(interactive)
(vc-dir default-directory))
Then, because the main place I actually want this is from within dired, a keybinding that sits nicely alongside the standard C-x v family:
(with-eval-after-load 'dired
(define-key dired-mode-map (kbd "C-x v D") #'my/vc-dir-here))
The mnemonic, such as it is, is that C-x v d is the normal vc-dir binding with its usual prompt, and capital D is the "here, right now, this directory" variant. Lowercase for the prompted version, uppercase for the zoomed-in one, which also pairs up nicely in my head with project-vc-dir being the zoomed-out project-wide thing on C-x p v.
So the little two-tier workflow I have settled into is:
C-x p v – project-vc-dir, show me everything in the project
C-x v D from dired – my/vc-dir-here, show me just this subdirectory
And that is really the whole post, nothing clever, no new package, just a three-line wrapper and one keybinding, but it has genuinely taken a surprising amount of friction out of navigating VC state in large repos where I know perfectly well the only thing I have touched is under src/foo/, and I do not particularly want to be reminded of every other outstanding change elsewhere in the tree.
And yes, magit can do this with some narrowing, but actually, I like vc-mode!
I have been doing a lot of fiddling with images lately, mostly through dired and image-dired, and one little thing has been bugging me for a while. When I open an image in Emacs, image-mode happily shows me the picture, but it never tells me the one bit of information I actually want to know, how big is the thing?, width, height, file size, that sort of thing. You can of course bounce out to a shell and run identify or file, but that feels silly when Emacs already has the image loaded.
So I thought, right, this should be a five minute job, just slap something into the header-line on image-mode-hook and be done with it. And it more or less was, although there was a small wrinkle along the way that is worth mentioning, because it caught me out.
Here is what ended up in my init.el:
A few notes on what is going on here. The or around image-get-display-property and create-image means we use the displayed image spec when it is there (cheaper, no extra file read), and fall back to building one from the file path when it is not. image-size with a non-nil second argument returns dimensions in actual pixels rather than canvas units, which is what I want. file-size-human-readable gives me a nice 2.4M rather than 2516582, because nobody reads bytes directly.
The condition-case is there because images can occasionally throw, especially when something has gone wrong with imagemagick or an unsupported format sneaks in, and I would rather see a polite header-line message than have the hook explode and pollute *Messages* every time I open a picture.
The second hook, image-mode-new-window-functions, is the one that handles the case where you flip between image and text view of the same buffer with C-c C-c, since the image gets re-displayed and the header-line needs to refresh too.
So now when I pop open an image, I get a nice little header-line that reads something like:
1920 x 1080 px 2.4M
A small thing, but the kind of small thing that makes Emacs feel like it fits a bit more snugly around how I actually work.
Let me begin by saying that web bookmarks are great. You can just save links that you find interesting or that you think you might need later, and there they are waiting for you when you need them. I hope that whoever came up with that feature got a good raise or bonus or something.
However, they are lacking in one key area, namely: they do not act as bookmarks in books do. When you keep a bookmark in a real book, it moves with your progress throughout the text, so you always know where you are in the larger work. Web bookmarks don’t do that. They stick to the page you marked forever, unless you manually change where they point.
Given the workings of the web—especially the early web—it makes sense. Most pages, even today, are standalone documents that you can read through in one go, so if you want to keep them, you just keep the whole thing.
However, there are many documents online that are part of some larger work. I’m currently reading through Clojure for the brave and true, and it’s one of these: each chapter is a web page, but the whole work spans many of them. Lots of books online follow this pattern, including Texinfo manuals, software instruction books, and web comics. These documents need something more akin to the real-book bookmarks, which in computer-talk is skeuomorphic.
I had this idea for some kind of skeuomorphic bookmarking system a little while ago, and since then I’ve been yearning for it. Since it’s nontrivial, I didn’t think I had the chops to pull it off—but luckily I don’t have to.
I’ve written before about the Emacs Web Wowser, and I’m happy to report that it has wowed me again. I discovered that, as I worked through Clojure in a split view with eww and my code, bookmarking my progress at the end of sessions, Emacs would suggest the same name for each new bookmark, overwriting it and effectively moving the bookmark further in the text.
In other words, Emacs already does exactly what I’ve been looking for! While I haven’t looked into the code to see what heuristics it uses, the sites I’ve tried it on have worked fine.
I’ve written this post to appreciate the Emacs developers for their forward thinking, and to let everyone who reads this (hi mom!) know about a good feature that has given me joy. Try the Web Wowser today if you haven’t yet!
This past Tuesday I typed C-x C-c in Emacs for the last time after 20
years of daily use. Though nearly half that time was gradually
retiring it, switching to modal editing, then to Vim. Emacs is a platform,
and I’d grown accustomed to its applications, especially those I built
myself. There was no particular hurry, so replacements came slowly. With
my newly-acquired superpowers I could knock out the last two pieces
in a few days’ work, namely M-x calc with stackcalc and
Elfeed with Elfeed2. I’m especially excited about the
latter because it already exceeds the original. Both are multi-platform,
native C++ GUI applications using native UI components.
These actively-in-use packages require new maintainers (apply on
the project’s issues/discussion):
No wonder it took so long for me to move on! I’m not handing these off to
just anyone, and you’ll need to establish your reputation. Having already
made contributions is a good sign, even if never merged. I’m willing to
transfer them off my namespace, though you’ll need to manage the Melpa
hand-off (on which I’ll sign-off). If there are no takers, these projects
will be archived but not deleted.
Trying out wxWidgets
The Emacs Calculator is amazing and the best calculator I’ve ever used,
which is why nothing I could find was going to replace it. My clone uses
GMP and MPFR for multi-precision, so it’s far faster, as to be expected,
but it’s not nearly at feature parity. It’s missing esoteric features
including symbolic processing. Though it’s enough to cover all of my own
usage. I can add more features later. The Emacs Calculator manual served
as a specification when building stackcalc.
Elfeed has been a cornerstone of my daily routines for the past 13 years.
Nothing else I’ve found scratches that itch for me, so I’ve always known
it would require a rewrite someday. Knowing it would take a few weeks of
work, and that I already had the feed reader I wanted, made motivation
difficult to find. Though now that I can accomplish ~3 weeks of old-way
work in a new-way day, this sort of project becomes that much easier to
start and finish. Though it’s not yet at a 1.0 release, after a couple
days Elfeed2 was working well enough to replace the original Elfeed.
While Dear ImGui was the right choice for dcmake, it would not be
so for these two applications. Active rendering doesn’t suit a feed reader
left running all day, and I needed a richer toolkit. Professionally I work
in Qt, but I wanted something lighter-weight for my projects, accessible
via CMake FetchContent. That naturally led to wxWidgets. While it
has issues — mitigatable character encoding problems, accidental quadratic
time in many places — it’s worked better than I anticipated, letting me
rapidly produce native-looking applications on Windows, macOS, and Linux.
Unlike Dear ImGui, wxWidgets is a platform, including sane I/O and
path handling. I mostly don’t need platform layers when building
applications like these. I can simply rely on wxWidgets’ utilities.
Both of these projects build out-of-the-box on w64devkit thanks to the
dependencies being FetchContent-compatible. On all platforms you just
need a C++ toolchain and CMake:
$ cmake -B build
$ cmake --build build
Now that I have experience with wxWidgets, learning its limitations and
capabilities, it’s likely to be a foundation of most of my GUI projects to
come, except where something like Dear ImGui is a better git.
Qwen 3.6 35b has been a fantastic thinking companion for me, anything
that I don’t know, I am not comfortable with, or having doubts
with, I would check with it. I found Qwen 3.6 + DeerFlow 2.0 is much
better than the paid version of Grok, and miles better than
Perplexity.
Today, I made it even better by giving it vision. Earlier I uploaded
an image of my staircase and asked it to check the conditions when I
plan the staircase renovation project.
This blog post highlights the key steps of how i did it.
Firstly, Qwen 3.6 has vision encoder built-in already, but it
requires an additional mmproj component to make it work.
Honestly I have no idea what does it mean at the moment, I just
think of it as the eyes to LLM.
Download the mmproj file from the Unsloth Qwen 3.6 repo1, add the
path to –mmproj argument for llama-server command, reboot
llama.cpp, that’s it.
The vision component requires additional 1-2GB of vram, so to make
them fit to RTX 3090, I had to quantize the mmproj component from
bf16 to q4:
llama-quantize mmproj-BF16.gguf mmproj-Q4_K_M.gguf Q4_K_M
llama-server Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \--mmproj mmproj-Q4_K_M.gguf \
... # rest of the llama-server arguments
To test it,
check the mmproj is loaded successful from the llama.cpp log,
This is the response I got, so it confirms it works. The image
will change from time to time, so the response will be
different.
The image is a scenic landscape photograph, likely taken in late autumn or winter. It features a vast mountain range in the background, rolling hills in the mid-ground covered in snow and trees, and a foreground of dry, grassy terrain. The sky is dramatic with a mix of blue and warm sunset/sunrise colors.\n\n**2. Breaking down the image into layers
if 1. success, but 2. failed, query the log file, grep vision or
image, e.g. this is what I got when i misspell mmproj in
llama-server at one point:
print_info: PAD token = 248055 '<|vision_pad|>'
srv operator(): got exception: {"error":{"code":500,"message":"image input is not supported - hint: if this is unexpected, you may need to provide the mmproj","type":"server_error"}}
The model is equipped for vision tasks, next step is to enable vision
on DeerFlow 2.0, all I need is adding the support_vision to true in
config, full model spec is listed below to avoid ambiguity
I have to add increase the timeout to 10 mins because the vision
component is a lot slower than text generation, with the default
value, DeerFlow will throw errors thinking the LLM is not
responding. the vision component can be optimised later to reduce
the runtime, but so far so good.
Now test DeerFlow 2.0. Restart the services (make docker-stop &&
make docker-start), open a new chat, upload a PNG file, and ask
to describe, wait for a bit, then boom!
I can also copy an image, and paste it to deerflow, which is very
nice interface.
Qwen 3.6 describes an uploaded image in DeerFlow 2.0
Lately I’ve found myself wanting a better, more fine-grained view of what’s going on in a file under git. For some reason, my default workflow has been to keep jumping in and out of project-vc-dir to check changes. It gets the job done, but honestly it’s a bit of a hassle
What I really wanted was something right there in the buffer. Not a full-on inline diff (that gets messy fast I would guess), but just a small visual hint, something that lets me "see" what’s changed without breaking my flow.
Turns out, that’s exactly what diff-hl does
It’s super lightweight and just highlights changes in the fringe. Nothing flashy but just enough to keep you aware of what you’ve modified. Once you start using it, it feels kind of weird not having it.
One thing I really like is how nicely it plays with the built-in VC tools, move to a buffer position that aligns with a highlighted change, hit C-x v = and it jumps straight to the relevant hunk in the diff. No friction, no extra thinking, it just works.
Here’s the setup I’m using:
By default, diff-hl-mode only updates when you save the file. That’s okay, but enabling diff-hl-flydiff-mode makes it update as you type, which feels more intuitive.
Oh, and that dired-mode hook? That turns on diff-hl-dired-mode, which gives you a quick visual overview of changed files right inside dired. It’s one of those small touches that ends up being surprisingly useful.
If you’ve got repeat-mode enabled, you can also hop through changes with C-x v ] and C-x v [, which makes reviewing edits really smooth.
I am enjoying diff-hl and is quietly improving my workflow without getting in my way. Simple, fast, and just really nice to have.
I have been slowly chipping away at my Emacs-DIYer project, which is basically my ongoing experiment in rebuilding popular Emacs packages using only what ships with Emacs itself, no external dependencies, no MELPA, just the built-in pieces bolted together in a literate README.org that tangles to init.el. The latest addition is a DIY version of dired-collapse from the dired-hacks family, which is one of those packages I did not realise I leaned on until I started browsing a deeply-nested Java project and felt the absence immediately.
If you have ever opened a dired buffer on something like a Maven project, or node_modules, or a freshly generated resource bundle, you will know the pain, src/ contains a single main/ which contains a single java/ which contains a single com/ which contains a single example/, and you are pressing RET four times just to get to anything interesting. The dired-collapse minor mode from dired-hacks solves this beautifully, it squashes that whole single-child chain into one dired line so src/main/java/com/example/ shows up as a single row and one RET drops you straight into the deepest directory.
So, as always with the Emacs-DIYer project, I wondered, can I implement this in a few elisp defuns?
Right, so what is the plan?, dired already draws a nice listing with permissions, sizes, dates and filenames, all I really need to do is walk each line, look at the directory, figure out the deepest single-child descendant, and then rewrite the filename column in place with the collapsed path. The trick, and this is the bit that took me a minute to convince myself of, is that dired uses a dired-filename text property to know where the filename lives on the line, and dired-get-filename happily accepts relative paths containing slashes. So if I can rewrite the text and reapply the property, everything else, RET, marking, copying, should just work without me having to touch the rest of dired at all!
First function, my/dired-collapse--deepest, which just walks the directory chain as long as each directory contains exactly one accessible child directory. I added a 100-iteration guard so a pathological symlink cycle cannot wedge the whole thing, which, you know, future me might thank present me for:
(defunmy/dired-collapse--deepest (dir)
"Return the deepest single-child descendant directory of DIR.
Walks the directory chain as long as each directory contains exactly
one entry which is itself an accessible directory. Stops after 100
iterations to guard against symlink cycles."
(let ((current dir)
(depth 0))
(catch 'done
(while (< depth 100)
(let ((entries (condition-case nil
(directory-files current t
directory-files-no-dot-files-regexp
t)
(error nil))))
(if (and entries
(null (cdr entries))
(file-directory-p (car entries))
(file-accessible-directory-p (car entries)))
(setq current (car entries)
depth (1+ depth))
(throw 'done current)))))
current))
directory-files-no-dot-files-regexp is one of those lovely little built-in constants I keep forgetting exists, it filters out . and .. but keeps dotfiles, which is exactly what you want if you are deciding whether a directory is truly single-child.
Second function does the actual buffer surgery, my/dired-collapse iterates each dired line, grabs the filename with dired-get-filename, asks the walker how deep the chain goes, and if there is anything to collapse it replaces the displayed filename with the collapsed relative path:
(defunmy/dired-collapse ()
"Collapse single-child directory chains in the current dired buffer.
A DIY replacement for `dired-collapse-mode' from the dired-hacks
package. Rewrites the filename portion of each line in place and
reapplies the `dired-filename' text property so that standard dired
navigation still resolves to the deepest directory."
(when (derived-mode-p 'dired-mode)
(let ((inhibit-read-only t))
(save-excursion
(goto-char (point-min))
(while (not (eobp))
(condition-case nil
(let ((file (dired-get-filename nil t)))
(when (and file
(file-directory-p file)
(not (member (file-name-nondirectory
(directory-file-name file))
'(".""..")))
(file-accessible-directory-p file))
(let ((deepest (my/dired-collapse--deepest file)))
(unless (string= deepest file)
(when (dired-move-to-filename)
(let* ((start (point))
(end (dired-move-to-end-of-filename t))
(displayed (buffer-substring-no-properties
start end))
(suffix (substring deepest
(1+ (length file))))
(new (concat displayed "/" suffix)))
(delete-region start end)
(goto-char start)
(insert (propertize new
'face 'dired-directory
'mouse-face 'highlight
'dired-filename t))))))))
(error nil))
(forward-line))))))
The key bit is the propertize call at the end, the new filename text has to carry dired-filename t so that dired-get-filename picks it up, and dired-directory on face keeps the collapsed entry looking the same as a normal directory line. Because dired-get-filename will happily glue a relative path like main/java/com/example onto the dired buffer's directory, pressing RET on a collapsed line takes you straight to src/main/java/com/example with no extra work from me.
A while back I added a little unicode icon overlay thing to dired (my/dired-add-icons, which puts a little symbol in front of each filename via a zero-length overlay), and I did not want the collapse to fight with it. The icons hook into dired-after-readin-hook as well, so I just gave collapse a negative depth when attaching its hook:
Lower depth runs earlier, so collapse rewrites the line first, then the icon overlay attaches to the final collapsed filename position. Without this, the icons would happily sit in front of a stub directory that was about to be rewritten, which is, well, fine I suppose, but it felt tidier to have them anchor on the post-collapse text.
Before, a typical Maven project root might look something like this:
One RET and you are in com/example, which is where all the actual code lives anyway. Marking, copying, deleting, renaming, all of it still behaves because the dired-filename text property points at the real deepest path.
One thing that initially bit me, is navigating out of a collapsed chain. If I hit RET on a collapsed src/main/java/com/example line I land in the deepest directory, which is great, but then pressing my usual M-e to go back up was doing the wrong thing. M-e in my config has always been bound to dired-jump, and dired-jump called from inside a dired buffer does a "pop up a level" thing that ended up spawning a fresh dired for com/, bypassing the collapsed view entirely and leaving me staring at a directory I never wanted to see.
My first attempt at fixing this was to put some around-advice on dired-jump so that if an existing dired buffer already had a collapsed line covering the jump target, it would switch to that buffer and land on the collapsed line instead of splicing in a duplicate subdir. It worked, sort of, but dired-jump in general felt a bit janky inside dired, it does a lot of "refresh the buffer and try again" under the hood and the in-dired pop-up-a-level path was always the weak link. So I stepped back and split the two cases apart with a tiny dispatch wrapper:
(defunmy/dired-jump-or-up ()
"If in Dired, go up a directory; otherwise dired-jump for current buffer."
(interactive)
(if (derived-mode-p 'dired-mode)
(dired-up-directory)
(dired-jump)))
(global-set-key (kbd "M-e") #'my/dired-jump-or-up)
From a file buffer, dired-jump is still exactly the right thing as you want the directory the file is in of course. From inside a dired buffer, dired-up-directory is just a much cleaner operation, it walks up one real level, no refresh, no splicing, nothing weird. But on its own that would lose the collapsed round-trip, so I gave dired-up-directory its own bit of advice that looks for a collapsed-ancestor buffer before falling through to the default behaviour.
The dired-up-directory only fires when the literal parent is not already open as a dired buffer, which keeps normal upward navigation completely unchanged:
If /proj/src/main/java/com/ happens to already exist as a dired buffer, dired-up-directory does its usual thing and just goes there, the up-advice never fires. It is only when the literal parent is absent that the advice kicks in and hands you back to the collapsed ancestor, which I think is the right tradeoff, the advice never surprises you when you were going to get the standard behaviour anyway, it only steps in when the standard behaviour would throw away context you clearly still had in a buffer somewhere.
End result, RET into a collapsed chain drops me deep, M-e walks me back out to the original collapsed line, and none of it requires doing anything clever with dired-jump's "pop up a level" path, which I am increasingly convinced I should not have been using in the first place.
Everything lives in the Emacs-DIYer project on GitHub, in the literate README.org. If you just want the snippet to drop into your own init file, the two functions and the add-hook line above are the whole thing, no require, no use-package, no MELPA, just built-in dired and a bit of buffer shenanigans, and thats it!, phew, and breathe!
Having solved it in Clojure, I decided to translate that code into a new Emacs Lisp solution. I suspected that I'd like it much more than my Elisp one from years ago — and that turned out to be true.
There's an old programming challenge where the digits in a list of phone numbers are converted to letters according to rules and a given dictionary file. The results of the original challenge suggested that Lisp would be a potentially superior alternative to Java, since Lisper participants were able to produce solutions in, on average, fewer lines of code and less time than Java programmers. Some years ago I tackled it in Emacs Lisp and Bash. I've now done it in Clojure.
Freelancing for small businesses and organizations leads to a position where you are juggling a
number of projects for multiple clients. You need to keep track of a number of tasks ranging from
software development to sending emails to project management. This is a lot easier when you have a
system that can do a bunch of the work for you, which is why I use Emacs as my freelancer command
center.
I would like to share some of the tools and workflows I use in Emacs to help me keep on top of
multiple clients’ needs and expectations.
Organization with org-mode
It should be no surprise that at the center of my Emacs command center is org-mode. I have already
written about it a lot. Every org-mode user seems to have their own way of keeping track of things,
so please don’t take my organizational scheme as some kind of gospel. A couple of years ago, I wrote
about how I handle to-do lists in org-mode, and I am still using that method for to-do keywords.
However, file structure is also important. I have a number of core files.
Freelance.org
This top-level file contains all of my ongoing business tasks, such as tracking potential new
clients, recurring tasks like website maintenance and checking my MainWP dashboard. I also have
recurring tasks for invoicing, tracking expenses, and other important business things.
This file is also where I have my primary time tracking and reporting. Org-mode already supports
this pretty nicely, I just use the built-in clocktable feature.
Clients/*.org
Clients that have large projects or ongoing business get their own file. This makes organization a
lot easier. All tasks associated with a client and their various projects end up in these individual
files. The important part is making sure that these files are included in the time-tracking
clock table and your org-mode agenda, so you can see what is going on every week.
References and Linking
I have C-c l bound to org-store-link and use it all the time to link to various files,
directories, URLs, and even emails. I can then use those links in my client notes, various tasks in
my to-do list, and so on. This helps me keep my agenda organized even when my filesystem and
browser bookmarks are a bit of a mess.
Email with mu4e
I have been reading and managing my email in Emacs for over 25 years. There have been a few breaks
here and there where I have tried out other software or even web mail clients, but it has always been
a headache. I return to Emacs! Long ago, I used VM (which seems to have taken on new life!), but
currently I use mu4e.
This gives me a ton of power and flexibility when dealing with email. I have custom functions to
help me compose and organize my email, and I can use org-store-link to keep track of individual
emails from clients as they relate to agenda items. I even have a function to convert emails that I
have written in Markdown into HTML email, and one that searches for questions in a client email to
make sure I haven’t missed anything.
The ability to write custom code to both process and create email is extremely powerful and a great
time saver.
Writing Code
I don’t know what else to say about this, I use Emacs for doing all of my software development. I
make sure to use Eglot whenever there is a language server available, and I try to leverage all the
fancy features offered by Emacs whenever possible. The vast majority of projects for clients are PHP
(thanks WordPress), Go, JavaScript, and TypeScript.
Writing Words
Previously, I have shared quite a bit about writing in Emacs. I like to start everything in
org-mode, but I also write quite a bit in Markdown. Emacs has become a powerful tool for writing. I
use the Harper language server along with Eglot to check grammar and spelling.
Track All Changes with Magit
Version control is essential, a lesson I have learned over 30+ years of software development. While
Git is not part of Emacs, the software I use to interface with Git is. Magit is a Git user interface
that runs entirely in Emacs. I use it to track my writing, my source code, and even all of my
org-mode files. Using version control is so essential that I have a weekly repeating agenda task
reminding me to check all of my everyday files to make sure I have checked-in my changes for the
week.
Thinking Music with EMMS
I like to have some soothing background music when I am programming, writing, or otherwise working
on my computer. However, if that background music has lyrics, it can be really distracting. It is
easy to make a playlist for various suitable SomaFM channels to load into EMMS (the Emacs Multimedia
System) using the command M-x emms-play-playlist.
Try saving the following into playlist.el somewhere, and using it the next time you are writing:
;;; This is an EMMS playlist file Play it with M-x emms-play-playlist((*track*(type.url)(name."https://somafm.com/synphaera.pls"))(*track*(type.url)(name."https://somafm.com/gsclassic.pls"))(*track*(type.url)(name."https://somafm.com/sonicuniverse.pls"))(*track*(type.url)(name."https://somafm.com/groovesalad.pls")))
And make sure to check out SomaFM’s selection to find some good background music that suits your
tastes!
And the tools I have missed
There are undoubtedly Emacs tools that I have missed in this brief overview. I have been wracking my
brain as I write, trying to see what I have forgotten or overlooked. Frankly, Emacs has become such
a central part of the organization for my freelancing that there are probably many tools, packages,
and processes that I use every day without thinking about it too much.
Emacs makes it possible for me to freelance for multiple clients and small businesses without losing
my mind with organization and task management. The tools it provides allow me to stay on top of
multiple projects, handle client relationships, and keep track of years worth of tasks,
communications, and projects. Without it, I’d be sunk!
What Emacs tools are you using to manage your freelance business? I am always looking for ways to
improve or streamline my process.
Flymake has been quietly sitting in my config for years doing exactly what it says on the tin, squiggly lines under things that are wrong, and I mostly left it alone. But recently I noticed I was doing the same little dance over and over: spot a warning, squint at the modeline counter, run `M-x flymake-show-buffer-diagnostics`, scroll through the list to find the thing I was actually looking at, then flip back. Two windows, zero connection between them.
So I wired it up properly, and while I was in there I gave it a set of keybindings that feel right to my muscle memory.
The obvious bindings for stepping through errors are `M-n` and `M-p`, and most people using flymake bind exactly those. The problem is that in my config `M-n` and `M-p` are already taken, they step through simply-annotate annotations (which is itself a very handy thing and I am not giving it up!). So I shifted a key up and went with the shifted variants: `M-N` for next, `M-P` for previous, and `M-M` to toggle the diagnostics buffer.
With M-M I wanted it to be a bit smarter than just "open the buffer". If it is already visible I want it gone, if it is not I want it up. The standard toggle pattern:
(defunmy/flymake--diag-buffer ()
"Return the visible flymake diagnostics buffer, or nil."
(seq-some (lambda (b)
(and (with-current-buffer b
(derived-mode-p 'flymake-diagnostics-buffer-mode))
(get-buffer-window b)
b))
(buffer-list)))
(defunmy/flymake-toggle-diagnostics ()
"Toggle the flymake diagnostics buffer."
(interactive)
(let ((buf (my/flymake--diag-buffer)))
(if buf
(quit-window nil (get-buffer-window buf))
(flymake-show-buffer-diagnostics)
(my/flymake-sync-diagnostics))))
Now the interesting bit. What I really wanted was a follow mode, something like how the compilation buffer tracks position or how Occur highlights the current hit. When my point lands on an error in the source buffer, the corresponding row in the diagnostics buffer should light up. That way the diagnostics window becomes a live index of where I am rather than a static dump and think in general this is how a lot of other IDEs work.
I tried the lazy route first, turning on hl-line-mode in the diagnostics buffer and calling hl-line-highlight from a post-command-hook in the source buffer. The line lit up once and then refused to move. Nothing I did would shift it. This is because hl-line-highlight is really only designed to be driven from the window whose line is being highlighted, and I was firing it from afar.
Ok, so why not just manage my own overlay:
(defvarmy/flymake--sync-overlay nil
"Overlay used to highlight the current entry in the diagnostics buffer.")
(defunmy/flymake-sync-diagnostics ()
"Highlight the diagnostics buffer entry matching the error at point."
(when-let* ((buf (my/flymake--diag-buffer))
(win (get-buffer-window buf))
(diag (or (car (flymake-diagnostics (point)))
(car (flymake-diagnostics (line-beginning-position)
(line-end-position))))))
(with-current-buffer buf
(save-excursion
(goto-char (point-min))
(let ((found nil))
(while (and (not found) (not (eobp)))
(let ((id (tabulated-list-get-id)))
(if (and (listp id) (eq (plist-get id :diagnostic) diag))
(setq found (point))
(forward-line 1))))
(when found
(unless (overlayp my/flymake--sync-overlay)
(setq my/flymake--sync-overlay (make-overlay 1 1))
(overlay-put my/flymake--sync-overlay 'face 'highlight)
(overlay-put my/flymake--sync-overlay 'priority 100))
(move-overlay my/flymake--sync-overlay
found
(min (point-max) (1+ (line-end-position)))
buf)
(set-window-point win found)))))))
My first pass at the walk through the tabulated list did not work. I was comparing (tabulated-list-get-id) directly against the diagnostic returned by flymake-diagnostics using eq, and it was always false, which meant found stayed nil forever and the overlay never moved. A dive into flymake.el revealed why. Each row in the diagnostics buffer stores its ID as a plist, not as the diagnostic itself:
(list :diagnostic diag
:line line
:severity ...)
So I need to pluck out :diagnostic before comparing. Obvious in hindsight, as these things always are. With plist-get in place the comparison lines up and the overlay moves exactly where I want it, tracking every navigation command.
The fallback lookup using line-beginning-position and line-end-position is there because flymake-diagnostics (point) only returns something if point is strictly inside the diagnostic span. When I land between errors or on the same line as an error but a few columns off, I still want the diagnostics buffer to track, so I widen the search to the whole line.
Finally, wrap the hook in a minor mode so I can toggle it per buffer and enable it automatically whenever flymake comes up:
(define-minor-modemy/flymake-follow-mode"Sync the diagnostics buffer to the error at point.":lighter nil
(if my/flymake-follow-mode
(add-hook 'post-command-hook #'my/flymake-sync-diagnostics nil t)
(remove-hook 'post-command-hook #'my/flymake-sync-diagnostics t)))
(add-hook 'flymake-mode-hook #'my/flymake-follow-mode)
(define-key flymake-mode-map (kbd "M-M") #'my/flymake-toggle-diagnostics)
The end result is nice. M-M pops the diagnostics buffer, M-N and M-P walk through the errors, and as I navigate the source the matching row in the diagnostics buffer highlights in step with me. If I close the buffer with another M-M everything goes quiet, and I can still step through with M-N/M-P on their own.
Three little keybindings and twenty lines of elisp, but they turn flymake from a static reporter into something that actually feels connected to where I am in the buffer.
I have been busy improving my annotation package! Simply Annotate, the latest release is 0.9.8 and I have put in a bunch of new features, so it felt like a good time to step back and show what the package actually does at this point, because honestly, quite a lot has changed since I last wrote about it.
There are annotation packages out there already, annotate.el being the most established. And they are good! But I kept running into the same friction: I wanted threaded conversations directly on my code, I wanted multiple display styles I could combine, and I wanted the whole thing to be a single file with no dependencies that I could drop onto an air-gapped machine and just use (yup, that again!)
So I built my own!, this is Emacs, after all.
At its core, simply-annotate lets you attach persistent notes to any text file (or Info manual, or dired buffer) without modifying the original content. Annotations are stored in a simple s-expression database at ~/.emacs.d/simply-annotations.el. The entire package is a single elisp file, requires Emacs 28.1+, and has zero external dependencies.
Basic setup is two lines:
Open a file, select some text, press C-c a j, and you have your first annotation. M-n and M-p step through them.
I am always fiddling around with styles, themes, backgrounds e.t.c, so I thought I would build this tinkering enthusiasm into this package. Simply-annotate has five display styles, and you can layer them together:
Highlight – classic background colour on the annotated region
Tint – a subtle background derived from your current theme, lightened by a configurable amount. Adapts automatically when you switch themes
Fringe – a small triangle indicator in the fringe, minimal and unobtrusive
Fringe-bracket – a vertical bracket spanning the full annotated region in the fringe, with a proper top cap, continuous vertical bar, and bottom cap
Subtle – overline and underline bracketing the region, barely visible but there when you need it
You can combine them, so (tint fringe-bracket) gives you a gentle background wash with a clear fringe bracket showing exactly where the annotation spans. Cycle through styles with C-c a '.
Toggle inline display with C-c a / and annotation content appears as box-drawn blocks directly in your buffer:
some annotated code here
▴
┌─ ✎ [OPEN/NORMAL] ──────────────
│ This function needs refactoring.
│ The nested conditionals are hard
│ to follow.
└─────────────────────────────────
New in 0.9.8 is the inline pointer, that little ▴ connecting the box to the annotated text. It is indented to the exact column where the annotation starts, so you always know what the comment refers to.
The fun bit is that the pointer is just a string, and it supports multiline. So you can customise it to whatever shape you like:
There is a full list of copy-paste options in the Commentary section of the elisp file. Set to nil to disable the pointer entirely.
So I think this is where simply-annotate really differs from other annotation packages. Every annotation is a thread. You can reply to it with C-c a r, and replies can be nested under any comment in the thread, not just the root. It prompts with a hierarchical completing-read menu showing the comment tree.
Each thread has:
Status – open, in-progress, resolved, closed (C-c a s)
Priority – low, normal, high, critical (C-c a p)
Tags – freeform hashtags for organisation (C-c a t)
Author tracking – configurable per-team, per-file, or single-user
The comment tree renders with box-drawing characters so the hierarchy is always clear:
┌— ® [OPEN/NORMAL] —
| james dyer (03/29 08:27)
| This is the original comment
| L james dyer (03/29 08:27)
| | here is a reply to this comment
| | L james dyer (@3/29 08:27)
| | and a reply within a reply!!
| L james dyer (03/29 08:28)
| Here is another reply to the original comment
└────────────────────
Annotations exist at three levels: file (whole-file overview), defun (function or block description), and line (individual elements). There is also an all pseudo-level that shows everything at once, which is the default.
Cycle levels with C-c a ] and C-c a [. The header-line shows counts per level (FILE:2 | DEFUN:5 | LINE:3) with the active level in bold, so you always know where you are, my idea here is to lean towards a coding annotation tool to help teach code or help to remember what has been implemented, so the levels start at a broad file overview and enables you to switch instantly to a more granular level.
The org-mode listing (C-c a l) gives you a foldable, navigable overview of all annotations in the current file, grouped by level. Press n and p to step through headings, RET to jump to source.
New in 0.9.6, the tabular listing (C-c a T) opens a fast, sortable table using tabulated-list-mode (a feature in Emacs I am starting to leverage more). Columns for Level, Line, Status, Priority, Comments, Tags, Author, and the first line of the comment. Click column headers to sort. This is brilliant for getting a quick birds-eye view of all the open items in a file.
For the global view, simply-annotate-show-all gathers annotations from every file in the database into a single org-mode buffer.
Enable simply-annotate-dired-mode and dired buffers show fringe indicators next to files that have annotations. You can see at a glance which files have notes attached:
Info manuals are also fully supported. Annotations are tracked per-node, and the listing and jump-to-file commands navigate to Info nodes seamlessly.
Press C-c a e and you can edit the raw s-expression data structure of any annotation. Every field is there: thread ID, status, priority, tags, comments with their IDs, parent-IDs, timestamps, and text. C-c C-c to save. This is the escape hatch for when the UI does not quite cover what you need.
Rather than writing paragraphs about how simply-annotate compares to other packages, I have put together a feature matrix in the README. The short version: if you want threaded conversations, multiple combinable display styles, annotation levels, a smart context-aware command, and zero dependencies in a single file, this is the package for you. If you need PDF annotation, go with org-noter or org-remark, they are excellent at that.
Toggle inline display with C-c a / and annotation content appears as box-drawn blocks directly in your buffer:
some annotated code here
▴
┌─ ✎ [OPEN/NORMAL] ──────────────
│ This function needs refactoring.
│ The nested conditionals are hard
│ to follow.
└─────────────────────────────────
New in 0.9.8 is the inline pointer, that little ▴ connecting the box to the annotated text. It is indented to the exact column where the annotation starts, so you always know what the comment refers to.
The fun bit is that the pointer is just a string, and it supports multiline. So you can customise it to whatever shape you like:
There is a full list of copy-paste options in the Commentary section of the elisp file. Set to nil to disable the pointer entirely.
Threaded Conversations
This is where simply-annotate really differs from other annotation packages. Every annotation is a thread. You can reply to it with C-c a r, and replies can be nested under any comment in the thread, not just the root. It prompts with a hierarchical completing-read menu showing the comment tree.
Each thread has:
Status – open, in-progress, resolved, closed (C-c a s)
Priority – low, normal, high, critical (C-c a p)
Tags – freeform hashtags for organisation (C-c a t)
Author tracking – configurable per-team, per-file, or single-user
The comment tree renders with box-drawing characters so the hierarchy is always clear:
┌— ® [OPEN/NORMAL] —
| james dyer (03/29 08:27)
| This is the original comment
| L james dyer (03/29 08:27)
| | here is a reply to this comment
| | L james dyer (@3/29 08:27)
| | and a reply within a reply!!
| L james dyer (03/29 08:28)
| Here is another reply to the original comment
└────────────────────
Annotations exist at three levels: file (whole-file overview), defun (function or block description), and line (individual elements). There is also an all pseudo-level that shows everything at once, which is the default.
Cycle levels with C-c a ] and C-c a [. The header-line shows counts per level (FILE:2 | DEFUN:5 | LINE:3) with the active level in bold, so you always know where you are, my idea here is to lean towards a coding annotation tool to help teach code or help to remember what has been implemented, so the levels start at a broad file overview and enable you to switch instantly to a more granular level.
The org-mode listing (C-c a l) gives you a foldable, navigable overview of all annotations in the current file, grouped by level. Press n and p to step through headings, RET to jump to source.
New in 0.9.6, the tabular listing (C-c a T) opens a fast, sortable table using tabulated-list-mode (a feature in Emacs I am starting to leverage more). Columns for Level, Line, Status, Priority, Comments, Tags, Author, and the first line of the comment. Click column headers to sort. This is brilliant for getting a quick birds-eye view of all the open items in a file.
For the global view, simply-annotate-show-all gathers annotations from every file in the database into a single org-mode buffer.
Enable simply-annotate-dired-mode and dired buffers show fringe indicators next to files that have annotations. You can see at a glance which files have notes attached:
Info manuals are also fully supported. Annotations are tracked per-node, and the listing and jump-to-file commands navigate to Info nodes seamlessly.
Press C-c a e and you can edit the raw s-expression data structure of any annotation. Every field is there: thread ID, status, priority, tags, comments with their IDs, parent-IDs, timestamps, and text. C-c C-c to save. This is the escape hatch for when the UI does not quite cover what you need.
Rather than writing paragraphs about how simply-annotate compares to other packages, I have put together a feature matrix in the README. The short version: if you want threaded conversations, multiple combinable display styles, annotation levels, a smart context-aware command, and zero dependencies in a single file, this is the package for you. If you need PDF annotation, go with org-noter or org-remark, they are excellent at that.
This month’s Emacs Carnival, my first participating, is titled “Mistakes and Misconceptions.” This story happened over the past few years, behind the scenes, and was just resolved this month when I posted about an annoyance I had with how I thought eww, the Emacs Web Wowser, just worked. Turns out, it was me all along.
the problem
Emacs has a reputation for shipping with, let’s say, questionable defaults, and I labored for some time under the assumption that navigating to a fragment in a URL—say, from <https://www.acdw.net/eww-fragments.html> to <https://www.acdw.net/eww-fragments.html#the-problem>—it just was how Emacs worked that it reloaded the entire page.
If you try clicking the second link above in your eww, you’ll probably find that that is not the case at all. I assumed that I’d run into one of the rough edges of Emacs, so I posted about it online to see if anyone had written some elisp to fix the issue. What I found was that no one else had the issue.
the solution
You can read the details of my conversation with Omar and mousebot in the thread linked above. Suffice it to say that, after some posting and learning about the url-debug feature, it turns out that I’d (setopt eww-use-browse-url “.”) in my init.el at some point, thinking that it would streamline my browsing. Instead, by unconditionally opening a link in browse-url, I caused emacs to reload the page on every link, regardless of where it was from. A quick fix to (setopt eww-use-browse-url “^[^#]*$”)—which matches any url without a fragment—solved the problem.
the takeaway
To sum up:
I made a premature optimization ages ago without fully thinking through the ramifications.
I labored under the false impression that some Emacs developer had failed to think through eww’s code in handling a fairly common case in urls.
I turned to the community looking for a fix (Emacs is endlessly customizable, after all).
I finally realized that the problem had been me all along.
One Emacs/Linux/libre software maxim I love is if it breaks, you get to keep both pieces. I have total control over my machine, and that means my mistakes are mine and mine alone. However, libre culture means that we’re all in this together: while the mistake was mine, the fix was a community effort—and the fact that the knobs were there to tweak at all is a result of decades of community building and shared development.
I made a mistake in changing a setting I didn’t truly understand, and another in assuming that Emacs just came that way. This time, I was just “holding it wrong.” But the great thing about Emacs is that, when there is functionality I need that’s not in the base product, I can write it—or ask for help writing it—myself. And of course, contribute it back.
Ever found yourself wanting to add a new AI provider to ollama-buddy? (probably not I would guess 🙂), only to realise you'd need to write an entire Elisp module? Or perhaps you're running a local inference server that speaks the OpenAI API, but can't be bothered with the ceremony of creating a dedicated provider file?
Fair question. That's exactly why I built ollama-buddy-provider-create — a single function that lets you register any LLM provider in seconds, whether it's a cloud API or your own local server.
The traditional approach required a separate .el file for each provider — OpenAI, Claude, Gemini, you name it. Each with its own defcustom variables, configuration boilerplate, and maintenance overhead. It worked, but it felt a bit… heavy-handed for simple use cases.
What if you just wanted to quickly add support for that new local LM Studio instance running on port 1234? Or point ollama-buddy at your company's internal AI gateway? Previously, you'd be looking at copying an existing provider file and modifying dozens of lines. Now? One function call.
The magic (yes, of the elisp kind!) happens in ollama-buddy-provider.el, which provides a generic provider registration system. Instead of requiring separate Elisp files, you can register any provider with a single call:
(ollama-buddy-provider-create
:name"My Local Server":api-type 'openai
:endpoint"http://localhost:1234/v1/chat":models-endpoint"http://localhost:1234/v1/models":api-key"your-key-here":prefix"l:")
Three API types are supported out of the box:
openai (default) — Any OpenAI-compatible chat/completions API
claude — Anthropic Claude Messages API
gemini — Google Gemini generateContent API
The system handles all the underlying HTTP requests, error mapping, and session management automatically. Your provider just needs to specify which API flavour it speaks.
Adding a local LM Studio instance:
(ollama-buddy-provider-create
:name"LM Studio":api-type 'openai
:endpoint"http://localhost:1234/v1/chat/completions":models-endpoint"http://localhost:1234/v1/models":api-key"not-needed"; LM Studio often doesn't require auth
:model-prefix"l:")
Connecting to OpenRouter (400+ models through one API):
After registration, your new provider appears in the status line and becomes available through the standard model selection interface. The model-prefix (like l: for local or r: for OpenRouter) lets you quickly identify which provider a model belongs to.
The provider system leverages ollama-buddy's shared infrastructure in ollama-buddy-remote.el, which extracts common functionality like request handling, error mapping, and response processing. This means your custom provider gets the same robust error handling as the built-in ones:
Proper HTTP status code mapping (rate limits, timeouts, authentication errors)
Async request support for non-blocking UI
Automatic model listing and caching
Integration with the existing session and conversation system
When you call ollama-buddy-provider-create, it registers your provider with the core system, making it available to all the usual entry points: the transient menu, model selection, and conversation buffers.
This approach is perfect for:
Local inference servers (LM Studio, llama.cpp, vLLM, Ollama's own OpenAI-compat layer)
Company/internal AI gateways
Quick experiments with new APIs
The beauty ojf this system is that it makes ollama-buddy genuinely extensible without requiring deep knowledge of its internals. Want to add support for that new AI service that launched yesterday? You can probably do it in five lines of configuration rather than fifty.
Next up I think this will be the big one, adding tooling for those external providers!!!
Update: For the rubes from /r/emacs, furrowing their brows in a vain attempt to understand the situation, my EmacsWiki page might help contextualize my views.
I stopped using Magit with Emacs a few months ago. Initially this was just because Magit added a new dependency cond-let that was conflicting the upstream development of a macro by the same name, but I had already taken issue with the number of dependencies such as llama, and generally do not enjoy Transient-based user interfaces, so I ended up sticking with the descision.
First things first: The absence of Magit in my regular arsenal is noticeable and slightly annoying. There are things I have gotten so used to doing via Magit, that I had to look up how I could do them without. And it is about this experience that I want to write about here.
The first and most immediate difference is that when I have to perform some Git-action, I don’t call magit-status (which I had bound to C-c g instead of the default global bindingC-x g). Instead I mostly use shell-command (M-!), or more specifically my “fork” shell-command+ that I had extended to always run certain commands asynchronously, such as git or even defer to the respective VC-mode commands. So M-! git diff RET actually calls vc-diff, instead of just spitting the output into “*Async Shell Command*” (even if in this specific case I’d usually invoke C-x v D (vc-root-diff)). Having the bash-completion package installed also helps to find the right flags for git subcommands.
In other operations I can now use VC-mode more effectively. As of Emacs 31, a number of useful features have been added (mostly by the now new Emacs maintainer Sean Whitton) such as the ability to edit previous commit messages by pressing e in the “*vc-changes-log*” buffer, which extends the previous capabilities to just amend an existing commit message more effectively. See etc/NEWS for more details.
But there are useful features in Magit that are actually composite commands. The ones I found missing the most were spinning of branches (creating a new branch and reverting the previous branch to the upstream position) and easy fixups of previous commits. When researching online, an idea I came up upon was to add git alii to my configuration. I have to admit that I was disinclined to do so, again not so much for necessarily technical reasons, but mostly because I associate the usage of a git alias with people who insist on saving time by writing git c instead of git commit (or worse yet, they add a shell alias gc). Instead, I recalled that any subcommand git foo checks if an executable git-foo is in PATH and can defer to that instead. So I wrote scripts for the most common use-cases that I ran into
git spinoff NEW-BRANCH-NAME, create a new branch NEW-BRANCH-NAME with the commits between HEAD and the upstream state of the current branch, and then reset the current branch to the upstream state.
git update, is basically just a shorthand for git pull --autostash --rebase.
git fixup COMMIT, tries to amend the changes in the staging area onto a COMMIT, and then rebase all subsequent commits onto the modified changes.
These are really basic scripts that I just invoke using M-!, no further magic involved.
Some Git commands require user input, which in a terminal would start a TUI editor like vi or GNU nano. As I am executing commands using M-!, this is an issue, since I am not emulating a proper terminal, and do not intend to do so. Magit handles this using the with-editor package, which I have been thinking about using as well, but for now I just set
(setenv "EDITOR" "ed")
in my init.el and have been running with it since. It is not ideal, but slightly funny, and 99% of the time all I have to do is write wq anyway.
All in all this was an interesting exercise, and for now it works well enough. I know I am more in the “integrating development environment” school of Emacs, so some of my practices might appears strange, but I hope some might also be intrigued. Finally, I have to clarify that this post is about how I work with Git, and is not a hit-piece on Magit or anyone working on Magit — the issues I take with Magit are due to my own preferences and requirements, and not normative statements on how everyone should use Git with Emacs.
There is now an in-buffer replace feature in ollama-buddy, so now an ollama response can work directly on your text, streaming the replacement in real-time, and giving you a simple accept/reject choice!, I have also added an smerge diff inline if desired to show the differences and give the user the ability to accept or reject
Toggle it on via C-c O → Actions → W, or run M-x ollama-buddy-toggle-in-buffer-replace
A small ✎ indicator appears in your header line, confirming the mode is active
Select a region of text you want rewritten
Invoke a command from the role menu; for example, C-c o then pick your rewrite command
Watch as the AI streams the replacement directly into your buffer
It is also worth noting that each custom menu can be defined with a :destination option, for either in-buffer or chat, so the global in-buffer replace doesn't need to be selected, and the custom transient menu can be tailored for each command. For example, in the default custom menu, refactor code and proofread have the destination of in-buffer set, but actions like git commit message or describe code will fall back to the global option, which by default is chat, which is probably what you would want for these options.
During streaming, you'll see a dimmed, italic [Rewriting...] placeholder where your selection was. The new text then flows in with a highlight overlay.
Once the stream finishes, you're dropped into a minor mode with two options:
C-c C-c → accept the changes (keep new text, clear highlighting)
C-c C-k → reject and restore your original text
The mode line helpfully shows [Rewrite?] while you're deciding. It's a bit like ediff but for AI-generated changes and uses smerge-mode.
What if the AI starts going off the rails halfway through? No problem. Press C-c C-k at any point during streaming and the network process stops immediately, restoring your original selection. No waiting for it to finish rambling!
Press C-c d and the original text gets inserted below the new text in the same buffer. Word-level differences are highlighted using smerge-refine-regions, so you can see exactly what changed at a glance.
Green overlays mark added or modified words. Red strikethrough overlays show what was removed. It's proper granular diffing, not just a before-and-after comparison.
This workflow is particularly useful for:
Refactoring code blocks you've already written
Rephrasing documentation or prose that feels clunky
Adjusting tone without losing your core message
Quick grammar and style improvements
It's less about "write this for me from scratch" and more "help me iterate on what I've already got."
For best results, I've found that smaller, focused selections work better than trying to rewrite entire files in one go. The AI has more context to work with, and you can make more granular decisions about what to keep.
Next up is probably some more polish on the diff highlighting, or perhaps exploring how this could work with multi-file projects. But for now, again I think this implementation is good enough.
One of the fundamental limitations of local LLMs is their knowledge cutoff - they don't know about anything that happened after their training data ended. The new web search integration in ollama-buddy solves this by fetching current information from the web and injecting it into your conversation context. Ollama has a specific API web search section, so it has now been activated!
The web search feature implements a multi-stage pipeline that transforms search queries into clean, LLM-friendly context, your search query is sent to Ollama's Web Search API, the API returns structured search results with URLs and snippets.
I have decided that each URL by default is fetched and processed through Emacs' built-in eww and shr HTML rendering, but this can of course be configured, set ollama-buddy-web-search-content-source to control how content is retrieved:
`eww' (default): Fetch each URL and render through eww/shr for clean text
`api': Use content returned directly from Ollama API (faster, less refined)
The shr (Simple HTML Renderer) library does an excellent job of converting HTML to readable plain text, stripping ads, navigation, and other noise, so I thought why not just use this rather than the return results from the ollama API, as they didn't seem to be particularly accurate.
The cleaned text is formatted with org headings showing the source URL and attached to your conversation context, so when you send your next prompt, the search results are automatically included in the context. The LLM can now reason about current information as if it had this knowledge all along.
There are multiple ways to search; firstly, is inline @search() syntax in your prompts (gradually expanding the inline prompting language!), so for example:
What are the key improvements in @search(Emacs 31 new features)?
Compare @search(Rust async programming) with @search(Go concurrency model)
ollama-buddy automatically detects these markers, executes the searches, attaches the results, and then sends your prompt, so you can carry out multiple searches.
You can also manual Search and Attach, Use C-c / a (or M-x ollama-buddy-web-search-attach)
The search executes, results are attached to your session, and the ♁1 indicator appears in the header line and the results can be viewed from the attachments menu, so for example would display something like:
* Web Searches (1)
** latest Emacs 31 features
*** 1. Hide Minor Modes in the Modeline in Emacs 31
*** 2. New Window Commands For Emacs 31
*** 3. Latest version of Emacs (GNU Emacs FAQ)
*** 4. bug#74145: 31.0.50; Default lexical-binding to t
*** 5. New in Emacs 30 (GNU Emacs FAQ)
with each header foldable, containing the actual search results.
There is a little configuration required to go through the ollama API, first, get an API key from https://ollama.com/settings/keys (it's free). Then configure:
(use-package ollama-buddy
:bind
("C-c o" . ollama-buddy-role-transient-menu)
("C-c O" . ollama-buddy-transient-menu-wrapper)
:custom;; Required: Your Ollama web search API key
(ollama-buddy-web-search-api-key "your-api-key-here"))
For clarification, the content source options are as follows:
The ollama-buddy-web-search-content-source variable controls how content is retrieved:
eww (default, recommended)
Fetches each URL and renders HTML through Emacs' eww/shr. Produces cleaner, more complete content but requires additional HTTP requests.
Pros:
Much cleaner text extraction
Full page content, not just snippets
Removes ads, navigation, clutter
Works with any website
Cons:
Slightly slower (additional HTTP requests)
Requires network access for each URL
api (experimental)
Uses content returned directly from the Ollama API without fetching individual URLs. Faster but content quality depends on what the API provides.
Pros:
Faster (single API call)
Less network traffic
Cons:
Content may be truncated
Quality varies by source
May miss important context
I strongly recommend sticking with eww - the quality difference is substantial.
By default, web search fetches up to 5 URLs with 2000 characters per result. This provides rich context without overwhelming the LLM's context window.
For longer research sessions, you can adjust:
Be mindful of your LLM's context window limits. With 5 results at 2000 chars each, you're adding ~10K characters to your context.
The web search integration fundamentally expands what your local LLMs can do. They're no longer limited to their training data - they can reach out, fetch current information, and reason about it just like they would with any other context, so hopefully this will now make ollama-buddy a little more useful
What is a Blog Carnival?
In case you haven’t heard about this tradition, the aforelinked EmacsWiki article summarises it as:
A blog carnival is a form of independent, personally curated content aggregation, often hosted by different people in rotation, in which the host would write a post linking to any posts submitted on the chosen topic.
As many of the preceding hosts have already done, I’ll also quote Christian Tietze, who summarized it as:
A blog carnival is a fun way to tie together a community with shared writing prompts, and marvel at all the creative interpretations of the topic of the month.
Prompt
My prompt is “Mistakes and Misconceptions”: I want to encourage contributors to reflect on their own learning experience and comment on what they recall things like:
What were assumptions I had when starting to use Emacs that held me back? For instance functionality you thought was missing, packages you assumed were necessary, etc.
Were there practices that you remember that you later discovered were needlessly inefficient?1
Can you recall things that had previously annoyed you, that you over time not only started tolerating but actually understanding and appreciating?
Do you recall any insights that lead to “viewquakes”?
Do you have examples of mistakes you have observed other users make, and have (hopefully) helped them by pointing these out?2
I am aware that this is a more negative spin that what had been the case in the previous months, but that doesn’t mean we have to go around insulting each other. Being able to recognise mistakes, be in in your own behaviour or that of others is important for personal growth. Trying to pin-point these experiences and deriving a general lesson, ideally one you can even share with others is well worth throwing aside the veneer friendly relativism!
My contribution
(This is actually a second take on a HN comment I wrote a few months back.)
I don’t have a specific memory of using compile for the first time. I do remember when I started learning Emacs, that a blog post recommended binding the command to C-c k (or was it C-c C-k?) which is still my key binding up until this day.
For those who are not familiar, when you start compile the first line in the *compilation* buffer is something like:
-*- mode: compilation; default-directory: "~/Source/emacs/" -*-
Compilation started at Wed Feb 25 21:50:08
make -k -j6
...
“Why this noise, I just want the output of the compile command?”, I recall wondering. Teenage minimalism, I guess. Only later did I find out that these were “file variables”: In this case setting the major mode of the current buffer to compilation-mode and the directory relative to which processes are started to my local Emacs checkout.
Later down you might encounter some output like
...
INFO Scraping 1577 files for loaddefs...done
INFO Scraping 27 files for loaddefs...
INFO Scraping 27 files for loaddefs...done
make[3]: Entering directory '/home/phi/Source/emacs/lisp'
ELC emacs-lisp/package.elc
In toplevel form:
emacs-lisp/package.el:700:1: Warning: reference to free variable ‘defcustom’
emacs-lisp/package.el:700:11: Warning: reference to free variable ‘package-review-directory’
emacs-lisp/package.el: Error: Invalid read syntax: ")", 713, 18
...
where of course, we know that the line with the line number is a hyperlink. You can press RET or click on it, and Emacs will open the offending file (again, relative to default-directory) on the right line.
I don’t know if this is the proper terminology, but I like to think of hyperlinks as “explicit” and “implicit”. The former is something like HTML, where you mark up some text, that then becomes hypertext. Compare that to something like Plan 9’s plumber or GNU Hyperbole, where the system recognises patterns in existing text and imbues these with actionability.
The line-reference in the error message above is kind of an implicit hyperlink, where the major mode has recognised this during initialisation, and annotated it as such (compare that to Acme where you’d use mouse button 3 to interact without any prior setup).
Here is what took me too long to appreciate: The fact that Emacs recognises the file links is due to the major mode, the major mode is annotated with (but not actually set by) the file local variable, in a buffer that is not associated with a buffer — but that doesn’t mean you cannot write the *compilation* buffer to disk. Not only is that possible, but it just works. You can have multiple *compilation* buffers lying around anywhere, referring to different other directories that remain just as interactive after killing the buffer and opening it again at some later point. Just press g (usually revert-buffer, but here recompile) and compilation-mode will execute the command in the first line (so make -k -j6 in the example above) replacing the output of the buffer. The same trick works with modes that derive from compilation-mode, such as grep-mode.
This is an excellent example of how a major mode can just enhance a regular file, creating hyperlinks and thus abstracting over line numbers, that make display-line-number-mode superfluous (another story, different mistake) and make re-execution simple. The entire state of the buffer is derived from the contents that are already on disk! There is no need to introduce complexity and implicit dependencies on other files in the same directory or in your .emacs.d; it is possible to leverage the strengths of both Unix and Emacs in synchronicity!
So all in all, looking back I’d try to urge myself not to dismiss the local file variables as noise, but the key to a powerful method of enhancing (or enchanting) plain text. I don’t know if I would have grok’ed it at the time, but then again, is that really what I would choose to talk about if I travelled back in time?
Other contributions
If you decide to write something, contact me and send me a URL to your blog post that I can add here.
For April’s Carnival, see https://www.emacswiki.org/emacs/CarnivalApril2026.
Note that I am not talking about “efficiency” in an economic sense, but rather just in the direct relation of a human to a computer, where instead of the computer taking care of the repetitive and mechanical work, the burden is needlessly shifted onto the human operator.↩︎
The common cop-out of saying something along the lines of “everyone is right in their own way and that is fine”. I hope to see strong and interesting opinions that advance a discussion, even if they turn out to be wrong or if you end up changing your mind at some point in the future.↩︎
One of the things that has always slightly bothered me about chatting with a local LLM is that it only knows what it was trained on (although I suppose most LLMs are like that) . Ask it about your own codebase, your org notes, your project docs - and it's just guessing. Well, not anymore! Ollama Buddy now ships with proper Retrieval Augmented Generation support built-in
What even is RAG?
If you haven't come across the term before, the basic idea is simple. Instead of asking the LLM a question cold, you first go off and find the most relevant bits of text from your own documents, then you hand those bits to the LLM along with your question. The LLM now has actual context to work with rather than just vibes. The "retrieval" part is done using vector embeddings - each chunk of your documents gets turned into a mathematical representation, and at query time your question gets the same treatment. Chunks that are mathematically "close" to your question are the ones that get retrieved.
In this case, I have worked to keep the whole pipeline inside Emacs; it talks to Ollama directly to contact an embedding model, which then returns the required information. I have tried to make this as Emacs Org-friendly as possible by storing the embedding information in Org files.
Getting started
You'll need an embedding model pulled alongside your chat model. The default is nomic-embed-text which is a solid general-purpose choice:
ollama pull nomic-embed-text
or just do it within ollama-buddy from the Model Management page.
Indexing your documents
The main entry point is M-x ollama-buddy-rag-index-directory. Point it at a directory and it will crawl through, chunk everything up, generate embeddings for each chunk, and save an index file. The first time you run this it can take a while depending on how much content you have and how fast your machine is - subsequent updates are much quicker as it only processes changed files.
Supported file types (and I even managed to get pdf text extraction working!):
Emacs Lisp (.el)
Python, JavaScript, TypeScript, Go, Rust, C/C++, Java, Ruby - basically most languages
Org-mode and Markdown
Plain text
PDF files (if you have pdftotext from poppler-utils installed)
YAML, TOML, JSON, HTML, CSS
Files over 1MB are skipped (configurable), and the usual suspects like .git, node_modules, __pycache__ are excluded automatically.
The index gets saved into ~/.emacs.d/ollama-buddy/rag-indexes/ as a .rag file named after the directory. You can see what you've got with M-x ollama-buddy-rag-list-indexes.
The chunking strategy
One thing I'm quite happy with here is the chunking. Rather than just splitting on a fixed character count, documents are split into overlapping word-based chunks. The defaults are:
(setq ollama-buddy-rag-chunk-size 400) ; ~500 tokens per chunk
(setq ollama-buddy-rag-chunk-overlap 50) ; 50-word overlap between chunks
The overlap is important - it means a piece of information that sits right at a chunk boundary doesn't get lost. Each chunk also tracks its source file and line numbers, so you can see exactly where a result came from.
Searching and attaching context
Once you have an index, there are two main ways to use it:
M-x ollama-buddy-rag-search - searches and displays the results in a dedicated buffer so you can read through them
M-x ollama-buddy-rag-attach - searches and attaches the results directly to your chat context
The second one is the useful one for day-to-day work. After running it, your next chat message will automatically include the retrieved document chunks as context. The status line shows ♁N (where N is the number of attached searches) so you always know what context is in play. Clear everything with M-x ollama-buddy-clear-attachments or C-c 0.
You can also trigger searches inline using the @rag() syntax directly in your prompt and is something fun I have been working on to include an inline command language of sorts, but more about that in a future post.
The similarity search uses cosine similarity with sensible defaults (hopefully!)
(setq ollama-buddy-rag-top-k 5) ; return top 5 matching chunks
(setq ollama-buddy-rag-similarity-threshold 0.3) ; filter out low-relevance results
Bump top-k if you want more context, lower the threshold if you're not getting enough results.
A practical example
Say you've been working on a large Emacs package and you want the LLM to help you understand something specific. You'd do:
M-x ollama-buddy-rag-index-directory → point at your project directory
Wait for indexing to complete (the chat header-line shows progress)
M-x ollama-buddy-rag-attach → type your search query, e.g. "streaming filter process"
Ask your question in the chat buffer as normal
The LLM now has the relevant source chunks as context and can give you a much more grounded answer than it would cold.
And the important aspect, especially regarding local models which don't often have the huge context sizes often found in online LLMs is that it allows for very efficient context retrieval.
That's pretty much it!
The whole thing is self-contained inside Emacs, no external packages or vector databases, you index once, search as needed, and the LLM gets actual information rather than hallucinating answers about your codebase or anything else that you would want to ingest and it will hopefully make working with local LLMs through ollama noticeably more useful and accurate.
The past year has brought a lot of changes, not the least of which is that I'm now working primarily in C after a few years of web development in NodeJS/React/general JavaScript land. There's some irony to this given that in my first PyCon talk back in 2022, I proudly declared that I wanted to write as little C/C++ as possible when reimplementing a constraint library in Python using Cython.
However, I've never been particularly happy with high levels of abstraction and black-box implementations going back to my days of frustration with structural engineering software, so maybe it's fitting now that I'm working at a lower level and pursuing performant programming in C. Of course, the switch has meant working with a completely different toolset than I've used in the past, and learning a lot about Emacs along the way.
I use Emacs 30.2 installed via the Emacs Plus Homebrew tap with native compilation, along with Doom Emacs to configure most of the features that I want to use.
Some of the packages not included in Doom Emacs that I find particularly useful for general development are:
git-link - allows you to copy a direct link to Github at the current line (useful for quickly pinpointing code for discussion)
rainbow-delimiters - highlights parentheses, brackets, braces in different colors (useful for nested input files)
wgrep - makes a grep buffer editable and allows you to write changes to all files modified in that buffer
My main mode when working in C is c-ts-mode, which is the C editing mode powered by tree-sitter
I believe installing tree-sitter per language now is much easier than it used to be, as I only needed to install the C grammar as outlined in this Mastering Emacs article per the "Compiling and Installing with the builtin method in Emacs" section. While the notes say that it doesn't work well unless you're using GCC and running Linux, I didn't have any problems with Clang and MacOS.
I still don't think I'm using Tree-sitter to its full capabilities in Emacs, but it has been fun to explore the code base using treesit-explore-mode and treesit-inspect-mode. I'd like to do more with its code navigation and highlighting, as I'm still relying primarily on projectile for code navigation. While researching some of the links for this blog post, I discovered the Combobulate package (disclaimer, from the same author as Mastering Emacs), which looks like it could be helpful.
Unfortunately, I can't use .clang-format, as there is an unorthodox set of formatting requirements that don't fit into any of the common C standards. As a result, I needed to add the following .clang-format to my base directory to avoid significant frustration.
One of the biggest boons for navigating a large, unfamiliar codebase (in a new language!) has been utilizing org mode to its fullest. It's how I write out work plans, write code snippets for investigation or debugging, and add in-line images to track current application state.
Early on, I added a function to copy an org mode link to the current file and line number (appropriately named copy-org-link-to-line). This has made it easy to navigate to functions I need to modify in my work plan and include additional context as my understanding of the codebase improves.
(defun copy-org-link-to-line ()
"Copy an Org mode link to the current file and line number.
Format: [[file+emacs:/absolute/path/to/file.txt::$line_number][file.txt:$line_number]]"
(interactive)
(if-let ((file (buffer-file-name))
(line (line-number-at-pos)))
(let ((org-link (format"[[file+emacs:%s::%d][%s:%d]]" file line (file-name-nondirectory file) line)))
(kill-new org-link)
(message"Copied %s" org-link))
(message"Buffer is not visiting a file")))
I've also struggled with tab/space presentation conflicting across different minor modes (probably an artifact from Doom, which has electric-indent-local-mode and ws-butler-mode), so I have a custom display setting for tabs in c-ts-mode:
I initially loaded and unloaded debugging files drafted in org-mode and loaded just into the command line lldb. This was largely because I couldn't get debugging with dap-mode to work at all with C. Surprise! This was because I use eglot instead of lsp-mode, as eglot seems to have won out for using language servers within Emacs, and dap-mode seems to be tied to lsp-mode.
In the interest of KISS, I opted to just use dape since it uses eglot out of the box.
I have two 'dape-configs set up in my config.el. One for launching/debugging the GUI, and another for launching/debugging a single test suite, which look something like this:
After my frustration trying to configure dap-mode, dape just worked! By default, the run adapter fields are easy to figure out before transferring that information to a dape-config as I did.
The dape-many-windows buffer layout is intuitive and easy to work with, providing immediate insight with 3 smaller buffers for Local/Global/Registers/Watch variables, the Stack/Modules/Sources, and Breakpoints/Threads, as well as positioning the dape-repl in a buffer at the bottom of the screen.
lldb shortcuts do not work in the provided dape-repl. This can get annoying with having to write out breakpoint --file file.c --line $line_number --condition "i == 200" over and over, rather than the shorthand brlc.
Watch variables or conditional statements set in the repl do not appear in the Breakpoints/Threads window, where you can easily delete them, which can be confusing while debugging.
It also seems to be impossible to "reset" breakpoint counts set using the dape command M-x M-a b (These are the ones that populate in the Breakpoint/Threads buffer) wwithout manually visiting the file. I've had to navigate to the breakpoint location and then toggle on/off to achieve this.
Editing watch variables via dape-info-watch-edit-node is really weird. I can't figure out how to get it to commit my changes despite the hints indicating it should just be C-c C-c.
Eglot as a process disconnects a lot. I don't know if this is due to dape or just in general while working in my project. I'd say 80% of the time I reach for xref-find-definition, I get a warning that eglot is not connected, which is very frustrating. (I don't get an error upon firing eglot-reconnect though!).
Especially in the age of LLMs, I'm becoming more and more leery of trusting online tutorials or resources. Since I'm working in C99, it seemed like a good investment to get some C standard texts, since the language is small enough to fit into pretty slim volumes.
Before starting my new job I read:
Effective C - though this was a bit of a challenge because it includes updates to cover all the way to C23!
Tool calling has landed in ollama-buddy!, it's originally not something I really thought I would end up doing, but as ollama has provided tool enabled models and an API for this feature then I felt obliged to add it. So now LLMs through ollama can now actually do things inside Emacs rather than just talk about them, my original "do things only in the chat buffer and copy and paste" might have gone right out the window in an effort to fully support the ollama API!
What is Tool Calling?
The basic idea is simple: instead of the model only generating text, it can request to invoke functions. You ask "what files are in my project?", and instead of guessing, the model calls list_directory, gets the real answer, and responds with actual information.
This creates a conversational loop:
You send a prompt
The model decides it needs to call a tool
ollama-buddy executes the tool and feeds the result back
The model generates a response using the real data
Steps 2-4 repeat if more tools are needed
All of this is transparent - you just see the final response in the chat buffer.
The new ollama-buddy-tools.el module ships with 8 built-in tools:
Safe tools (read-only, enabled by default):
read_file - read file contents
list_directory - list directory contents
get_buffer_content - read an Emacs buffer
list_buffers - list open buffers with optional regex filtering
search_buffer - regex search within a buffer
calculate - evaluate math expressions via calc-eval
Unsafe tools (require safe mode off):
write_file - write content to files
execute_shell - run shell commands
Safe mode is on by default, so the model can only read - it can't modify anything unless you explicitly allow it, I think this is quite a nice simple implementation, at the moment I generally have safe mode off but always allowing confirmation for each tool action, but of course you can configure as necessary.
Example Session
With a tool-capable model like qwen3:8b and tools enabled (C-c W):
>> PROMPT: What defuns are defined in ollama-buddy-tools.el?
The model calls search_buffer with a regex pattern, gets the list of function definitions, and gives you a nicely formatted summary. No copy-pasting needed.
Custom Tools
You can register your own tools with ollama-buddy-tools-register:
The registration API takes a name, description, JSON schema for parameters, an implementation function, and a safety flag. The model sees the schema and decides when to call your tool based on the conversation.
A ⚒ symbol now appears next to tool-capable models everywhere - header line, model selector (C-c m), and model management buffer (C-c M). This follows the same pattern as the existing ⊙ vision indicator, so you can see at a glance which models support tools.
That's it. Pull a tool-capable model (qwen3, llama3.1, mistral, etc.) or use an online tool enabled model from ollama and start chatting. Next up is probably some web searching!, as again the ollama API supports this, so you will be able to pull in the latest from the interwebs to augment your prompt definition!
If you've ever enabled a new theme and noticed your tab-bar faces stubbornly hanging onto old colours or custom tweaks, I have found often that the tab-bar, tab-bar-tab, and tab-bar-tab-inactive faces don’t always blend cleanly with freshly loaded themes - especially of the older variety (a bit like me) and especially ones that came out before the tab bar was introduced into Emacs.
So how about a simple solution?, Can I implement something, that whenever I load a theme, the tab-bar faces update based on the theme’s default faces to establish a visually pleasant and coherent look?
Yes, yes I can!; the result is a tiny Elisp enhancement that hooks directly into Emacs' theme-loading process.
Firstly however we need to have a method that will reliably pass over the themes default faces to the tab-bar. Here’s the function that realigns the tab-bar styling with your active theme:
This simply rebuilds the key tab-bar faces so they derive their colours from the current theme’s normal face definitions, so any old themes should now not leave the tab bar faces hanging.
Now for the function activation; Emacs 29 introduced enable-theme-functions, a hook that runs every time a theme is enabled - perfect for our use case, but as always I have my eye on older Emacs versions, so lets fall back to a classic approach: advice on load-theme.
Here’s a version‑aware setup that does the right thing automatically:
(if (version<= "29.1" emacs-version)
;; Emacs 29.1+ - use the official theme hook
(add-hook 'enable-theme-functions
(lambda (_theme)
(selected-window-accent-sync-tab-bar-to-theme)))
;; Older Emacs - fall back to advising load-theme
(progn
(defun selected-window-accent-sync-tab-bar-to-theme--after (&rest _)
(selected-window-accent-sync-tab-bar-to-theme))
(advice-add 'load-theme :after
#'selected-window-accent-sync-tab-bar-to-theme--after)))
With this tweak in place, every time you change themes, your tab-bar instantly updates, colours stay consistent, clean, and theme‑accurate without you having to do anything at all! The downside to this of course is that any newer themes that were created after the advent of the tab bar in Emacs will have their tab-bar faces overridden, but for me this solution is good enough and gives a pleasant coherent visual tab bar experience.
Yay!, yet another Yak shaved!
For the previous post on file completion in Emacs I wanted to attach a short screencast demonstrating the feature in action.
I recorded it with macOS's Screenshot app and then converted into gif using ffmpeg, so that it plays automatically:
ffmpeg -i emacs-cape-file.mov emacs-cape-file.gif
By default, the resolution of the videos recorded by Screenshot is quite high, so no surprise that the output gif turned out to be huge 1.4M.
Even though it was only part of the screen recorded, the resolution was 1730x1658, which is way more than I needed to publish it on a web page.
This reduced the resolution to 768x736 and the file size to 427K, which is much better but still too big for such a short video.
Another thing we can do is to reduce the frame rate before converting to gif.
In screencast videos there are usually not many things that are changing between frames, so you may not even notice a difference:
However, it only reduced the file size by about half: 247K.
Converting the screencast using an encoding that supports compression, like H264, for example, would reduce the size to just 61K.
But I didn't want it to be displayed as video.
I've been using AVIF format for pictures on this blog for a while.
All of the pictures I attach are automatically converted into several options using @11ty/eleventy-img plugin and included into a page using <picture> tag.
Avif is a modern image format based on the AV1 video format.
It's supported by all major browsers now.
What I didn't know is that it also supports animated image sequences with proper compression and ffmpeg supports it too.
It is about five years ago that I submitted a package called setup to GNU ELPA1. The premise was simple, I had grown dissatisfied with use-package and wanted to implement my own “configuration macro” that would be a better fit for packages written in a good style.
In this post I’d like to look back on the package and the macro, to chronicle my experience and the lessons I took away.
Issues with use-package
Among people who like Emacs, especially those who avoid learning to use Emacs by configuring it, it is popular to recommend use-package when configuring Emacs. The idea is that you can avoid a lot of repetitive idiosyncrasies by instead having a macro translate a declarative-esque DSL into the right code. It also allows the user to request that a package be installed at startup if it hasn’t yet been installed.
It was written by the renowned John Wiegley around the early 2010’s — which is important because the context of the time is an important influence on the features it implements: Keywords like :autoload, :mode:interpreter or :magic are not necessary if working with Emacs packages that have properly configured autoloads and use these to adjust variables like auto-mode-alist. I also assume that the default behaviour, not to install packages, goes back to this.
Totalling at around 3500 lines of Elisp it is also not trivial to keep an overview of the code, and in my opinion it is also more tricky to extend with new keywords — at least compared to the ease at which you can define a new command.
Finally, something I never managed to figure out how to reasonably have a consistent configuration that would also use use-package for built-in, non-packages. You can add these kinds of configurations to the pseudo-package emacs, but that felt like a kludge to me.
Idea of setup
Superficially the syntax of setup is different but not a radical departure from the use-package approach. Instead of using keywords like
Note that the fact that the function symbol is a keyword is in no way significant. We could have also used &bind.
There were a few ideas and goals underlying the design:
It should be easy to add a new local macro or change the behaviour of an existing one. I created a EmacsWiki page for people to collect ideas. For the sake of convenience, a function setup-define helps set up a local macro and provides a few shorthands to enable implicit looping and coercion. As a side-effect, you also get Xref integration on local macros!
The setup macro is not declarative! Therefore it is nice if you can macro-expand the form and see what is going on. Take the two examples from above2:
The local macros use a separate, lexical scope to determine exactly what to expand to. Conventionally, the local context consists of the feature being configured (useful when wanting to generate code that should only be evaluated after the respective feature has been loaded), the current keymap, mode-hook, etc.
Here are a few examples from my init.el that I consider to demonstrate that applying these principles results in easy to read and write code:
For further details on the macro, I would recommend going to the EmacsWiki page linked above. The technical details are not of primary interest in this post.
A History of setup
Here I’d like to list a few events and comments in roughly chronological order:
Upon submitting the package to GNU ELPA, it was Stefan Monnier who made a few important and very helpful comments that influenced critical features of the package. For this, and all advice since, I remain extremely grateful!
The macro was featured in a stream later that year (2021) by David Wilson, on alternatives to use-package.
Many people were upset over the name setup, and tried to get me to rename the package; I on the other hand loath (package) names that were obviously just chosen to be easy to look up on a search machine.
The package was surprisingly popular on /g/. Some nice quotes:
“you should try setup.el sometime, its what use-package should’ve been”
“you should try setup.el next, the based non-bloated configuration package”
“Apparently, setup.el is better but use-package is built-in now”
The last quote references to a major development in Emacs 29, that use-package, which had a tricky copyright situation until then, was added to Emacs OOTB. I actually ended up helping out with the process, though it was mainly driven by Payas Relekar and Stefan Kangas. By this move, setup lost an edge since before you first had to enable MELPA (later NonGNU ELPA, if it wasn’t already enabled) to install use-package, while now you can just use use-package directly.
During the use-package merge discussion I had an exchange with John Wiegley where I demonstrated how an auxiliary macro could translate a use-package-like syntax into setup to simplify the processing. In the end the idea didn’t go anywhere.
I had a number of nice conversations with people who liked the macro, and multiple contributions by Okamsn (maintainer of the loopy macro, which I don’t think is coincidental).
There haven’t been many changes for the last two years. The last local macro I added was in 2024 (over two years ago at this point) and I have been deprecating more stuff since then (mostly macros like :option that are not context-specific and are better replaced using a more general macro like setopt), but haven’t removed it yet.
This is both a sign of decline and stability. While fewer people are using it, the fact that no further changes have been necessary is a sign that it does what it should. There are a few things I might change if setup is ever bundled with Emacs, for instance I might want to deprecate :when-loaded (the analogue of :config in use-package) because I think that anything you’d write within this local macro should either be replaced by a specific local macro or is a bad upstream design of the package maintainer.
The future of setup
I have to admit I am not as enthusiastic about the macro as I was 2021. It is not that I dislike it, the implementation still seems good and I enjoy reading it, but I am mostly indifferent about the need for configuration macros in general.
Over the last two years I have configured Emacs from start on two work machines, and both times I was more than just fine to append stuff to my init.el and use Easy Customization Interface. At the same time I continue to use setup on my personal machine, and as I grow ever more convinced of the fact that (obsessive) configuring of Emacs is a sign of a fatal misunderstanding of Emacs on some level, I don’t have the motivation to rewrite it because I don’t really have anything to gain from it. Most of the times I touch my init.el is to bind a new command, set a new user option or in the best case delete some code if an analogous feature has been upstreamed.
I don’t really know who uses setup anymore, but I am aware of people who have used it in the past and given up. Interesting nobody points me to a specific reason or issue, but my impression is that most people run into a similar kind of lethargic apathy, but decide to jump to something else instead (again, since the upstreaming of use-package I myself have considered switching a few times to simplify my configuration, but never gone through with it).
Finally, I recently suggested replacing the idea of setup macros with a more non-Lisp’ish configuration syntax, and explained why in the aforelinked article. I haven’t continued developing the proposal, though I still think it is more user friendly to someone who prefers to think of configuring Emacs not as programming with a understanding of what is going on but as toggling the right knobs to get the intended behaviour.
So the future is certain and reliable that the macro will stick around, and I guess I’ll keep on maintaining it?
Actually I had first tried to add it to MELPA a year before, but it got rejected.↩︎
Note that setting use-package-expand-minimally can help here↩︎
The Emacs Carnival for February is hosted by Sacha Chua and its topic is “Completion”. Below is my entry for this month's carnival.
Emacs has a myriad of ways to complete things for you in different contexts.
While the one I’d like to share with you may seem trivial, the feature is so ingrained into my workflow so that I even forgot that it’s not part of the core Emacs functionality.
I often switch from IntelliJ IDEA to Emacs (using a convenient shortcut) just to use the feature.
The feature provided by a wonderful cape.el package which extends the Emacs’ standard completion-at-point function with a bunch of handy completion functions or Capfs, one of which is cape-file.
What’s good about it is that it works in any mode, programming or not, and in any context be it a string literal or a comment.
I run the Regolith Desktop Environment on my laptop, which I love because it provides a convenient GNOME wrapper and interface for the i3 tiling window manager. Regolith relies on a program called ilia for application launching, and sometimes ilia gets caught in some kind of CPU-churning state that locks up my whole system. I have not been able to figure out what is causing it, so I (of course) turned to Emacs for a solution.
Turning to consult-omni
Armin Darvish has created a powerful Emacs package called consult-omni, which provides a wrapper around consult for searching through any number of information sources. I believe consult-omni was originally intended to query web search engines and document databases, but Darvish has also provided a search mode for your local desktop applications, and can act as an application launcher.
Darvish provides an example application launcher in his consult-omni YouTube tutorial. The source code is straightforward, but I wanted to tweak it just a little. You can view his original on the project’s wiki on GitHub. You can watch him explain his technique below.
After a few tweaks, here is what I came up with.
(defunconsult-launcher()"A launcher suitable for use from a window manager."(interactive)(let*((width(floor(*0.6(display-pixel-width))))(height(floor(*0.6(display-pixel-height))))(left(floor(*0.2(display-pixel-width))))(top(floor(*0.2(display-pixel-height))))(params`((name."omni-launcher")(width.,(cons'text-pixelswidth))(height.,(cons'text-pixelsheight))(left.,left)(top.,top)(minibuffer.only)))(frame(make-frameparams)))(with-selected-frameframe(select-frame-set-input-focusframe);; If i3 is running and there is a control socket, let's tell;; it we are a floating frame.(if(getenv"I3SOCK")(call-process"i3-msg"nilnilnil(format"[id=%s] floating enable"(s-trim(shell-command-to-string"xdotool getactivewindow")))))(unwind-protect(progn(consult-omni-apps-static".*"(propertize"> "'face'consult-omni-path-face))nil)(progn(when(frame-live-pframe)(delete-frameframe))nil)))))
I made two changes to get this to work nicely with i3. First, I removed the yequake dependency. Second, I added a call to i3-msg that sets the launcher frame as floating, which makes it much nicer to use. Like Darvish’s version, you can run this from the command line:
emacsclient -e '(consult-launcher)'
Adding an ilia fallback
Don’t tell all the other Emacs users, but I don’t have Emacs set up to launch automatically when I start my computer and log into X11. I probably should, huh? Also, there are times when I (gasp!) shut down Emacs, usually to restart it or fix something that I have broken. When those times happen, I want to be able to launch applications, so I need a failsafe in case consult-launcher isn’t available!
To solve this, I created a simple shell wrapper script, which looks like this:
#!/bin/bash# Check if Emacs server is running by looking for the server socket# Default server name is "server", but you can change this if neededSERVER_NAME="${EMACS_SERVER_NAME:-server}"SERVER_FILE="${XDG_RUNTIME_DIR:-/tmp}/emacs/${SERVER_NAME}"if[-S"$SERVER_FILE"];then# Emacs is running, use emacsclient to launch your application
emacsclient -e'(consult-launcher)'else# Emacs is not running, fall back to ilia
ilia -p apps
fi
If you want to use this, the important part is that SERVER_FILE points to the socket that your Emacs server uses. Make sure that emacsclient and ilia are both in a reasonable location so your shell can find them, then bind this command to whatever you usually use to launch ilia.
By the way, if you are using Regolith’s normal method of launching ilia, you can add your shell script to your Regolith configuration pretty easily. Open $HOME/.config/regolith3/Xresources in your text editor, and add the line:
You can then run xrdb -override $HOME/.config/regolith3/Xresources and it should work! Good luck.
Drawbacks
One of the nice things about ilia is that it keeps track of applications your run frequently, so they tend to bubble up to the top of its application listing. The Emacs method doesn’t do that. I don’t mind so much, I always end up typing in application names. It is fun to use Emacs as an application launcher, and I hope that it helps me avoid the CPU-churn problem that ilia has been experiencing far too often.
Have I come up with a clever solution, or a lazy workaround? I’m looking forward to hearing your thoughts.
Bash-based little brother of org-capture. Sort of. Emacs not needed: from anywhere, add to the end of an Org file of your choice a subtree with the contents of your clipboard.
I while back I wrote up a sketch to interpret Ini-style configuration files for Emacs. Here I want to present and discuss the idea briefly.
Syntax
For those wondering how a “ini-style init file” looks like (I certainly would), I translated parts of my init.el into a syntax that ini.el can handle, to give an idea of how these files look like:
enable inhibit-startup-screen
set mode-line-compact long
set font-use-system-font t
disable menu-bar-mode
bind C-c k compile
eval set compile-command (format "make -k -j%d " (num-processors))
[package avy]
unset avy-single-candidate-jump
global bind C-z avy-goto-word-or-subword-1
[/package]
add-hook text-mode-hook flyspell-mode
add-hook prog-mode-hook flyspell-prog-mode
[feature flymake]
bind M-n flymake-goto-next-error
bind M-p flymake-goto-prev-error
(defalias 'list-issues #'flymake-show-buffer-diagnostics)
[/feature]
[package bash-completion]
(bash-completion-setup)
[/package]
[package proof-general]
[package focus]
[package markdown-mode]
add-hook before-save-hook time-stamp
The idea here is to do the right thing most of the time, while still allowing for arbitrary complicated configurations using Eli’s, as lines that start with an opening parenthesis are just regular Lisp expressions that treated just like they would be in an init.el.
More details on the syntax can be found in the ini.el file under the “Syntax” section.
Comparison and potential advantages
There are three main approaches to configuring Emacs that one can compare this idea to:
A plain and simple init.el,
The usage of configuration macros like the well known use-package or my not well known setup.el,
A customize-centric approach, which usually means not writing Elisp on your own.
I consider my approach to be a merge of the first two. Settings like bind and set are mapped directly to their obvious corresponding S-expressions. Depending on the exact command and modifiers the concrete replacement might differ, but the point is that it does what you want it to do. You cannot bind keys in a map before the map has been loaded, so the bind M-n flymake-goto-next-error above will expand to
You can of course just write this out directly, as any line beginning with an opening parenthesis is read directly and interpreted directly. The connection to configuration macros is that this allows us to just express what we want to (“bind this command”), without having to write what we have to (“… after loading this feature”).
The main advantage of my advantage I see is that the syntax is more conventional and thus easier to adopt for some people who are not yet familiar with Lisp, and haven’t internalised what to quote and when. I would claim that for most packages, the keywords I have defined up until now will cover well over 90% of the use-cases. The fact that set expands to setopt also means that it is easy to respect user options as user options and not mistreat them as symbols. Prior to the addition of setopt in Emacs 28, it was necessary to use customize-set-variable, which much like set required user option names to be quoted and would only set a single user option per expression. I think a good maxim of design is to make it easy to do the right thing, where “easy” and “right” are intentionally kept vague. Looking through my init.el, I notice patterns such as the fact that most user options I set are either self-evaluating forms or quoted, so it made sense for me that set would be non-evaluating by default as well.
Another advantage of the simpler but rigid syntax is that I can imagine an extension to customize that can reasonably figure out what to change in a init.ini file, which is much more difficult in a general elisp file, as we have no notion of structure such as the feature or package blocks I have proposed above. This would serve to bridge the gulf between generated custom-set-variables blocks that many users sadly discard by setting custom-file to "/dev/null" and the manually written configuration file. I have heard many people dismissing the customize interface just due to the reason that it generates code that they do not want in their init.el, which I think is a shame.
So now what?
The initial sketch appears to work, though I have to admit that I am not using it personally yet, as I don’t have a need to switch myself and seldom adjust anything anymore in my init.el (remember, compulsive and constant tweaking of configuration files is a sign that you are doing something wrong!). So I would be interested in the opinions of new and intermediate users as to the potential advantages and pitfalls of this idea. Feel free to send me a message with your unfiltered opinions or to share this page with someone who might be interested.
If this idea seems interesting, it would be worth developing into a proper package, with good error messages and syntax highlighting. If it is well received and reliable, it would be interesting to consider merging it into Emacs itself one day.
Publify is the Rails based blogging engine that started out as Typo, which I ended up maintaining for a while before handing it off to Frédéric de Villamil, who is still on the current maintenance team. Go him!
incarnation to a static site generated from markdown files by Hugo. In that migration, I fucked up and truncated a buttload of posts and didn’t realise what I’d done until long after (about a week ago now) I had misplaced the database that the site had originally been generated from.
Oops.
However, the Internet is still a marvellous place, blessed with useful sites like The Wayback Machine, which lets the interested reader browse historic versions of web pages. Which means, provided a page got noticed by archive.org’s crawlers, I can fetch a page from back before I fucked up and, with a little bit of massaging, turn it into something that Hugo can understand and get the whole article back again.
I can even recover the comments, which I had deliberately left out of the initial import, thinking “I’ll get around to importing those as well one day!” Thanks to my ADHD, that never quite happened. Unless you count the current activity, of course.
“Provided” is doing a lot of work there, and some posts definitely got missed, but something is better that nothing.
I’ve reached the point in my recovery process that I’ve started to hate the ad hoc way I was grabbing stuff from the archive. I want to only grab an archived page if the archived version is from before I buggered things up. So I’ve written some Emacs lisp. Obviously.
The idea being that I fetch the archived version of the post into a temporary buffer where I can run it through HTML Tidy and a few extr HTML cleanup steps
An ever expanding list of cleanup steps. Every time I have to tidy something up by hand for the second time, I add something to the cleanup pipeline.
before converting it to Org format with Pandoc, continuing the cleanup in org mode (I prefer Emacs org mode tooling to its HTML tooling) and returning a nice clean string to insert into the org capture buffer.
I plan to return to the cleanup in future articles,
but we’re just concerned with with-wayback-page-from-before for now.
If you want to see the full code (along with way more stuff), you’ll find it in my dotemacs repo on Github. I’m exceedingly unlikely to turn this into a full wayback.el package, but you’re more than welcome to use it as a starting point.
It’s a macro that uses wayback-get-capture-before to find the URL of the most recent capture of our target url before the given date, fetches it into a temporary buffer and executes the body of the macro. A common Emacs pattern.
do M-x describe-function and type with- and check out the completions to see just how common
The real meat lies in wayback-get-capture-before, which uses the Wayback CDX Server API to discover the capture url we’re interested in. There are other, simpler to use Wayback machine APIs, but they only let us find the closest capture to our date of interest, and we want to find the most recent capture that’s strictly before our date and that requires the CDX API. I’ve been a little lazy and used the request package to do the web request stuff because I prefer its API to the native url-retrieve in vanilla Emacs.
(defvarwayback-cdx-endpoint"https://web.archive.org/cdx/search/cdx""The endpoint for the Wayback Machine's CDX server.")(defvarwayback-cdx-json-parser(apply-partially'json-parse-buffer:array-type'list)"Parser for json data returned from the CDX server.")(defunwayback-get-capture-before(urldate)"Use the CDX applet to find any version of URL captured before DATE string.
Returns nil if there's no such capture"(let((capture-urlnil))(requestwayback-cdx-endpoint:params`((url.,url)(to.,(if(or(numberpdate)(stringpdate))date(format-time-string"%Y%m%d%H%M%S"date)))(collapse.digest)(output.json)(fl."timestamp,original")(limit.-1)):parserwayback-cdx-json-parser:synct:success(cl-function(lambda(&keydata&allow-other-keys)(setqcapture-url(pcase(cadrdata)(`()nil)(`(,timestamp,target-url)(s-lex-format"https://web.archive.org/web/${timestamp}/${target-url}")))))))capture-url))
The CDX API’s JSON response format is derived from a CSV style text file. We’re only really interested in the timestamp and the “original” url that our target url resolved to, so we set (fl . "timestamp,original") in the request parameters and limit the results to the most recent one ((limit . -1)) before ((to . ...)) the date we’re interested in. That gives us:
which passes an empty list through, or grabs the timestamp and target-url from the second entry in the results list and uses s-lex-format to generate a wayback machine URL. Easy.
This has the makings of a more general package,
but that’s very much a back burner project. It does what I need, and does it well enough that I can consider this yak shaved and get on with the job of recovering my truncated blog posts. I’ll continue on my way of not releasing anything that anyone might want me to support.
We explain how to test a new feature of Emacs, in development, which
makes Emacs more responsive.
This is a new Garbage Collector which you can try as part of the IGC
branch, and instructions are provided.
(...)
When last we left off we had worked out how to grab all the mentions of this site that Webmentions.io knew about and now we want to write that out to the data/ directory in a format that’s easy to deal with in Hugo.
If you’ve read my earlier note, you’ll know that I’ve been evolving the data schema towards something that’s easy for Hugo to deal with and reasonably comprehensible for me too.
As things currently stand, I’ve settled on dropping all the mentions in a single file, data/mentions/all.json
I’d rather use data/mentions.json, but Hugo’s data system doesn’t seem to pick that up, so I’ll live with the slightly more clunky option.
which is structured along these lines:
{"/note/7/":{"like-of":[{"type":"entry","author":{"type":"card","name":"Daniel Kelly Music","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png","url":"https://aus.social/@yasslad"},"url":"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078","published":null,"wm-received":"2025-09-07T01:31:46Z","wm-id":1936817,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078","wm-target":"https://bofh.org.uk/note/7/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/7/","wm-property":"like-of","wm-private":false},{"type":"entry","author":{"type":"card","name":"Jess Robinson","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg","url":"https://fosstodon.org/@castaway"},"url":"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318","published":null,"wm-received":"2025-09-07T07:19:01Z","wm-id":1936873,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318","wm-target":"https://bofh.org.uk/note/7/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/7/","wm-property":"like-of","wm-private":false}],"in-reply-to":[],"mention-of":[],"repost-of":[],"other":[]}}
Just imagine that JSON object has a bunch more paths as keys referencing similar objects keyed by mention type.
As it stands, wm--fetch-all is returning a flat sequence of webmention objects that we want to process into a more structured object,
JSON/Javascript calls them objects, old Perl heads like me think of them as hashes, and they’re “Hash Tables” in Emacs Lisp. I’ll be calling them hashes from now on.
in other words we want to fold (or “reduce” in Emacs Lisp terminology) the sequence into a hash. And I know just the function for that. Let’s see what describe-function has to say about seq-reduce:
seq-reduce is a byte-compiled function defined in seq.el.gz.
*Signature*
(seq-reduce FUNCTION SEQUENCE INITIAL-VALUE)
*Documentation*
Reduce the function FUNCTION across SEQUENCE, starting
with INITIAL-VALUE.
Return the result of calling FUNCTION with INITIAL-VALUE
and the first element of SEQUENCE, then calling FUNCTION
with that result and the second element of SEQUENCE,
then with that result and the third element of SEQUENCE,
etc. FUNCTION will be called with INITIAL-VALUE (and then
the accumulated value) as the first argument, and the
elements from SEQUENCE as the second argument.
If SEQUENCE is empty, return INITIAL-VALUE and FUNCTION
is not called.
This does not modify SEQUENCE.
and that will handle the business of iterating over the sequence of mentions for us, and all we have to do is write wm--add-mention-to-hash-table to populate the hash we made with (make-hash-table :test 'equal)
We need to use that :test ’equal part because json-insert wants a hash with strings as keys and the default hash returned by (make-hash-table) compares keys using eql which might or might not work when comparing strings. Not a problem which equal has.
one mention at a time, and return the modified hash (You and I both know that it’s the same old hash mutated, but let’s pretend it isn’t, eh?).
What does that function look like? Here’s what I wrote:
(defunwm--add-mention-to-hash-table(accmention)"Helps reduce a list of mentions into a two level hash."(require'dash)(let*((path(-->mention(gethash"wm-target"it)(url-generic-parse-urlit)(url-path-and-queryit)(carit)))(mentions-hash(or(gethashpathaccnil)(wm-new-mentions-hash)))(mention-type(gethash"wm-property"mention))(mentions(or(gethashmention-typementions-hash)(progn(setqmention-type"other")(gethashmention-typementions-hash))))(new-mentions(if(seq-containsmentionsmention)mentions(vconcatmentions(listmention)))))(puthashmention-typenew-mentionsmentions-hash)(puthashpathmentions-hashacc)acc))
We grab the path from the "wm-target" key, which is actually a URL rather than a simple path
We could just use the URL, and that would work fine on this site, but not when I’m running on localhost. The path will always match with .RelPermalink, but the host part of .Permalink is different in development than in production.
we’ll thread mention through that series of transformations using dash.el’s threading macro, -->.
We use the path to grab mentions-hash from the acc-umulating hash and, if there isn’t already one there, we grab an empty, but structured hash using wm-new-mentions-hash, which looks like this:
(defunwm-new-mentions-hash()"Make a new empty hash to hold categorised webmention data."(copy-hash-table#s(hash-tabletestequaldata("like-of"[]"in-reply-to"[]"mention-of"[]"repost-of"[]"other"[]))))
Now we look up "wm-property" in mention, and use that to grab its associated vector of mentions. Well, we would, but there’s a small wrinkle.
We’re only currently interested in four kinds of mention,
but Webmention.io doesn’t know that. We could throw the extras away, but what if we became interested in bookmark-of mentions or whatever somewhere down the road. So let’s collect them under the other key. Which is where this hacky section of our let* form comes in:
First, we make a guess at the mention-type we’re going to file the current mention under by grabbing the "wm-property" and use that value to lookup the mention type in mentions-hash. If it’s one of the four types we’re interested in, that will be a vector, which is truth-y, otherwise we get nil, which is false-y so we change the mention type to “other” and grab that vector from the mention hash.
We now know the key path we’re going to store our mentions in, and we have the current vector of mentions associated with it. So, if we already know (seq-contains mentions mention) about the current mention, we reuse that, otherwise we make a new vector with the current mention added to it.
That done, it’s a simple matter of putting the new mentions vector into our mentions hash, putting the mentions hash into our accumulating hash and returning that.
With that done, it’s a simple matter of opening data/mentions/all.json, erasing the buffer, calling (json-insert (seq-reduce ...)) to update the data and saving it. Here’s the code which does exactly that.
(defunwm-unflatten-mentions(mentions-vec)(seq-reduce'wm--add-mention-to-hash-tablementions-vec(make-hash-table:test'equal)))(defunwm-fetch-mentions()"Fetch the webmentions of `wm-domain'."(interactive)(save-current-buffer(let((all-entries(wm--fetch-all)))(with-temp-file(expand-file-name"all.json"wm-data-dir)(erase-buffer)(json-insert(wm-unflatten-mentionsall-entries))))))
Over in the Hugo partial
that renders the bit of the page immediately after this, we can get at the data like this:
I’ll leave the rest as an exercise for the interested reader. However, I will note that the Webmention.io API includes the option to pass in a since argument, so it wouldn’t be hard to write
Mondo can translate into (almost) every language the names of the world's scripts, languages, territories (including countries), and countries' subdivisions. Translations happen locally — in your machine.
Do you have fonts that can display emoji flags? Vexil can convert from country code to country flag, and vice-versa. Likewise for countries' subdivisions.
You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago,
I turned off the home server that was handling the web hook calls from Webmention.io, planning to quickly move it and spin it up again. Ask me how that’s going.
but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.
Start by faking it
I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.
# Grab the most recent 5 webmentions of bofh.org.uk
GET https://webmention.io/api/mentions.jf2?domain=bofh.org.uk&sort-dir=down&per-page=5&token=:token
Which produces the following JSON data:
Disclose this for the full wall of JSON
Don’t say you weren’t warned!
{"type":"feed","name":"Webmentions","children":[{"type":"entry","author":{"type":"card","name":"Mike Spencer","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg","url":"https://mastodon.scot/@mikerspencer"},"url":"https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190","published":null,"wm-received":"2025-09-07T09:35:22Z","wm-id":1936897,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190","wm-target":"https://bofh.org.uk/note/8/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/8/","wm-property":"like-of","wm-private":false},{"type":"entry","author":{"type":"card","name":"Jess Robinson","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg","url":"https://fosstodon.org/@castaway"},"url":"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318","published":null,"wm-received":"2025-09-07T07:19:01Z","wm-id":1936873,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318","wm-target":"https://bofh.org.uk/note/7/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/7/","wm-property":"like-of","wm-private":false},{"type":"entry","author":{"type":"card","name":"Daniel Kelly Music","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png","url":"https://aus.social/@yasslad"},"url":"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078","published":null,"wm-received":"2025-09-07T01:31:46Z","wm-id":1936817,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078","wm-target":"https://bofh.org.uk/note/7/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/7/","wm-property":"like-of","wm-private":false},{"type":"entry","author":{"type":"card","name":"Nick Anderson","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg","url":"https://fosstodon.org/@nickanderson"},"url":"https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511","published":null,"wm-received":"2025-08-28T21:48:52Z","wm-id":1934308,"wm-source":"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511","wm-target":"https://bofh.org.uk/note/6/","wm-protocol":"webmention","like-of":"https://bofh.org.uk/note/6/","wm-property":"like-of","wm-private":false},{"type":"entry","author":{"type":"card","name":"Nick Anderson","photo":"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg","url":"https://fosstodon.org/@nickanderson"},"url":"https://fosstodon.org/@nickanderson/115108589417986943","published":"2025-08-28T21:48:05+00:00","wm-received":"2025-08-28T21:48:51Z","wm-id":1934307,"wm-source":"https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711","wm-target":"https://bofh.org.uk/note/6/","wm-protocol":"webmention","content":{"html":"<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>","text":"@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"},"in-reply-to":"https://bofh.org.uk/note/6/","wm-property":"in-reply-to","wm-private":false}]}
Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn jq again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the default url-retrieve interfaces, I went and grabbed the emacs-request package instead because I found its API more comprehensible.
Because request is a function where restclient is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.
The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary :parser function, it’s trivial to convert the returned JSON into a native Emacs lisp hash table
It’s not hard to generate an old school alist either, but that was annoyingly hard to serialise back to JSON, so I went with the default types because they seem to just work.
which looks a bit like this:
A wall of Lisp
#s(hash-table test equal data
("type" "feed" "name" "Webmentions" "children"
[#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Mike Spencer" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg"
"url"
"https://mastodon.scot/@mikerspencer"))
"url"
"https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190"
"published" :null "wm-received"
"2025-09-07T09:35:22Z" "wm-id" 1936897
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190"
"wm-target" "https://bofh.org.uk/note/8/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/8/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Jess Robinson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg"
"url"
"https://fosstodon.org/@castaway"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318"
"published" :null "wm-received"
"2025-09-07T07:19:01Z" "wm-id" 1936873
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Daniel Kelly Music" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png"
"url"
"https://aus.social/@yasslad"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078"
"published" :null "wm-received"
"2025-09-07T01:31:46Z" "wm-id" 1936817
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511"
"published" :null "wm-received"
"2025-08-28T21:48:52Z" "wm-id" 1934308
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/6/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://fosstodon.org/@nickanderson/115108589417986943"
"published" "2025-08-28T21:48:05+00:00"
"wm-received" "2025-08-28T21:48:51Z" "wm-id"
1934307 "wm-source"
"https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "content"
#s(hash-table test equal data
("html"
"<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>"
"text"
"@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"))
"in-reply-to" "https://bofh.org.uk/note/6/"
"wm-property" "in-reply-to" "wm-private"
:false))]))
Verbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.
The interesting stuff lives under the "children" key, which we can get with (gethash "children" data).
To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with
Then turn what we learn into a commands
Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function, wm--fetch-all to do that for us.
The (while more? ...) loop keeps requesting more data until it gets a short response, at which point more? becomes false and we return the accumulated all-entries vector
And relax
We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s data directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.
The following is a contribution to the Emacs Carnival, specifically topic set of August, 2025: “Your Elevator Pitch for Emacs”. I have to be honest that I don’t like participating in these kinds of trends, but the idea of having a shared writing prompt is interesting so I’d like to contribute my take on how I “sell” Emacs.
For the quick and short “elevator pitch”, I will translate/paraphrase a snippet from an Emacs course I organised last year. The bulk of the article will consist of my footnotes elaborating on my specific choice of words:
Emacs aspires1 to be a “Emancipatory Computing System”2, i.e.
it has a uniform user interface3, where intuition accumulates4,
it has transparent5 access6 to its own source code7 and documentation8,
It has a community9 of hackers10 that share their extensions11 and experiences1213.
So if that sounds like something that interests you, you might like Emacs.
The phrase “aspires to” indicates that GNU Emacs (or any other Emacsen for that matter) doesn’t satisfy the conditions I give. It too has historical cruft and limitations that prevent it from unfolding the full potential of the ideal I describe.↩︎
A matter of attribution: I have to thank Florian for helping me come up with the term “Emancipatory Computing System”, that one can conveniently abbreviate as “EmaCS” 🙂.↩︎
The “uniform user interface” of Emacs is text-based, and is mostly manipulated using key-chords. This specific choice is something that makes sense for GNU Emacs, considering how it plays a role in GNU’s Unix implementation (remember: Emacs is more of a shell than just a text editor or something like an OS!), but I don’t think that it is inherently a superior choice to anything else.↩︎
To illustrate what “intuition accumulates” means, I like to remember a moment while still acquainting myself with Emacs: I had just discovered or understood dabbrev-expand (M-/) a short while ago. While writing some HTML I wanted prompted via the minibuffer to input a color code. The color code I wanted to use was already on the display. Despite not yet thinking of the minibuffer as just another text buffer, I typed the first two characters and pressed M-/ and Emacs did what I mean, by expanding it to the right key. A while later I also understood that I don’t need a special key to undo a unwanted dabbrev expansion, as C-/ (i.e. undo) does the job just as well↩︎
Read “transparent” as in you appear to be able to “see through”. Contrast it with an “opaque” system, where you cannot directly see what is going on or what you are changing. Some people like to use phrases like living or malleable. To me the most important aspect to this is the possibility of a quick feedback loop that lowers to boundaries to proactively changing your environment instead of accepting accidental inconveniences.↩︎
It should be clear here that “access” ought not only to refer to reading but also to writing. In my case, Emacs was the first medium by which I actually understood to live the principles of Free Software, instead of just abstractly agreeing with them. The latter is what I see as the cause of “open source”, which in my eyes in a development strategy for projects that doesn’t have an inherent interest in inviting users to invest time to change anything for their own sake.↩︎
On the matter of Emacs’ source code: There is sometimes a discussion on the “choice” of (E)Lisp, as opposed to a more “mainstream” language like JavaScript or Python. I do understand where this objection is coming from. If you want to make the power of people accessible to as many people as possible, it appears reasonable to use a language that as many people as possible already speak. To avoid being misunderstood, I want to emphasise that I am by no means a Lisp mystic! I don’t know if it manages to really hit a unique balance between minimalism and expressiveness. What I do think that differentiates Lisp from other programming languages, is a attitude to “language” in a more general sense. This is something I think Lisp shares with the APL or Forth traditions, though obviously in different ways. At least Forth and Lisp both have an amalgamating approach, where boundaries between a core language, a standard library and a program dissolve: Programming is the problem of finding the right way to express a computation; or facing the consequences of not having done so. What this means for Elisp is that there is a constant drive to improve the language of text manipulation and OS-interaction.↩︎
It is important to emphasise that Emacs has a “documentation culture”. Documentation strings are pervasive and held to high regard. Perhaps I am biased, but a script that doesn’t bother to properly document functions has something insulting about it — which is not to say that I do not understand that writing documentation is not easy, especially good documentation. Just like with regular commentary, it is tricky to mentally invert ones perspective and explain what a reader would want to know, instead of what you know.↩︎
I don’t like to phrase “community”. Not because of the phrase itself, but because I feel that it has been co-opted to have a corporate and dishonest after-taste. It is not just a group of people with a shared activity, in the plainest sense of the phrase. Instead the German phrase “Gemeinschaft” appeals to me much more, as it emphasises a personal aspect to the relations of people with shared beliefs and values. I would say that this doesn’t have to be a global Gemeinschaft. While there is an international group of people that interact online, in my estimation it doesn’t exceed a few hundred at any given point, where most are probably only familiar with a few dozen of the most prominent and active members. In fact, it is probably more powerful if you get to partake in it with people you know and interact with in your everyday life.↩︎
I did hesitate when translating this, as I have recently been pondering about Joseph Weizenbaum’s Criticism of Hackers, thought the following doesn’t directly relate to the core of his argument. This is something I would have to elaborate on somewhen else, to give a sketch of what is causing my unease: Hackers are weird (read awkward, even unpleasant in a non-subversive way) as some of their idiosyncrasies can frighten away and estrange “average” people from engaging with otherwise general questions that do pertain to their everyday lives as people living in computerised societies. If we go by the standard definition of Hackers being people who enjoy a certain kind of problem solving, then this would be less of an issue, but “hacker-culture” does.↩︎
This is also a subpar choice of word, as Emacs is not “extended” in the sense of the extension being separate to something prior. Instead it is extended in the same way as population of a pond is extended by another fish you might throw into it: It might be a harmonious addition, that enriches the overall appearance, or it might turn bad if the new element disturbs the existing ecosystem. There is a real risk, that a more conventional “add-on” system avoids by means of separation and distance. Perhaps there is a better way to imbue the distinction, the danger and the potential?↩︎
Here I want to bring together the two points of “Gemeinschaft” and Free Software commented on above: The act of sharing experiences, which can take the form of simple “Did you know that…” or “Let me show you how I solved …”, and continue on to more direct mentoring/teaching relationships. Note that experience doesn’t necessarily have to be transferred by speech (be it written or verbal), but also in form of scripts: A clever script might even go by unnoticed or unappreciated for a number of years — but when rediscovered by the right person, can flip a mental switch. We might be talking about a clever use of some existing functionality (for instance, I recall being in awe when seeing Daniel Mendler’s idea to use replace-regexp-in-string as an ad-hoc template system), or make an idea that might at first appears abstract, suddenly practicable (such as the distinction between explicit and implicit hyperlinks; the latter occurring reoccurring in different places over the history of user interfaces, such as in Plan 9, Android or Embark)↩︎
What I mean to emphasise here is that without a shared repository of experience, that might take the form of accumulated folklore or package archives, one fails to achieve a inertial mass required of an Emancipatory Computing System. This is where most Emacs-clones currently still fail, as they are still developing in their larva stages. It is therefore worth emphasising that Emacs is not inherently special. I don’t know if it will be used as such in 100 years. Thinking about Human-Computer interaction over that period of time is inherently strange. For now it is what i am stuck with, and the point of describing what I like in Emacs in these abstract terms, is to clarify what an eventual successor would have to surpass GNU Emacs in. Critically this perspective also avoids mistaking accidental properties of present Emacs such as Org or Magit as selling points, which I consider to be a common mistake that is popular amongst newer proselytizers.↩︎
I’ve been dealing with some enduring niggles in my literate Emacs configuration, and have just landed Commit dbc316b, and I thought I’d write about it here because it’s an example of what keeps me on Emacs more than thirty five years after I first started using it.
Back when I started, Emacs was merely a text editor that felt easier to use than vi, but these days, it’s all that and more. I don’t live in Emacs quite as much as I used to; I don’t hang on USENET or IRC any more, and I check my email as little as possible, so I’ve not got around to configuring Emacs for that again.
Back in the day, it all went through the behemoth that is GNUS, but I lost that version of my .emacs file several computers and ISPs ago.
These days, what I like about Emacs is it’s malleability. It’s not uniquely malleable, either. Vim and Neovim diehards will no doubt have tales to tell about their setups, but Emacs is my editor. There are others like it, but this is mine.
I’ve been seduced by the Literate Programming idea of maintaining my ~/.emacs
Emacs used to keep its config in a file in your home directory called .emacs, then it moved to ~/.emacs.d/init.el, which made it a little easier to break a big config out into multiple files. These days, Emacs looks first in ${XDG_CONFIG_HOME}/emacs, which defaults to ~/.config/emacs and that’s where my config files live today.
file in a single org-mode file, which gets ’tangled’ into early-init.el and init.el, which are the files that Emacs actually loads.
One feature of Org mode’s literate programming support that I rather like and take advantage of is the ability to write things out of order and assemble them correctly using noweb. The idea is that src blocks can be named, and then referenced from other source blocks. For instance, you might have:
#+begin_src emacs-lisp
(dolist(template`(("t""Task with annotation"entry(file,pdc/org-inbox-file)"* %?\n:PROPERTIES:\n:created: %U\n:END:\n\n%i\n\n~ %a":prependt)))(add-to-list'org-capture-templatestemplatet(lambda(ab)(equal(cara)(carb)))))
#+end_src
which will eventually be tangled as a dolist that adds all your capture templates to org-capture-templates.
That <<capture-templates>> is what makes the magic happen. When we tangle README.org, the exporter gathers up the contents of any source blocks associated with the tag and replaces <<capture-templates>> with them
It applies some heuristics as well, which is why that dolist isn’t indented exactly how it would be without the noweb reference, but you can read the docs to find out more.
So, elsewhere in my README.org, I can keep capture templates related to a particular app close to the rest of the configuration of that app, where it makes sense to me. For example, in the section where I configure my blogging tools, I do this:
Notice the header arguments. :tangle nil tells the exporter not to simply write the code out at the current position, and :noweb-ref capture-templates tells the exporter to instead write it out wherever it sees <<capture-templates>> in a source block.
We have to be careful to ensure that the including code gets run when and where all the values variables and functions used in the included fragment are in scope, which can be fiddly, but it’s definitely doable.
Which brings me to what I actually want to write about!
I don’t type all that #+begin_src … stuff out by hand every time. Because doing that is simply asking for errors. Instead, I take advantage of an org feature called structure templates, so I have:
Then I’d fill in the correct :noweb-ref and the appropriate code.
It was fine.
Well, it was better than typing it all by hand, but as soon as I’d expanded a template, I’d immediately hit C-c ' to edit the code block in a separate buffer that was in the correct mode to edit Emacs Lisp (or whatever language the source block was for).
Eventually, I got annoyed enough by the repetition to work out how to make that happen automatically. Normally, I’d expect to add a function to a hook variable somewhere, but that doesn’t quite work here. Time to break out the Swiss Army Knife that is advice-add.
We’re adding advice to two different functions here, because there are two different mechanisms for inserting a structure template, either via the <foo expansion, or by calling M-x org-insert-structure-template, both of which I make use of on different occasions.
This version lasted a while.
It did 90% of what I wanted after all. Indeed, for most of my structure templates, it does 100% of what I want.
But there’s always that one case, isn’t there? Take a look at that ett template from earlier. Notice that it’s missing the value to assign to :noweb_ref. In a perfect world, we should fill that in before we start editing the code. Or rely on remembering to do it after the fact. Because we always do that, don’t we?
So, this morning, that chunk of code looked a little like this.
I’ve had to reconstruct the code from memory, I’m afraid because it never made it into the git repo. You can call me sloppy if you like, but this is personal code written on my own time, so you can fuck off.
Because an org file is Just A Text File™, we could have written this using Emacs’ basic buffer editing commands, but we’ll take advantage of some of org and org-babel’s helper functions to make life a little easier and (hopefully) to help me understand what I’m doing and why I’m doing it when I come back to the code later. Named behaviour is great
Especially in a self-documenting editor like Emacs. If I’m not sure what a particular function in this code does, the documentation, or even the source code, is always a couple of keypresses away.
for helping make code more understandable.
What we do here is save our place in the buffer, jump back to the very beginning of the source block and look at the header arguments. If they end with a property name (:like-this), then we deduce that more information is needed, so we use read-from-minibuffer to ask for it, add the answer to the end of the header arguments, jump back to wherever the template originally left us (by exiting the save-excursion block) and call org-edit-special to start editing the file in a dedicated buffer.
Job jobbed, no?
Well… kinda. See, we’re filling in the value of :noweb-ref using an error prone free text value, rather than presenting a list of known noweb references to choose from. In the case where the unset parameter is :noweb-ref, we really want to use completing-read. A quick trawl through the existing code didn’t find a function to do what we want, nor did a web search. An org file’s Just A Text File though, and org-babel-noweb-wrap returns a regular expression that will match a noweb reference in the current file
<<reference>> is the default form for a noweb ref, but the delimiters are overridable. At one point, I was using «reference» rather than the defaults, so it’s handy to have a function that deals with that for us.
so we could save our place, jump to the beginning of the file and find every match for that regular expression and use that to build a completing-read candidates list. But <<foo>> is only a noweb reference if it’s in a source block, so we could end up with a bunch of false positives. We want to search through every source block, ignoring the rest of the file. Surely there’s already something in existence to let us do that since it’s the sort of thing that happens during the process of tangling a file.
A quick M-x describe-function org babel src block yields a bunch of interesting functions, including the promising sounding org-babel-src-blocks. The documentation reads:
Signature
(org-babel-map-src-blocks FILE &rest BODY)
Documentation
Evaluate BODY forms on each source-block in FILE.
…
It goes on to explain that the body is evaluated with some useful variables set. The one we’re interested in is body, which is a “string holding the body of the code block”. Sorted.
With that, and other helper functions, we can write:
(defun+org-babel-noweb-refs()"Find all the noweb refs in the current buffer."(require's)(require'dash)(let((match-exp(org-babel-noweb-wrap))result)(org-babel-map-src-blocksnil(let((plain-body(substring-no-propertiesbody)))(setqresult(-concatresult(-map(-partial#'s-replace"(.*)\\'""")(-map#'second(s-match-strings-allmatch-expplain-body)))))))(-sort#'string<result)))
It’s a bit unsubtle, but it’s quick enough and accurate enough for my purposes. Then we can rewrite the relevant bit of our advice function along these lines:
Note the use of cond here even though we could use a single if. I’m making it easier to special case behaviour for header arguments other than :noweb-ref. I’m probably not gonna need it, but it’s easy enough to be kind to the future me who does need it.
I’m sure your editor of preference
can do something like this. If it can’t, then why on earth do you put up with it? I know Emacs though, and it it’s taken me longer to write about this whole process of eliminating a bump in my road than it did to implement the necessary functions and advice in the first place. I didn’t even have to restart Emacs, and it remained usable throughout the process
The code was actually written within a source block
but it got smoother and smoother with every step.
I’m not the first person to point out how powerful Just A Text File can be, especially if you’ve also got a huge pile of functions at your disposal to manipulate that file in useful ways and with mode specific semantics. Provided, of course, you’ve got decent tools to search through that pile. Emacs is just such a pile of useful functions and is a great tool for sifting through it. Give it a go, why don’t you.
Take the time to look at how you use your editor. I’m sure some point of friction will come to mind. Then work out how to write some code that will make things a bit smoother. You’re not looking for perfect here, you’re looking for better. The remaining roughness will no doubt niggle at you enough for you to take another pass at sanding it down one day, but for now luxuriate in the fact you’ve made life a little better for yourself.
I’d suggest that, if you keep doing that as part of your practice, you’ll eventually have your tools working exactly how you want them to. But I’ve been an Emacs user since 1988 and I’ve yet to reach that point. But it’s about the journey, not the destination, isn’t it?
I work solely inside of Emacs, so when Emacs is down, I cannot do any
work. Emacs itself is very reliable, but there might be some risks
of downtime when upgrading Emacs or any of the 3rd party libraries
that I use.
The downtime can be minimised by always having multiple Emacs versions
and their 3rd party libraries available. This blog post documents
how I implement it.
Installation
Firstly, install each Emacs into its separate folder, e.g. on my
Debian box, I have ~/bin/emacs30.0.92/ installed 8 months ago and
~/bin/emacs30.2/ installed yesterday. This is easy to achieve by adding
the prefix option when building Emacs from source, e.g.
Then have a separate systemd service for each Emacs version. Taking
version 30.2 as an example, its unit file is saved as
~/.config/systemd/user/emacs30.2.service.
In that unit file, the Emacs executable is specified in full path to
wherever it is installed
In the unit file, I also added the initial optioninit-directory so
it has its own .emacs.d directory. It ensures the 3rd party packages
will be installed there.
Note if there is an init.el file in that directory, Emacs will use
that instead of the ancient ~/.emacs file.
GUI
Finally, to open an Emacs GUI that connects to the Emacs 30.2 daemon,
run
~/bin/emacs30.2/bin/emacsclient -s work -c .
from the command line.
I sometimes found it is more natural to have a desktop application for
GUI, so I have ~/.local/share/applications/emacsclient-30.2.desktop
file, and the content is
There could still be some risks of downtime due to conflicts between
Emacs/package versions, or caused by updating the OS/other
programs. These cases are rare, so this setup is good enough for me.
Well-known Emacs package creator Protesilaos Stavrou interviewed me for his video podcast series, “Prot Asks.” We talked about a wide variety of topics, from Emacs to podcasting to Portland to public transit. I thought it was a great time, and perhaps you will like it, too!
Had a chill day walking around the canal path in London, at 6:30 pm, I
was keen to continue on the home office project.
The existing wall is not plumb, so when I put the architrave, there’s
a gap. This is a typical issue, and a small gap (less than 3mm) can be
filled with deco chalk. In my case, the bottom has a 10mm gap, which I
have to address.
10mm gap at the bottom between the architrave and the wall
In general, there are two ways: either add a small piece to the door
lining to fill the gap, or rebate the architrave to accommodate the
wall protrusion. I jumped to the rebate approach as I didn’t have any
additional strips of wood for the first approach (happy skip days).
A quick measurement told me the architrave needs a rebate of 45mm
wide, and the depth varies: starting from 1300mm height, reaching to
10mm deep at the bottom.
The easiest way to do this in this scenario is to cut 10mm deep across
the board, as it is okay to have some voids behind the architrave, and
there are still 25mm for the architrave to be fixed on.
My first attempt was using a track saw: first cut was at 45mm line,
from the bottom all the way to the 1300mm mark. The next cut is right
next to the previous cut to increase the rebate area. Repeating this
process many times to get to the whole 45mm area. The groove in the
photos below is made from 3-4 passes.
With a blade kerf of size 1.8mm, I figured it requires 25 cuts to get
to 45mm. My efficiency-seeking brain took over and said: There must be
a better way.
So I pulled out the Dewalt router from the drawer, set the depth, and
clamped the architrave down to the table. The immediate problem I
faced was that it didn’t cut in a straight line: it went like 45-60
degrees for some reason, so I couldn’t cut a long groove like I had
done with a track saw.
Rebate using a track saw and a router
So I turned the router 90 degree and cut small and short chunks
instead. It worked well: the grooves I cut using a track saw serve as
a stopping line so I won’t cut extra. It was not perfect because there
were tons of dust coming out from the router, and it made so much
noise.
I put my headphones on and made a few more passes. I started seeing
how it can be done for the whole 1300m length. Then I saw my neighbour
over the fence, asking what I was doing. Well, it turned out to be
7:30 pm already, so I had to stop and leave it for tomorrow.
In hindsight, the track saw can do a much better job because I
realised only 5-10 passes would be enough. The small pieces between
grooves can be knocked off rather easily using a chisel. The track saw
has better dust collection, and the noise is much lower.
Another completely different approach is to remove the protrusion on
the wall using a multi-tool: placing the blade on the door lining so
the cuts will be flush with the door lining, and pre-cutting the 45mm
line to have a neat finish.
The standard Cat 6 plug is a pain to work with: I have to untwist the 4
pairs, make them perfectly straight, lay the 8 wires side by side with
no gaps, and then insert all of them into the RJ45 plug in one go.
It sounds easy, but since the wires are flexible, it is actually very
hard: very often the wires move around and become misaligned or
misplaced during the fitting. If that happened or any other part of it
went wrong, I would have to pull out the whole lot and restart again.
I had successes before, usually after a couple of attempts, often
accompanied by frustration in between. It requires me to activate the
fight mode, give it 100% focus while sitting in an “Orz”
position1, so there’s quite a lot of energy poured into it.
At Height
However, even if I want to, it becomes physically impossible when it
comes to fitting a plug in the air for the CCTV cameras: the ladder is
a bit wobbly with uneven ground underneath it, and it is windy and
raining due to a summer storm.
Since I wasn’t happy with the normal Cat 6 plug, I was keen to try new
products. So when I first saw the IDC Punch Down to RJ45 Plug from
Kenable2, I ordered a few. It turned out to be a smart little
move (this time).
This product has a built-in RJ45 plug that is already wired up, so I
can skip that difficult part. All I have to do is punch down the wires
into the IDC terminal. Punching down itself is very easy; I can do it
half-minded with one hand.
Another benefit is that I can split the fitting into multiple steps,
and I can take mini breaks for my arms between steps. Once one or two
wires are inserted into the IDC terminal, it binds the cable to the
plug. The binding is strong, so it hangs in the air and swings a bit
with the wind with no issues. Then I take my time for the rest of the
wires. If you don’t appreciate how important it is, trust me, your
arms become rather fatigued when working with your hands overhead.
I had to extend the main Ethernet cable that connects the main router
in the living room to the secondary router in the new
office. Technically, the cable size is spot on, but I had to cut back
2-3 times because a combination of my lack of experience and the LAP
data module from Screwfix is rubbish.
It seems like an unusual task given that there are only a few products
available on the market. I tested two, and I am happy with the
results, so I am documenting here for people who might find it useful.
Jelly Crimps
The first product I tested was from my electrician. It took me a while
to find out that its name is Jelly Crimps. You can get it from TLC or Amazon.
The little connector has two long sleeves that host two wires. It has
a button in the middle; press it very hard, and it will release the
gel. I highly recommend using a piler unless you have super strong
figures.
The process is simply: insert the wires, press with a piler to release
the gel, and repeat 8 times for each wire.
Jelly Crimps in Use
It costs about £0.2 to extend one cable, so it is very
cost-effective. I wasn’t sure it would work, but it does, and my
electrician vouches for it.
The only problem with this product is that it is not
maintenance-free. According to my electrician, I will have to put
these connectors into a back box and put a front cover over it, which
changes it to a much bigger job.
It got 2 terminate blocks built-in, one for the incoming cable, and
one for the outgoing. There is a diagram of the Type B protocol
printed on the product, so I don’t have to look it up on my phone. All
I have to do is punch down the 16 wires one by one. With a quality
punch down tool it is a lot easier and quicker than I thought.
Inline Coupler In Use
The product itself is solid, much better quality than the LAP data
module. I didn’t have to worry about damaging the terminal or face
plate when pushing it against the wall while using a punch-down tool.
The size is on a sweet spot, about 24mm depth, just enough to tuck it
into the 25mm service void. I am not sure if it is maintenance-free or
not, but I am comfortable leaving it in the service void as it has an
enclosing cover on it.
Kenable is the only place that sells it at a reasonable price, about
£2 each, while the rest of the sellers is asking for £5 so thank you
Kenable for making it affordable.
This is my contribution to month two of Greg Newman’s Emacs Carnival. The topic this month is “Writing Experience,” which is perfect, since I write in Emacs all the time. In fact, I am writing this blog post in Emacs right now!
Emacs Writing History
I do not know how long I have been writing in Emacs, but I have documents written in LaTeX from the early 2000s that were definitely written in Emacs, and I know those weren’t the first. I guess what is interesting about my writing experience in Emacs is how it has developed over the years. Where I originally wrote everything in plain text or LaTeX, over time I moved to org-mode and Markdown.
Writing LaTeX in Emacs taught me to view documents and writing the same way I view source code. Since all of my writing was in text formats, I could easily use version control software to store archives full of text. I began by using Concurrent Versions System (CVS), but eventually moved to Git. I have a few Git repositories with writing, but my biggest dates back to 2009. Those early commits are all LaTeX and plain text files.
commit c64500897e2509bc3316d252cda10a9119384933
Author: Erik L. Arneson <pXXX@XXX.XXX>
Date: Sat Dec 5 10:51:06 2009 -0800
Initial import of an old repository.
The oldest org-mode file in my writing repository dates to December 27th, 2010, and is titled “Beer and Pizza.” The commit message indicates that I pulled some writing over from another repository in this commit, so obviously I had been writing with org-mode for quite a while. I know, for instance, that “Beer and Pizza” was an article that I wrote for an early incarnation of Southern Oregon Magazine, and my notes are dated back to February of that year.
Org-mode Is A Life-changer
You have probably read other people’s experience with writing in org-mode in Emacs, so instead of sharing an exhausting list of all the life-changing, writing-improving things that it has brought me, here are a few bits that I really love about it.
It is fantastic at organizing writing projects. From being able to shift around headings and blocks of text, to combining to-do lists with writing projects, to being able to create macros and include different files—all of these things make org-mode perfect for working on medium and large projects. I write everything from blog posts to podcast scripts to books in org-mode, and it helps me stay organized.
I do not have to write in Word or Google Docs. The export functions in org-mode do a great job creating Word files, ODT files, HTML files, and all kinds of formats. If org-mode’s export abilities aren’t good enough, it can also interface with pandoc to cover all the other cases! This means I get to use all of my familiar tools and processes when writing, and I don’t have to worry about awful word processing software!
Everything I need to write can happen in org-mode! I have several websites that are built entirely out of org-mode. When I need to create slides for a lecture, I do it in org-mode. When I write adventures for Dungeons & Dragons, I do it in org-mode. When I need to whip up a spreadsheet to handle my household budget, I do it in org-mode. When I need to write a formal letter, I do it in org-mode. Frankly, it is so expressive and flexible, that I don’t need to use other tools for writing. It’s all Emacs and org-mode.
Emacs Can Be Distraction-Free
I have a hotkey connected to writeroom-mode, which plonks me right into a full-screen, distraction-free writing mode. Easy as pie! This means that when I really need to get down to business and get a lot of writing done, I don’t need to switch to a new app, leave my familiar tools behind, or go through any extra trouble. It’s just right here, built in.
No Going Back
I’ve been writing with Emacs for decades. It is my comfortable writing spot. I honestly cannot see myself abandoning it for another writing tool, because it works for me. I just used a couple of commands to check my three biggest writing repositories, and I have approximately 630,000 words written in org-mode between them. That’s a commitment!
I am aware that there are a lot of people out there using Emacs for writing, so I am excited to see what others share. I’m very pleased with Greg Newman’s choice for this second writing topic.
Edit: I realized that what I was calling my second biggest writing repository was actually my third biggest. I updated some numbers to reflect another 130,000 words of writing I found.
The best time to learn difficult thing is in travelling. During my
last two-week’s trip to Singapore/Malaysian, I was reading about
Ledger Cli causally. without putting much efforts, it clicked. It
suddenly started to make sense to me.
The more I learn, the more I want to learn more. I cannot wait for the
next opportunities to open Emacs and dive into Ledger’s brilliant
documentation. This is me with my Emacs in Changi Airport next to the
Jewe.
Emacsing next to the Rain Vortex in Changi Airport
I was able to apply the learning and came up with the project-rule to
keep data hygiene (will blog next). The positive feedback energise
me. The flight to London is ready for boarding but I don’t want stop
exploring during the 14 hours flight without WIFI.
That’s where I re-discovered the Info documentation system. I used it
to read the ledger.el library between sleep sessions 8000 feet above
the ground. Reading in plain text inside of Emacs has great benefits,
no distractions, fraction free in taking notes. it was a breeze.
Then I stepped into learning the Info documentation system itself, how
to navigate, search text/index and all that. I was able to pick it up
quickly, the concepts and shortcuts are native to me as an experienced
Emacs user.
Why Info is not Popular?
I envisioned myself to use it to read all the documentation,
e.g. Pandas library’s in Python. That would be ideal I told
myself. However, I soon realised that Info documentation system is a
niche tool: it is mostly used in GNU projects and Emacs libraries.
Why it is no popular? I was wondering myself. I decided to have a go
myself. well, the journey to start is already full of hiccups. This is
typical theme in learning legacy system, and could put many people
off.
So there are few notes that helped me to learn Info. Hopefully it can
bring more new users to the Info system.
dir the Index File
The first and most important thing I realised is, in the context of
Info, the dir is not a directory, but a plain text file. I simply
call it index file, then the rest becomes so much clearer.
The =dir=/index file is the entry point of the Info program. it has a
lists of the available Info manuals with their name, Info file
location, and desecration.
Setup Info in MacOS
Then there is a bug in emacs-plus: during the installation of Emacs,
the dir file somehow got deleted in the cleaning process. So the
manuals for the default libraries that comes with Emacs are not
available. In my case, I only have few Info from the packages I
installed post-installation, like orderlies, org-roam for example.
I took a slightly different approach to fix this problem: I kept the
system level tools separate from the Emacs’s library, so i have two
dir files.
# manuals of system level programscd /opt/homebrew/share/info
for file in*;do install-info "$file"dir;done# manuals of Emacs and Emacs librariescd /opt/homebrew/share/emacs/info
for file in*;do install-info "$file"dir;done
Then tell Emacs the locations of those dir files as below.
Note, the convention is for each directory in the list, there is a
dir file, on in Emacs, we are specifying the file using directory,
and the file is happened to be called dir. I feel the naming can be
improved to avoid such confusion!
After restarting Emacs, Info will show there are about 500+ manuals
available, e.g. find tool, mu4e library, and Ledger3.
Lastly, an quick note install-info. As shown above, it is used to
install Info manuals, taking ledger3.info as an example, to install
it requires
I have been testing using Ledger-Cli to track my expenses, so far I
have found the tagging system useful. In my ledger journal, each
transaction is associated with a project, for example, the below
transaction is assigned to project “2024 Monitor Stand”
This constraint I came up with helps avoid meaningless spending on new
shiny tools. Operationally, imposing this limitation on my book
provides flexible ways of querying the data.
For example, bring up the transactions that do not have projects
assigned to:
Table 2: Number of items purchased for each project
No. Items
Project
39
2024 Loft Lights
34
2024 Loft Insulation
32
2025 Garage Conversion
8
2024 Monitor Stand
2
General
The data shows the “2024 Loft Lights” project is by far the largest
. That was a simple project by itself, however, since that was my
first electrical project, I had to purchase a lot of stuff, 1.5mm
cables, clamps, grommets, connectors, switches, sockets etc.
Finally, I have the “refund” tag so I can flag up the items to remind
of myself to check if I received the refund fully.
In the past few months, I've had a number of contracting and side projects crop up that have required a lot of software architecture work, including developing Entity Relationship Diagrams, which I find helpful when planning out a project (and especially when working with other people not as familiar with the subject matter). While at work, I have a bevvy of SaaS options to do much of this (primarily with Lucidchart), I wanted to see what options there were in OSS.
I've mentioned before that org-mode is my daily multitool for organization, prototyping, and experimentation. As such, I figured there had to be some kind of literate programming solution for ERDs, given that I've used CLI tools like graph viz in the past.
In one of my projects, we're working on integrating Mermaid into our UI, and a quick web search revealed that there was more than just a JS library, there's also a command line tool!
...and reasonable supposition based on the ingenuity of emacs/org-mode users led me to search for an org-mode/Mermaid integration...
which of course there was: ob-mermaid from Alexei Nunez!
The setup in Doom Emacs was straightforward (just follow the Github instructions), and now I can easily generate SVGs of my ERDs turning this code:
#+begin_src mermaid :file blt_erd.svg :background-color #FFFFFF
erDiagram
BUILDING {
INTEGER id PK
TEXT project_name
TEXT project_country
TEXT project_postal_code
TEXT asset_type
TEXT building_construction_type
TEXT building_use_type
TEXT project_units
TEXT gfa_measurement_method
TEXT gross_floor_area
TEXT enclosed_parking_area
}
LCA_METADATA {
INTEGER id PK
INTEGER building_id FK
TEXT tool_lca
TEXT project_phase_at_time_of_assessment
INTEGER operational_energy_included
INTEGER biogenic_carbon_included
INTEGER substructure_included
INTEGER shell_superstructure_included
INTEGER shell_exterior_enclosure_included
INTEGER interior_construction_included
INTEGER interior_finishes_included
INTEGER services_mep_included
INTEGER sitework_included
INTEGER equipment_included
INTEGER furnishings_included
}
TALLY_RECORD {
INTEGER id PK
INTEGER lca_id FK
TEXT revit_design_option
TEXT revit_general_category
TEXT revit_category
TEXT revit_type
TEXT revit_famly_name
TEXT revit_material_name
TEXT revit_building_element
REAL thickness_of_material
INTEGER total_instance_count
TEXT cumulative_material_volume
REAL cumulative_material_area
REAL cumulative_instance_volume
REAL cumulative_instance_area
REAL cumulative_instance_length
REAL cumulative_instance_perimeter
TEXT tally_entry_division
TEXT tally_entry_category
TEXT tally_entry_name
TEXT material_group
TEXT material_name
TEXT life_cycle_stage
INTEGER service_life
REAL acidification_potential_total
REAL eutrophication_potential_total
REAL global_warming_potential_total
REAL ozone_depletion_potential_total
REAL smog_formation_potential_total
REAL primary_energy_demand_total
REAL nonrenewable_energy_demand_total
REAL renewable_energy_demand_total
REAL mass_total
TEXT timestamp
}
BUILDING ||--o{ LCA_METADATA : "has many"
LCA_METADATA ||--o{ TALLY_RECORD : "contains many"
#+end_src
into this diagram:
So far ob-mermaid seems to support all the features of Mermaid (at least what I need)!
For a large portion of git operations I use daily, I find Magit’s interface much more convenient than git’s own CLI or IntelliJ’s UI. Although, I don’t use Emacs for all my coding projects, I’d like to have a shortcut to quickly open Magit for any git repository, either from a terminal or IDEA.
When I’m in a terminal and need to open a file or a directory in Emacs, I call the following wrapper, which I have in my $PATH as emacsclient:
#!/usr/bin/env bash
# Bring Emacs frame to the foreground
osascript << EOF
tell application "System Events"
tell application process "Emacs"
set frontmost to true
end tell
end tell
EOF
/Applications/Emacs.app/Contents/MacOS/bin/emacsclient "$@"
The wrapper simply brings the current Emacs window to the foreground and calls emacsclient passing the arguments to it.
Now, to open Magit from a terminal, I have another bash script that opens Magit’s status view for the current git repository in the running Emacs.
Now coming to the IDE, I didn’t know that before but you can configure a third-party command-line application as an external tool.
You can pass the project’s root or any of the predefined macros to it as an argument.
I explain how to implement key layout ideas from the Space Cadet. I configure shift as parens, and change caps lock to a dual use key for backspace and control.
(...)
There's three layered rate-limiters in here that are applied to only certain URIs:
One does a per-IP limit excluding my Tailscale network and some ASNs I connect from. Each IP can make one costly request per minute, otherwise receive a 503.
One tries to map certain cloud providers in to a single rate-limit key and gives each of these providers 1 RPM on these endpoints. Each group of cloud IPs can make one request per minute, otherwise receive a 503.
One puts a limit to 1 RPS of all traffic on each "site feature" in Gitea.
So now if you try to browse my Gitea instance http://code.rix.si or make a git clone over HTTP that will work just fine, but a handful of expensive endpoints will be aggressively rate-limited. If you want to look at the git blame for every file in my personal checkout of nixpkgs, you can do that on your own time on your own machine now.
So far installing this on my "edge" server seems to work really well, cutting the load of the small SSL terminator instance in half. Let's see if this is Good Enough.
However, after upgrading to macOS 15.3, emacsclient stopped bringing the Emacs frame up front.
One of the solutions I could find suggested adding (select-frame-set-input-focus (selected-frame)) to the Emacs config.
However, it only works when you start a new instance of Emacs.
What worked for me is adding this to the script before calling emacsclient:
osascript << EOF
tell application "System Events"
tell application process "Emacs"
set frontmost to true
end tell
end tell
EOF
Emacs 30.1 has been released. I summarise the purpose of which-key, a
package now built in Emacs 30.1, which I contributed patches to include
in core Emacs.
(...)
I use Orgzly Revived on my phone to capture to-do items, tasks, writing ideas, and projects. Its files are then synced with an ownCloud server. Those same files are also constantly open in Emacs on my computer, which means there can be some gnarly issues with things getting out of sync. It took me a while to figure out a good way to manage this, and in this brief blog post, I would like to share my solution.
Setting up a remote storage system is outside the scope of this server. Orgzly Revived works well with Dropbox, ownCloud, and nextCloud, though support for the other two is nicer as it can support automatic syncing in more situations. The Orgzly Revived documentation has an excellent description of how that configuration works.
Note that to keep things simple, I keep my Orgzly Revived files and only those files in a directory on my remote server. On my computer, the location for these files is ~/org/orgzly/, which is how I shall refer to it later on.
Configuring Orgzly Revived Auto-sync
To configure the auto-sync capabilities in Orgzly Revived, go to the settings screen in the app and then navigate to Sync > Auto-sync. Ensure that the following options are toggled on:
Auto-sync
Note created
Note updated or deleted
App started or resumed
This ensures that Orgzly Revived is constantly checking your remote storage and both fetching and uploading changes.
Configuring Emacs and Org-mode
On my computer, I want any changes that Orgzly Revived has updated to be automatically loaded into Emacs. Since I always leave Emacs running, that means it needs to detect changes on the disk for those particular files. I use two different methods to do this. First, in my init file, I have the following piece of code:
This uses the rx macro to easily create a regular expression that matches any filename ending in ".org" in the directory containing Orgzly Revived files. You may need to play around with the regular expression to get the right match. By adding those files to revert-without-query, Emacs will not bug you with extra questions and confirmations when changes are detected. However, this only works when auto-revert-mode is enabled!
At first, I thought there may be a way to enable auto-revert-mode on a file-by-file basis, but quickly realized that this could cause too many problems with creating new files in the Orgzly Revived app. Eventually, I realized that the solution would be to make a .dir-locals.el file in ~/org/orgzly/. Open that file in Emacs, and stick the following in it.
((nil.((eval.(auto-revert-mode1)))))
That ensures that any file in that directory will have auto-revert-mode enabled.
Once you have done this, you can add task files or inbox files in ~/org/orgzly to your org-agenda-files variable, and they will show up in your agenda views as usual.
Hopefully these instructions work for you. Please let me know if you can see any improvements or problems with the way I have implemented this. Happy task tracking!
My personal desktop is not booting (the motherboard is probably dead)
so I have been setting my server so I can work while sorting things
out.
I got stuck in getting magit working in emacsclient: I thought I could
run ssh-add inside of Emacs that would allow magic to access my
git repos using ssh, but apparently, it is not the case.
After some digging, I learnt that the problem I have to solve is to
run one ssh-agent in the background and then make the Emacs/Magit or
any programs hook onto it. Then once I run ssh-add and type the
passphrase for the first time, either inside of Emacs or in a bash
terminal, everything would work.
Implementation
Drop the following unit file below to
~/.config/systemd/user/ssh-agent.service.
The environment variable SSH_AUTH_SOCK is specified. It can be
anywhere as long as this environment variable in other programs
points to the same location.
ssh-agent is invoked with the -a option to provide an address
specified in the above step.
The $t is a specifier1 in systemd, it is equivalent to
$XDG_RUNTIME_DIR variable in Debian. It points to the runtime
temporary directory which apparently is safer2 than the /tmp
directory. The runtime directory was cleaned up after stopping the
ssh-agent so it is non-persistent.
There are programs developed to solve this specific problem (see Debian wiki). While using such a program seems like a simpler
alternative (e.g. keychain), I prefer to use systemd as the unified
approach for managing background services. I have been using it for
emacsclient, and I’m adding ssh-agent to it.
What is your preference? How do you solve this problem?
The little Raspberry Pi 4 (RP4) served me well in the last two
years. I used it to host NextCloud/Syncthing for syncing files between
devices, scraping financial data from Yahoo Finance and TimeMachine
for MacOS backup.
The latest addition to the service stack is paperless-ngx. It allows
my Canon printer/scanner to send digital copies of documents directly
to the RP4 or Gmail.
The RP4 handles all the demands without showing any signs of struggle.
It costs as little as 6kW per hour while the Xbox One S draws 11kW
while sleeping. Thanks to the energy crisis in the UK, I started to
appreciate the energy efficiency of RP4. The ARM chips in it really
impressed me.
Lack of NAS Capacity
A 3TB portal hard drive (WD My Passport) was attached to the PR4 to
store media data. The USB 3.0 connector is surprisingly stable and
fast. With both ends connected by ethernet cables, the file transfer
speed can reach up to 100 MB/s. When my MacBook Pro uses Wi-Fi, the
speed drops to about 40-50 MB/s but it is still great because of the
convenience.
Later I started using it as a NAS to store the Final Cut Pro
library. The 4k home gym videos I shot using iPhone 12 Pro are
numerous 1! The hard drive keeps getting filled up.
I can get another portal hard drive, but then it will get filled up
again, say in less than a month? So it occurred to me that I need a
proper home server with full NAS capacity.
Looking for a Successor
I did a bit of research but I am not able to find a good product. I
suspect the reason is the NAS build is a niche area while the PC
industry is gaming-centric, focusing on getting faster, bigger, and
fancier hardware with unnecessary RGB lights, that is where the
profits are I presume.
I came across some innovative products on AliExpress from China, such
as the TopTon N5105 board. It is more powerful, consumes
slightly more electricity, and it has 6 SATA cables! It would be a
perfect successor for my PR4.
But I am not comfortable ordering electronic stuff from
AliExpress, returning it or sending it back for repair would be a nightmare.
PS: The company is growing fast, it continued to innovate, and the product
lines extended to Intel N100 with an additional NVME drive and a USB-C. Their website
and marketing materials look notched up quite a bit. I kind of regret not
taking the risk back then.
Unexpected
The other day, I was re-organising (again) my home office, so had to
move a bookshelf. I started moving it without taking everything off,
then a motherboard fell off. It was the z170a with an i5-6600k and a
heat sink attached to it. The motherboard was in my first desktop that
I purchased 10 years ago when I started participating in Kaggle competitions
in 2014.
After a quick inspection, I saw some pins were bent. I felt ashamed and
sorry for the motherboard that I had not taken care of it. So I made a
promise: if it survived the fall, I would use it for my NAS.
Well, it did so I found my NAS.
Setting up z170a
While putting it up, one SATA port was snapped and came up, but the
rest is still fine. Apart from that, everything else went smoothly. The
Debian 12 became much easier to install with the isohybrid technology
and the non-free firmware is now part of the installation image
itself.
The server setup scripts and configuration are saved in a
selfhosted-services git repository so restoring the services took
little efforts.
I had one little trick: I assigned the IP address of RP4 to the new
z170a server so that on the client side I didn’t have to change
anything. This was achieved rather easily: few clicks in the ASUS
router web UI and then a reboot.
While setting it up, I noticed the z170a system is much more
responsive, thanks to the 3.5 GHz i5-6600k CPU and a much faster SSD
over the SD card. I was able to run multiple processes at the same
time.
The longest part is copying files from the 3TB portal hard drive to the
z170a’s internal HDD, which took about 20 hours.
It has great extensibilities: there are 3 free SATA for HHD and two
PCIe slots.
Power Consumption
The only downside is that it consumes a lot more electricity. When testing
in barebone, it drew only 10W. After putting everything together with
additional HDDs, fans, and ethernet cable, the power metre jumped to
45W. I removed hard drives one by one to see where the bottleneck is.
No HDD, 27W
IronWolf alone, 32W, 5W increases.
IronWolf + Seagate, 37W, another 5W increase.
IronWolf + Seagate + Toshiba, 45W, 8W increase.
So I kept only IronWolf which is a 3TB NAS grade HDD.
I also tried tweaking the BIOS and Linux kernel to get to C-states but
I felt it was over-engineering so I am happily settled down with 27W.
Footnotes
1 I record weightlifting to correct and improve my
techniques.
I used ledger-cli1 before and it was a painful experience. The
problem was not rooted in the tool but in how I intended to use it: I
wanted to track all my expenses, from buying a cup of coffee to
booking a holiday package. When I started this journey, there was a
massive jump from knowing little to nothing about personal finance to
doing double-entry accounting in plain text.
Though I gave up, it introduced me to the idea of owning my bank
transaction data in text files on my personal computer. So over the
years, I manually curated about 8 years of historical transaction
data.
If you haven’t done so, I strongly recommend you go to your banks’
website and download the transaction data manually, going as far back as
you can. You will notice that the banks only give access to 3-5 years
of data2. It’s a shame that banks use outdated technologies but
it is better than having nothing.
Since I had the data, I did some analysis and charts in Python/R. But
I kept wondering what ledger-cli can offer. I occasionally saw blog
posts on ledger-cli in the Emacs communities, so there must be
something out there.
It also has become a personal challenge. I turned not to give up but
put it aside to tackle it again after I got older.
Baby Steps
Hopefully, I had become smarter as well. This time, to ensure I can
successfully adopt the tool, I am going to reduce the scope to limit
to only tracking DIY project expenses.
I love DIY and I wish I had more days for DIY projects. It is usually
labour-intensive and I feel hyped and extremely confident after a couple
of DIY. Pairing it with learning ledger-cli, a cognitive-intensive
activity, would make them a nice bundle3.
Though the usage is simple, the question it can answer is important. I
want to know, during or after the DIY project, how much it exactly
costs. I could use a much simpler tool, like spreadsheets or a
pen/notebook, but I want it to be a stepping stone to acquire
ledger-cli properly in the future.
Why? - Effort Estimation
I need an accurate answer to the actual costs so that I can use the
data to train myself in cost estimation. This is an very important
skill to have as a homeowner, it would put me in a much better
position in negotiation with the tradesman. A lot of the people
in the UK complained that they or their relatives got ripped off by
tradesman.4
In general, house repairs and improvements are getting much more
expensive every year, due to the shortage of labourers, inflation and
Brexit etc. To give an example using my last two quotes, adding an
electrical socket costs £240 and replacing a small section of water
pipes costs £500.
I have a good habit of using org-mode to track time, my goal to add
ledger-cil to my system to track the expenses. After that, I would
know if it is really worth doing the DIY or finding a proper
tradesman. The total cost itself is not the only metric that matters,
but n very essential one to have.
With Sparkly you can transform numbers into multiline sparks, which can help you quickly visualize data. You have control over the number of lines to use.
Exemplify-Eval creates Exemplify-ERT–like examples. With it you can easily insert arrows followed by the evaluation of the sexp at point or of all sexps in a region.
When working in Python, I've traditionally used virtualenvwrapper, but I recently tinkered with some Jupyter widget work, which necessitated installing miniconda as well.
So yeah, my environments are a superfund site and reminiscent of this XKCD comic.
Just to make things more entertaining, I've started using uv on my Python projects, thanks to the prosthelytizing of both Jeff Triplett and Michael Kennedy and Brian Okkan on the Python Bytes Podcast
This has been great for some of the data and application projects I've recently spun up, and I think I'll be moving to using exclusively uv from now on (and might even try out its packaging capabilities for CyTriangle).
The only source of frustration has been the lack of support/tooling for uv on Emacs (though I would be happy to be proven wrong!). It's a bit difficult to search for, since uv is also a library in Python, and all my search turned up so far is a nascent (as of 3 weeks ago) uv-menu.
Really what I wanted was a drop-in function for pyvenv-workon after looking at pyvenv.el, so I wrote one myself!
(defun uv-activate ()
"Activate Python environment managed by uv based on current project directory.
Looks for .venv directory in project root and activates the Python interpreter."
(interactive)
(let* ((project-root (project-root (project-current t)))
(venv-path (expand-file-name ".venv" project-root))
(python-path (expand-file-name
(if (eq system-type 'windows-nt)
"Scripts/python.exe"
"bin/python")
venv-path)))
(if (file-exists-p python-path)
(progn
;; Set Python interpreter path
(setq python-shell-interpreter python-path)
;; Update exec-path to include the venv's bin directory
(let ((venv-bin-dir (file-name-directory python-path)))
(setq exec-path (cons venv-bin-dir
(remove venv-bin-dir exec-path))))
;; Update PATH environment variable
(setenv "PATH" (concat (file-name-directory python-path)
path-separator
(getenv "PATH")))
;; Update VIRTUAL_ENV environment variable
(setenv "VIRTUAL_ENV" venv-path)
;; Remove PYTHONHOME if it exists
(setenv "PYTHONHOME" nil)
(message "Activated UV Python environment at %s" venv-path))
(error "No UV Python environment found in %s" project-root))))
Running this function via M-x uv-activate should mean that then M-x run-python will spin up an interactive python process with your uv environment for tinkering.
Two Updates: Org+Nix dev streams, and my new DNS resolver
I've started to stream on Thursdays where I'll explore salt dunes and arcologies
The last few weeks I have started to work in earnest on Rebuild of The Complete Computer , my effort to provide a distribution of my org-mode site publishing environment in a documented, configurable Concept Operating System . My "complete computing environment" will be delivered in three parts:
a set of online documents linked above that are explaining how I manage a small network of private services and a knowledge management environment using my custom web publishing platform, The Arcology Project .
a set of videos where I work through the documents, eventually edited down in to a set of video lectures where you are guided from complete fresh fedora VM to installing Nix and a bare-bones org-roam emacs, bootstrapping a NixOS systems management environment, and then use Org files to dynamically add new features to those NixOS systems.
a handful of repositories which i'll finally have to treat like "an open source project" instead of Personal Software:
The arcology codebase which you'll have a copy of on disk to configure and compile yourself
the core configuration documents that are currently indexed on the CCE page, a subset which will be required to run the editing environment, and a number of other bundles of them like "ryan's bad UX opinions", "ryan's bad org-mode opinions", "ryan's bad window manager", etc...
I hope that by reading and following along with the documents while utilizing the video resources, one can tangle source code out of the documents, write and download more and an indexing process will extract metadata from the files that can be later queried to say "give me all the home-manager files that go on the laptops", for example, and produce systems that use that.
Two weeks ago I produced a three hour video where I played Caves of Qud and then spent two hours going over some of the conceptual overviews and design decisions while setting up Nix in a Fedora VM, ending with the Arcology running in a terminal and being used to kind-of-sort-of clobber together a home-manager configuration from a half-dozen org-mode files on disk. It was a good time! This is cataloged on the project page, 0x02: devstream 1 .
This week I came back to it after taking a break last week to contribute an entry to the autumn lisp game jam, and it was a bit more of a chaotic stream with only two hours to get up to speed on the project; there are many implicit dependencies in the design and implementation of the system because it's slowly accreted on top of itself for a decade now. That was 0x02: devstream 2
This week I'll work on cleaning up things to smoothly bootstrap and next week we'll come back with a better way to go from "well home-manager is installed" to "home-manager is managing Emacs and Arcology, and Arcology is managing home-manager" and then from there we build a NixOS machine network...
I have probably a three or six month "curriculum" to work through here while we polish the Rebuild documents. I will be streaming this work and talking about how to build communal publishing networks and document group chats and why anyone should care.
With the news from the US this week, it feels imperative to teach people how to build private networks, if only because the corporatist monopolist AI algorithm gang are going to run rough-shod on what's left of the open web the second Lina Khan and Jonathan Kanter are fired if they haven't already begun today. We can host Fediverse nodes and contact lists and calendars for our friends for cheap and show each other how to use end-to-end chat and ad-blocking and encrypted DNS; we oughta.
I'll stream on twitch.com/rrix on Thursdays at 9am PT and upload VODs to a slow PeerTube server I signed up for. Come through if this sounds interesting to you.
I re-did my DNS infrastructure
Years ago I moved my DNS infrastructure to a pi-hole that was running on my Seattle-based edge host. It worked really nicely without thinking about it when I lived in Seattle, but I hesitated fixing it for the years since I moved a half a hundred milliseconds away. The latency finally got annoying enough lately so I finally got around to it this week.
On my devicies, I've been using Tailscale's "MagicDNS" because DNS is a thing that I think should just have magic rubbed on it, as it is i've already thought way more about DNS in my life than I'd like. If you enable MagicDNS and instruct it to use your pi-hole's address as the global nameserver, any device on your Tailnet will use the pihole for DNS. Neat.
Pi-hole isn't packaged in nixpkgs and I was loathe to configure Unbound etc and a UI myself so I put it off and fnord ed the latency for months. I finally got around to it this week by deploying Blocky on my LAN server which has the feature-set I need, and rather than shipping a UI it ships a minimal API and a Grafana dashboard:
With the querying back on my LAN and managed by my Nix systems instead of a web GUI on an unmanaged host, I can list my blocked domains and block lists in a human-legible format, I can have different DNS results to route all my server's traffic direct over the LAN to my homelab instead of round-tripping to the SSL terminator, I can have custom DNS entries for local IPs. All this is managed in that one document which you'll soon be able to download from my git instance; that's the Concept Operating System promise.
If you're a content pihole user but never use the web UI and need to move, consider taking this thing for a spin.
trev over on #systemcrafters was asking about blogs switching from
straight.el
back to package.el, the built-in package manager for Emacs. I have done this, so here is a blog about switching back.
why straight.el?
First, let’s have some background. Straight has a fairly complex bootstrapping process and requires rewriting much of one’s Emacs configuration, so why use it in the first place?
When I was using straight, the answers were basically:
You can easily install packages that haven’t been released on an elpa, including locally-developed ones
You can specify a particular branch of a package’s repository
You can be sure that <b>only</b> the packages in your init.el are loaded
Basically, straight.el is <b>great</b> for power users that want to specify their configurations exactly. That used to be me, back when I started with Emacs and had my Furious Development period.
Nowadays, though, I’m older and tireder. So I’ve switched back to package.el.
why package.el?
As it happens, package.el is <b>just fine</b> for daily use by casual-to-mid-core users, in my opinion. It installs packages. It updates them. It even lists them out if you want to shop! For most Emacs users, it’s good enough..
What’s more, in recent releases of Emacs package.el has become even more powerful. With package-vc, you can install packages from external repositories, mitigating a major benefit of straight.el.
With package-vc, the only unique benefit remaining to straight is its declarativeness. It’s true that if you install a package using package.el, it will stick around in your .emacs.d and be loaded even if you delete its configuration from your init.el. But is this really a problem? You can mitigate it easily:
Run package-remove on it
Delete its directory from your .emacs.d (or hell, delete the whole elpa directory!)
Just Don’t Worry About It ™
That last point is powerful. So much of the Emacs community online seems to be hyperfocused on shaving down Emacs startup time to nothing, or maintaining a “minimal” config with “no bloat.” I have a lot of opinions on the subject of “bloat” in software, but that’s its own topic that I might write about later.
The point I’m making is, those worries are largely unfounded. So you have 15k of unused elisp files on your hard drive. So your Emacs takes an extra 1/3 second to start up. So what?
what I do
Okay, enough pontificating. Let’s get down to code snippets (that’s what you came here for, right?). My config is on
my website,
but the cogent part is excerpted here from my early-init.el:
package-ensure wraps package-install and package-vc-install for a unified API, and with-package wraps <b>that</b> to group package configuration together in a form. Notice that I don’t use use-package even though it’s now built-in to Emacs: I find use-package to be too “magical” for me, but that’s probably also another blog post.
do your thing
Of course, you don’t have to do what I do. You can use use-package with :ensure t (it defaults to package.el), or you can just add lines like this to your init.el:
Emacs is endlessly malleable, but that doesn’t mean we have to perfectly shape it to our exacting desires. Sometimes, it’s Good Enough™ to use the kit that comes with the kaboodle, as it were.
When you are self-employed, you need to be very well organized. There is never anybody looking over your shoulder, reminding you of everything on your to-do list. You don’t have a project manager reminding you of every step in your big projects. I use Org-mode in Emacs to manage all of my tasks, to-do lists, and projects.
Many people have already written aboutwhy Org-modeis a good choicefor this, so I am not going to. I will just mention that many years ago, perhaps around 2007, I read Getting Things Done by David Allen and got inspired to implement something like it in Org-mode. The system has slowly evolved over the years, but it has also become unweildy.
For the past few years, I had been tagging my tasks with the states TODO, NEXT, and DONE, primarily. I had some extra states sitting around for things that got canceled or delegated, and at some point I added a WAITING tag for when I needed somebody else to finish something, first. But the problem was, the TODO items really piled up. They became uncomfortable to sort through.
Inspired by a blog post by Sacha Chua earlier this week, I cleaned up my to-do states. I added STARTED and SOMEDAY, then went through the big list of outstanding items and re-evaluated their proper states. SOMEDAY won in 80% of cases, which really cleaned up the list. Now I can begin my day with a custom agenda command that looks for just NEXT and STARTED tasks, so I know what is most important. And I can end my day looking at TODO tasks to see if any should be switched to NEXT.
Here is what my configuration looks like now. First, I configure org-todo-keywords to handle the various states that my to-do items need. I am hoping that someday I’ll pare this down, but for now, this works.
I then added a “daily driver” command to my agenda to let me see the most important tasks today. This would probably be a good place for add-to-list instead of setq, but it’s just an example!
Next, after Sacha suggested it in a post on Mastodon, I configured to-do items to automatically switch to the STARTED state when I clock-in to them.
(setqorg-clock-in-switch-to-state"STARTED")
Finally, I have been playing around with configuring org-stuck-projects to be more useful. I tag all of my projects with a @project tag, and then have my to-do entries underneath them as keywords. I do not think that this works as intended yet. I don’t think I will be able to figure out the proper settings here until I have another stuck project; let’s hope that never happens, and I never need this report.
(setqorg-stuck-projects'("+@project/-DONE-SOMEDAY";; Keywords to identify non-stuck projects("TODO""NEXT""STARTED");; Keywords to identify stuck projects.("WAITING")""))
The only lesson that I can really hope to share with you, dear reader, is that it is a good idea to examine your task management system regularly to fine-tune it to your needs. What I have noticed is that I can come up with great task management plans, but the implementation rarely survives contact with the real world.
If, upon reading this, you have questions or suggestions, especially for org-stuck-projects, I would love to hear about it in the comments, or on Mastodon. Thank you for reading!
From a modelling perspective, it is not a big problem to have highly
correlated features in the dataset. We have regularised Lasso/Ridge
regression that are designed to deal with this kind of dataset. The
ensemble trees are robust enough to be almost immune from this. Of
course all model requires proper hyperparameter tuning with proper
cross validation.
The problem raises in understanding the feature contributions: if
there are 5 features that are highly correlated, and their individual
contribute could be tiny, but their true contribution should be
aggregated by adding the contribution together and considered them as
a group, e.g. adding their coefficients in Ridge, and adding feature
importance in LightGBM.
If their aggregated feature importance turns out to be indeed little,
I can remove them from the model to have a simpler model. A mistake I
used to make is removing the correlated features based on their
individual feature importance, it leads to less performant models.
A better and cleaner approach is to the clean up correlated features
to begin with, then I won’t need to do the feature importance
aggregation, and it would speed up the model development cycle: there
are less features to look at, to train the model, to verify the data
qualities etc. When the model goes live in production, it translates
to less data to source and maintenance.
Implementation
So I need to enrich my tool set to identify highly correlated
features. I couldn’t find an existing library that does that, so I
implemented it myself.
The key steps are:
Based on the correlation matrix, create a correlation long
table. Each row stands for the correlation between feature $X_1$
and feature $X_2$. Assuming there are three features in the
dataset, the table looks like this.
Row
X1
X2
Corr
1
A
B
0.99
2
A
C
0.80
3
B
C
0.95
Remove rows if the correlation is less than the threshold $T$. It
significantly reduces the input to Step 3.
If the threshold is 0.9, then the Row 2 will be removed.
Treat the correlation table as a directed graph,
Let $E$ be the unexplored nodes, filled with all the features
$X_1$ in the start, $R$ is the result.
For each node in $E$,
Continue to travel the graph in depth-first fashion until
there is no connections left, and add the connected node to
the result $R$ at each visit.
Remove the connected nodes in $R$ from the remaining nodes to
explore in $E$.
The vanilla Python code corresponding to Step 3 is listed below. The
ds object is a pandas.DataFrame, multi-indexed by $X_1$ and $X_2$,
so ds.loc['A'].index gives all the connected features from $A$ whose
correlation with $A$ is large than the provided threshold.
deffind_neighbors(ds:pd.DataFrame,root:str,res:set):"""recursively find the nodes connected with root in graph ds.
"""res.add(root)ifrootinds.index:ns=ds.loc[root].index.tolist()forninns:find_neiboughr(ds,n,res)else:return[]deffind_correlated_groups(ds:pd.DataFrame):"""
The ds object is a pandas.DataFrame, multi-indexed by X1 and X2.
"""res=defaultdict(set)# contiune til all nodes are visited.
cols=ds.index.get_level_values(0).unique().tolist()whilelen(cols)!=0:# always start from the root as ds is directed graph.
col=cols[0]find_neighbors(ds,col,res[col])# remove connected nodes from the remaining.
forxinres[col]:ifxincols:cols.remove(x)returnres
The result is a collection of mutually exclusive groups. Each group
contains a set of highly correlated features, for example
Group A: {A, B, C}
Group D: {D, K, Z}
The next step is to decide which feature to keep and remove the rest
within each group. The deciding factors can be data availability
(e.g. choose the one feature with less missingness), costs in data
sourcing (e.g. free to download from the internet) or familiarity
(e.g. the feature is well understood by people) etc.
Parameterisation
There are two hyperparameters:
The correlation type: It can be Pearson for numerical data and Spearman
for ordinal/categorical data. For a large dataset, it would take
some time to calculate the correlation matrix.
The correlation threshold $T$: The higher the threshold, the less
number of features to remove, so it is less effective. However, if
the threshold is set too low, it leads to a high false positive
rate, e.g. two features can be correlated, but they can still
complement each other in the model.
I would test a range of values from 0.9 to 1, and review the
results. Below graph shows the number of features to remove
with varying thresholds.
When $T=0.9$, there are about 95 groups, and in total 153 features to remove.
When $T=1$, there are 29 groups, and in total 37 features to
remove.
Proper end-to-end test runs are required to identify the best
hyperparameters. As a quick rule of thumb, those 37 duplicated
features identified with $T=1$ can be dropped without further testing.
The group sizes with varying thresholds $T$ provide an interesting
insight of the data. The 75% percentile of the group sizes is plotted,
which suggests that apart from the 33 duplicated features, there are a
large number of paired features (i.e. group size is 2) whose
correlation is large, more than 92%.
The display code in Emacs includes many optimizations. In this
article I explore some of the comments and code in the Emacs source.
There are indeed a lot of detailed comments in this part of the code.
We start with a small part related to TTY.
(...)
(Date: 20 September 2024)
Summary
I select highlights of jwz's previous post about a physical terminal.
This provides motivation for studying the Emacs display engine in a future
article, with associated optimizations.
Ann Arbor terminal
Jamie Zawinski wrote a 2016 blog post on reconnecting his physical
terminal from 1982--83, by using a Raspberry Pi.
Here are some excerpts.
(...)
KMonad is a free program which can provide advanced keyboard
customization, with possibilities useful for Emacs. For example, keys
can play multiple roles, making space act like Control, or shift keys
producing parens, or double tapping enabling a whole new keyboard
layer. The advanced technique of 'home row modifiers' is
also possible.
Summary:
EMACS was developed at the MIT AI Lab in 1976. The specifics of the
origin have been documented by different people in various places.
There is an interesting thread which was discussed on the blog of the
late Dan Weinreb, and preserved by archive.org. Ultimately, Guy
Steele pulled up his records (in the form of printed emails). The
below is an extract which is of historical interest and includes
emails from the first couple months of Emacs in 1976. I quote some
sections and include the verbatim text at the bottom, which starts
with an ITS email from RMS to GLS.
tl;dr I am now running RiverWM on my NixOS distribution and have a published configuration for it therein.
I'm not particularly happy to write this. For literally half of my life, from 16 to 32 years, I ran the KDE Plasma desktop but recently I was forced to swap away. I really like using my computer without a mouse and KDE has made it difficult to impossible for me to do so.
For quite a number of years I was able to run KDE Plasma with XMonad as the window manager under X11 by just setting a session variable in my .profile to KDEWM=/usr/bin/awesome and that worked great for many years; I was even able to go full emacs for a long time with EXWM (which allows one to treat Emacs as the "root" environment with X11 windows acting as Emacs buffers, rather than a WM being the root environment and Emacs being a window). At some point the simple solution of setting an environment variable stopped working but all you had to do to get Plasma with another WM on X11 was set up a SystemD user-unit overlay and then basically ask the WM to manage the Plasma windows.
When I got my GPD Pocket 3 , however, I found that despite it being an Intel integrated graphics machine, X11 did not run well on it and I set up my first Wayland-native environment which meant risking and considering throwing away a decade and a half of good experience with stacking tiling window managers. I set up Bismuth which was a KDE Plasma 5 KWin plugin which would auto-tile and lay out windows and manage an ordered stack so that I could Super-j and Super-k up and down the stack of windows and Super RET to push the active window to the top of the stack and re-organize the windows so that that window was the largest. It was basically good enough, Plasma 6 is a really nice desktop and the stacking features of Bismuth meant that my three or four main applications could be driven from my keyboard's home-row.
Bismuth does not work on Plasma 6 and I started to have some really frustrating crashes in Plasma 5 KWin Wayland which was resolved by a fix only applied to Plasma 6 and XWayland would randomly exit with code 0 and no log output. I eventually got Plasmas X11 to work decently on the GPD Pocket 3 but at the cost that touch events and other libinput stuff didn't work well enough to be my daily driver that I've had to look to other shores.
Bismuth doesn't work with Plasma 6, and is no longer under active development so I had to try to go back to a non-stacking tiling WM (think sway/i3) and tried Polonium, which unfortunately did not work with how i want to use my computer. Not having an auto-tile system was driving me mad. The other Wayland tiling window managers were mostly interested in emulating this tiling philosophy that i3 uses rather than an auto-tiling system with layouts built in like XMonad or Awesome.
River WM is the tiling system that gives you that, basically, with caveats. so I spent the last week or two while my personal life un-winds and comes back together, building a desktop configuration with RiverWM and getting really angry with the state of modern linux desktop systems along the way.
Any intuition I have about how a Linux desktop system is composed is thrown out the window with the move toward D-Bus and SystemD managed desktop sessions. I have spent the last week dealing with fiddly fucking environment variable propagation and XDG Desktop Portal configuration files to get things like my Matrix.org local proxy service Pantalaimon to connect to D-Bus and auto-spawn a secret service which contained the OLM keys. After every NixOS rebuild, I would end up with two waybar instances running and spent all day today trying to get it to work within a systemd-run ephemeral user unit, but for some reason despite verifying the environment variables being basically-equivalent by poking at /proc/$PID/eniviron, the user-unit version of waybar would not show icons in the taskbar, despite it working if I spawned waybar in the initialized desktop session. Infuriating shit.
I have a system I am mostly content with but I know that the "long tail" of issues in maintaining and spit-shining this thing will probably end up being about the same amount of work as it would take to just grit my teeth and get used to PaperWM or Polonium running within a desktop that provides all this ugly plumbing.
But it lends a problem, I can no longer recommend the software I use to anyone else, especially "normies." I've spent the last year carefully polishing a Linux distribution and Emacs environment that I could hand to family to plug in to a cloud that is not owned by a tech monopoly and believe they could use built on a desktop I've tried to understand the control surfaces and plumbing of since high school. I thought I was close with the Rebuild of The Complete Computer , but this feels like a set back.
Now I have a fucked up tiling window manager system that no one else can use and a bunch of shell-script- and JSON- configured microservices that aim to provide a desktop that sucks less but mostly end up making me hate the Linux desktop and the isolated islands it's become. And meanwhile the display power management system on my laptop still barely works.
The Hey Smell This factor of going with the "choose your own Wayland Desktop" is nearly unbound, I can't recommend this to anyone in good faith unless they know what they're getting in to.
I hope some day that there is a window manager like River that implements the same Wayland protocols as KWin and I can swap it back in over the Plasma desktop, because I am oh so tired of dealing with desktop Linux plumbing. I probably should put my money where my mouth is one of these days.
The River CCE module is laid out nicely, though I still need to un-tangle some of the more frustrating bits of configuration like the XDG Desktop Portal configuration. If you want to try out River, there is a "batteries included" lightly opinionated home-manager.nix and nixos configuration for you to walk through.
AUCTeX includes the @code{preview-mode} facility, which generates
embedded images of LaTeX fragments, typically mathematical
expressions.
This needs to be called manually, with @code{preview-at-point}.
Paul Nelson has written the package preview-auto, providing the
preview-auto-mode.
This runs preview as needed after certain changes.
This is a very useful feature which works well alongside AUCTeX, on
which it depends.
This package is available in GNU ELPA.
published: A High Level Overview of a Concept Operating System
The first chapter of the Rebuild of The Complete Computer project has begun with the publishing of A High Level Overview of a Concept Operating System. I'm beginning to sketch out a workbook or textbook of sorts that walks through how to construct a self-documented Linux system out of org-mode documents, and how those documents can be used as kindling for a small community's dream of self-hosting.
In short, the Concept Operating System is a set of documents that manage a Linux operating system and an Emacs text editor environment, a map and a terrain for you to implement your own. By following along with this document and future documents you'll be able to build and maintain your own computing system. These documents describe a system of documents which will build upon and synthesize the author's existing Concept Operating System, which he refers to as "The Complete Computing Environment". In the next section we'll take a look at each component of the Complete Computer and see what it provides and how we can evaluate each layer and ultimately build our own.
In short it's a bunch of B.S. but it might be your kind of B.S.
After upgrading to NixOS 24.05, the Wallabag module I implemented based on dwarfmaster/home-nix's wallabag module stopped working and I didn't understand how it worked well enough to fix it. I had planned to swap back to running Wallabag in Docker once I got around to it, but stumbled across someone's link to another wallabag module for NixOS which had the benefit of running Wallabag straight out of the Nix Store and resolving some issues around cached resources which may have been responsible for the un-reproducable issues I was having with my old module.
I made some changes to the module along the way, adding configuration options to it, stripping out some DRY library helpers which I didn't need, etc. It's quite easy to use:
Regardless of how powerful and convenient the R/Python data ecosystem
becomes, there is still value in looking at the data in Excel,
especially when exploring the data together with less technical
people.
Thanks to its trivial interface Excel is widely used in data
analysis: hoover the mouse to select columns, apply filters then
calculate some statistics. Most of the time that is all it takes to get
the answers the clients are seeking.
I recently realised that having transparency and working with the
tools that clients use plays a crucial role in strengthening the trust
and delivering the impacts to the business. Sometimes I think I should
do more in Excel.
But
The problem with Excel is reproducibility - I’m not able to codify the
clickings done in Excel and integrate them into the automated data
pipeline. It is rather foolish to have quality control procedures,
including code reviews, automated testing, CI etc in the system but in
the very end drop all those gatekeepers and go for error-prone manuals.
Plus it is way more efficient to have everything done in one place to
have a smooth process with no fractions. It is a key factor in
enabling quick turnaround.
So I had the motive to limit the usage of Excel to deliver data to the
business and pair data analysis. Again I have been looking into how
much it can be done without leaving Emacs.
Emacs Has More To Offer
I was pleased to discover the ess-view-data package and its Python
counterpart python-view-data. They interact with an active R/Python
session in Emacs and print out data.frame objects in plain text,
a.k.a. view data. What’s more, it can process the data before viewing,
for example, subset the data row/column-wise, summarise the dataset
etc.
The package keeps a record of the data processing pipeline so in the
end I would have a copy of the R/Python code that generates the
output. I can then effortlessly transfer the code to a script to
ensure reproducibility in the future.
Another benefit derives from having a plain text buffer for the data.
It is handy in exploring large datasets with an excessive number of
columns. For example, the dataset I work on daily basis has about 300
columns. It contains different flavours of the financials, the raw
values, imputed, ranked, smoothed etc.
It’s not possible to remember all the column names even after more
time was spent in giving meaningful names or ensuring the correct
columns are referred to. Having a persistent plain text buffer that I
can search for makes finding the right column names a lot easier. It
also helps to check what’s in and not in the data.
That’s my first impression of Shuguang Sun’s packages, it looks
promising.
[2024-03-11 Mon] Eugene, OR: The Rebuild of the Complete Computer Project Committee announce the launch of the Rebuild of the Complete Computer Project, a reprisal and reintegration of the Emacs+NixOS Concept Operating System accreted over the years by The Rebuild of the Complete Computer Committee's Chair Ryan Rix . This multi-sensory multi-media experience promises to provide you and your community with new meta-cognitive powers or your money back [ed: the Committee refers us to Very Serious Legal Requirement #3 and reminds us that you actually are responsible for both pieces if it breaks].
The Rebuild of the Complete Computer is an attempt to curate a system where others could build a self-publishing platform built on top of The Arcology Project and use that same knowledge base to deploy and manage their own computer systems. It's an attempt to curate a system where myself or other interested nerds could deploy and manage a small fleet of systems for their friends and family and provide access to powerful, private, meta-cognitive tools.
I've recently re-launched the Arcology Project as a piece of Python software built using Django, you might be reading this using that software right this second. Or perhaps you're reading it in your RSS reader, which fetched the content from this software. Or you're reading it in on the Fediverse , where the CCE Wobserver 's mastadan talked to your mastadan and told it about a new post, an HTML page crammed in to an ActivityPub message.
Today I continued to work toward a replicable and reputable Arcology release, I spent some time today winnowing what software is included in my "stack" so that I could pull out the base dependencies and layers of the Arcology Project and the CCE, and in the process I've structured an interesting document constructing A Holistic View of the Arcology and Complete Computing where I describe the topology of a Concept Operating System in the context of the actual Arcology ideology, the theoretical ecologically just, sociologically friendly, self-sustaining human habitat idealized by Paolo Soleri and the architectural researchers at Arcosanti. You might find that interesting. Idk.
When I've done a bit more of this file organizing work, and a bit more drafting and scripting in some un-published documents and built some OBS overlays, I will begin doing a set of live-stream semi-scripted videos documenting the design and construction of a fresh Emacs+NixOS Concept Operating System and the publishing platform it's integrated with.
I'll likely be doing that on my new MakerTube channel, "Complete Computing", feel free to subscribe there or here if this is interesting to you.
I wasn’t aware of Jekyll’s draft mode. My workaround was manually
changing the published field in the front matter to true when the post
is ready to publish. It works fine. However, with naive support from
Jekyll, there are more benefits to using the draft mode.
To start with, I like the drafts saved in the _drafts folder, not
mixed with other published posts in the _posts folder. It is way more
cleaner and easy to manage. With a glimpse of my eyes, I can see what
are the posts that I am drafting.
It also gives a piece of mind: only posts under the _posts folder are
exported and shown in my blog. It ensures I don’t accidentally publish
a post in draft.
Once there are files in _drafts folder, adding --drafts argument to
the jekyll serve command is all I need to be able to see the drafts
locally.
Of course, I also need to write a bit of Lisp code to integrate the
draft mode into my blogging workflow. This is the remaining of this
post is about.
Implementation
For a blog post, I have the source file in org-mode and its exported
file in Markdown. Now there is a new location dimension: they can be
either in the _drafts or _posts folder.
mode
source file (org mode)
exported md in Jekyll
draft
org/_drafts/on_image.org
jekyll/_drafts/on_image.md
publish
org/_posts/2027_02_08_on_image.org
jekyll/_posts/2027_02_28_on_image.md
In terms of content, the published post and its final draft, and their
exported counterparts are the same, only in different locations. Their
content can be different to have some flexibility, e.g. published post
has higher resolution of screenshots. This feature is possible to
implement in the future. For now, I follow the simple “same but in
different places” rule.
The new process looks like this: When I publish a post, it moves the
org file from _draft to _posts folder, adds a date to the filename
(which I have already), and then triggers the exporting process. To
avoid duplication, it removes the original org file and its exported
draft in Markdown.
To achieve that, the main missing piece from my current Emacs
configuration is the yt/jekyll-find-export function (see
below). For a post in _drafts or _posts, it finds the full path of the
corresponding exported markdown file. I can then delete it or start
the exporting process.
(defunyt/jekyll-is-draft-p()"if the file is inside of the draft directory, it is a draft."(let((draft-dir(file-truenamejekyll-source-drafts-dir))(filepath(file-truename(buffer-file-name))))(string-prefix-pdraft-dirfilepath)))(defunyt/jekyll-find-export()"find the full path to the exported file of the current post."(let*((src-file(file-name-nondirectory(buffer-file-name)))(dest-file(file-name-with-extensionsrc-file".md")))(if(yt/jekyll-is-draft-p)(file-name-concatjekyll-site-draft-dirdest-file)(file-name-concatjekyll-site-post-dirdest-file))))
I do my best to keep my blog simple, I would not use images/videos
unless I can’t demonstrate well enough in plain text, for example, to
demonstrate a mobile app using a screenshot (Learn in Emacs - Building
Up Vocabulary) or how to represent stock price charts for neutral
network (Speed Up Sparse Boolean Data).
Even when I do, I keep the usage to the bare minimum: all I do is
insert the image, make it centralised, and put a caption on top of it.
The Org-mode supports images well, with a few additional HTML
attributes for each inserted image, I can fine-control the images’
position, alignment, size etc.
However, I can’t get the benefits because I migrated my blog posts
from HTML to the Markdown format for its simplicity. Plus Jekyll comes
with its little quirks when it comes to Markdown images. So I have to
write something for myself.
I managed to achieve a satisfactory workflow for my simple usage of
images in Emacs. It works well for the Jekyll site. Here’s the code
and explanation.
org-download : is the package that I use to create the images
for blogging from various sources.
I can drag images from external applications to Emacs, including
browsers, Preview, or iPhoto. The images will be saved in the
/project/assets/org-download folder per my matrix/project setup.
For the application that I can’t drag the images, I take a
screenshot inside Emacs by calling the org-download-screenshot
function.
yt/jekyll-copy-from-org-downkload: is a little helper function
that transfers the files under the org-download folder to the
/assets folder in a Jekyll site.
It lists the files in the source org-download folder and provides
them as a selection list. It comes with auto-completion and fuzzy
matches to help me choose the file.
It also strips out the special characters in the filename otherwise
the URL will be broken in Jekyll.
yt/jekyll-insert-image: lists the files in the /assets folder so
I can choose easily which image to use.
It brings up the Liquid template for image so I don’t have to
remember its syntax. It ensures the file path is in the correct
format (starts with /assets/), I just fill in the caption and size
after selecting the file.
An extract of the code is listed below for demonstration
propose. Future updates will be reflected in my .emacs.d git repo.
(defunyt/jekyll-insert-image(srccaption)(interactive(list(read-file-name"images to include: "jekyll-assets-dir)(read-string"Caption: ")))(insert(formatjekyll-insert-image-liquid-template(file-name-nondirectorysrc)caption)))(defunyt/jekyll-copy-org-download-to-assets(file)"copy file from project org-download folder to the blog assets folder.
it ensures there's no underscore(_) in the file name.
"(interactive(list(read-file-name"file to copy: "org-download-image-dir)))(let*((ext(file-name-extensionfile))(base(file-name-basefile))(dest-base(jekyll-make-slugbase))(dest-file(expand-file-name(file-name-with-extensiondest-baseext)jekyll-site-assets-dir)))(copy-filefiledest-file)dest-file))
Research shows having effective and rapid communication can boost
creativity and spark joy1. I believe in it from my personal
experience in conversing, reading a book in my native language or
trying to understand a large codebase.
I wasn’t enable to achieve similar results when it came to using
English. In the past I have been trying to improve my English language
skills to boost my productivity in reading books and to make it more
enjoyable. The approach was practising more in reading and writing.
However, I started questioning the effectiveness. This year I decided
to take one step back to focus on the basics and improve my
vocabulary.
I want to take the “slip-box” method2 which proved to be effective for
learning Emacs Lisp language. It is a bottom-up approach, so I would
have one note for each word with the explanation in it, links to other
similar words, or words I got confused with.
One advantage is that I can also leverage my existing setup.
Workflow for Building Vocabulary
When come across a new word that I’m not sure about its meaning, I
will
move the cursor to the word,
press F1 d to look into the dictionary, the result will shown in
the osx-dictionary buffer,
read its meaning and try to understand it,
press r to listen the pronunciation and read after it. I usually
repeat it a couple of times to deepen the memory,
press a to create an atomic note. it has the dictionary meaning
in it for future reference,
edit the notes to add my understanding and copy the
sentence/paragraph that contains the new word.
press C-c C-c to save it to my vocabulary database, which is just
a folder with flat org-mode files.
There’s quite a lot of automation so I can focus on understanding it
(Step 3) and write a good note (Step 6) in my own words.
This workflow depends on two Emacs packages:
osx-dictionary: it interfaces with macOS’s dictionary app. It
displays the meaning and says the pronunciation.
The package is well written and easy to work with; I managed to
extend it to add Steps 5-7 with little effort.
It has limitations: it works only in macOS and it only outputs one
dictionary. Adding the meaning in Chinese requires a few more manual
steps: 1) press o to open the Dictionary.app, 2) go to the Chinese
dictionary tab and copy the meaning, and 3) paste it to the note in
Emacs.
I personally find the Oxford dictionary macOS uses is not easy to
follow. From time to time I have to visit
https://dictionary.cambridge.org/ to find the explanation that I
could understand. In transforming my old vocabulary notes to the new
format, I found the explanation from vocabulary.com is the best. I
might have to resurrect my voca-builder3 package.
org-roam: it interfaces with org-mode for creating atomic
notes. It avoids duplication: if there’s a note for the word that
exists already, it opens the note, so I can have a look and enrich
it.
I can link notes/words in my vocabulary database which is very
useful because for me learning by comparing is super effective.
The org-mode provides a lot of functionalities that might be useful
to facility learning in the future.
Revision on Mobile Devices
Once I have a fleet of notes, the next step is to revise them on a
regularly. The routine I’m trying to get myself into is rereading the
notes I created for the last few days while waiting for the tube/bus,
I call it a revision break.
So far I have an Emacs lisp program4 that filters all my notes by time
so I have last_24_hours.org, last_3_days.org and
last_7_days.org. These files are synced with iCloud so they are
available to review on my iPad and iPhone using the beorg App.
Reading my vocabulary notes on iPhone
Org-mode Based Simple Study Strategies
For the dedicated study sessions, I need a few strategies to shortlist
the notes. I think They will be based on the metadata of the
note. With org-mode’s API, it should be easy to implement.
I haven’t done it yet, but the idea is to score the notes from 0 to 5,
5 means the most important notes so I would study them first, 0 means
not important notes so will be at the bottom.
There can be multiple scores, for example, one for pronunciation. the
word that I got the pronunciation completely wrong would get a 5, and
the word would get a 3 if sometimes I got it wrong, and sometimes I
got it right.
Another score is how many times I looked into the word. There are
words that I just keep forgetting about it, or keep confusing with
another similar word. So the property of ‘visited_at’ gets a timestamp
appended at the time of visiting and the score is calculated by the
number of timestamps.
Emacs Lisp Implementation
Adding an action to the headline in osx-dictionary’s buffer.
(require'osx-dictionary)(setqosx-dictionary-mode-header-line(append'((:propertize"a"facemode-line-buffer-id)": Add to vocabulary"" ")osx-dictionary-mode-header-line))
Adding yt/add-to-vocabulary to key a in osx-dictionary buffer. It
creates an note using org-roam.
(defvarvocabulary-repo-dir"~/matrix/learning/meta-leanring/vocabulary""where to save the vocabulary notes.")(defvaryt/voca--roam-template'(("d""default"plain"%?":target(file+head"%<%Y%m%d%H%M%S>-${slug}.org""#+title: ${title}
%?
#+begin_example
%(yt/osx-dict--get-meaning)
#+end_example
"):unnarrowedt))"roam template for vocabulary notes")(defunyt/add-to-vocabulary()"add a new vocabulary note for ther highlihted region or word at
point."(interactive)(let*((org-roam-directory(expand-file-name"notes"vocabulary-repo-dir))(org-roam-db-location(expand-file-name"org-roam.db"org-roam-directory))(org-roam-capture-templatesyt/voca--roam-template))(org-roam-node-findnil(yt/osx-dict--get-word-and-pronounce))))(defunyt/osx-dict--get-word-and-pronounce()"extract the word and its pronunciation from the *osx-dictionary* buffer"(with-current-buffer"*osx-dictionary*"(goto-char(point-min))(search-forward"|"nilnil2)(buffer-substring-no-properties(point-min)(point))))(defunyt/osx-dict--get-meaning()"wrap the *osx-dictionary* buffer cnotent as a string"(with-current-buffer"*osx-dictionary*"(buffer-substring-no-properties(point-min)(point-max))))(define-keyosx-dictionary-mode-map"a"'yt/add-to-vocabulary)
Footnotes
1 reference is lost; it is somewhere in the book “The Second
Mountain”, the chapter on religions.
2 from the book “How to Take Smart Notes”. I plan to reread this
book in early 2024.
Emacs is made to be modified. One of my skeptical colleagues said the fact that Emacs has state scares him, but I think it's one of its strengths (and why it has endured).
I started doing some Ruby and Rails review over the holidays and had just finished configuring Emacs for Ruby usage. While spinning up my Ruby project, Emacs became unresponsive and wouldn't open the .rb file I needed to get into. It would hang without any additional CPU usage (confirmed by htop), but none of the keybindings worked. I initially thought that something with the new packages for working with Ruby, but it became evident this was much more insidious. Upon rebooting Emacs, I couldn't open any files without the application hanging..
I use Doom Emacs because it's really easy to set up and configure, and for me it had significant performance improvements over Spacemacs (I build Emacs with native compilation, which Doom supports). The suggested troubleshooting for the framework starts with running $ doom doctor, which yielded a few expected warning and no errors.
However, I did get the following line printed after exiting the process (Ctrl-C)
/var/folders/hc/${weird_hash}/T//doom.3016.0.sh: line 1: #!/usr/bin/env: No such file or directory
I started troubleshooting on the Recurse Center Zulip (which has a very active Emacs user group), and someone else confirmed that they got the same line printed after exiting doom doctor with their working installation of Emacs. This seemed to me like it wasn't actually a problem so I cast my net wider...
I opened Emacs with the fresh install and tried to open the init.el file and it hung. again.
##Binary Troubleshooting
Doom emacs init.el file comes with a few modules enabled by default. To continue to troubleshoot this, I started toggling and untoggling modules until I finally discovered that the vc version control module was the culprit for the hanging process upon opening any file.
While I'd used the debugger within Emacs, it had always been in a situation where hitting Ctrl-G enough would exit the process (and I could just use toggle-debug-on-error or debug-on-entry). Other recursers suggested I take a look at the Doom Emacs discord, which led me to the following discourse about backtracing, including a section on backtracing from frozen Emacs!
I was able to set debug-on-entry for find-file if the vc module was installed, found the active Emacs process number, and ran $ kill -USR2 $EMACSPID. Lo, I finally generated a backtrace!
So I knew that with the vc module enabled and running find-file, Emacs would hang. I also discovered that running any magit command would also hang with the vc module disabled. I was able to open files, just not run magit. Some recursers suggested that I compare the backtraces for the two processes and discovered both had call-process("/opt/homebrew/bin/gpg" nil..., which suggested the issue was actually with gpg.
It turned out I had an expired gpg key, and command line troubleshooting for gpg just kept generating lockfiles. I removed the keys, and finally Emacs worked as expected!
While killing an Emacs process without killing Emacs is now in my org file, I'll probably forget all the other steps it took to get here, so I thought it was important to write down for posterity!
Once again, I have collected far too many links over far too long a period of time. Anyhow, here is a collection of blog posts and links from around the web that I found to be good reading over the past couple of months. Is it too late in January to say Happy New Year?
2024 has been a pretty weird year for me so far. I spent the first couple weeks of the year in isolation, and then Portland got hit with a Snowpocalypse (I love how that’s a regular thing now), followed by freezing rain. This is the third day in a row that the sidewalk outside my front door is basically an ice skating rink. That means it’s a great time to do some reading!
Security
Privacy is Priceless, but Signal is Expensive[Security]
I really appreciate this incredible breakdown of Signal’s costs and how they use donations. Signal
is cool. You should be using it. For real.
Breaking Laptop Fingerprint Sensors (Bruce Schneier) [Security]
Do people actually use their laptop fingerprint sensors? I hate mine. I just disable the dang
things and use a password.
Holding NSA’s Feet To The Fire (jcs) [Security]
Here is a second post about Senator Ron Wyden. He appears to be fighting the good fight! Go Senator Wyden!
Emacs
Transient for convenience with emms[Emacs]
Tory Anderson shares a really convenient EMMS control panel that uses the new version of transient.
Talk to Emacs with a GPT4 Co-Worker[Emacs] Gallagher Pryor shares a method he has for speaking to Emacs using ChatGPT-4. This is from back in November, so perhaps he has a package now!
Jeremy Friesen: Mentoring VS-Coders as an Emacsian (Jeremy Friesen) [Emacs]
I really enjoyed this talk about moving people away from VS Code and toward Emacs. The big takeaway
for this talk seemed to be that demonstrating what Emacs could do was going to be more effective
than just arguing. Let Emacs win just by letting it do what it does.
Jeremy Friesen: Test Driving a Campaign Status Document (Jeremy Friesen) [Emacs]
This post has some superb ideas for running a TTRPG campaign from Emacs. I’ve been doing this
for a while, but my documents get really messy. I need something a bit more structured.
Charles Choi: Improving Emacs isearch Usability with Transient (Charles Choi) [Emacs]
This is pretty cool. I recently learned about how rich the isearch feature set is, but I haven’t
played around with it a ton yet. Another thing to add to my Emacs to-do list!
William Denton: Basic citations in Org (Part 1) (William Denton) [Emacs]
This is an excellent introduction to how citations work in org-mode. There are four parts in the series so far, so keep reading!
Programming
Pike’s Rules Of Programming (jcs) [Programming]
These are some good rules, even if they can make some parts of programming a little less exciting.
Variations on styling variables in SSGs (Bryce Wray) [Programming]
I am still using Sass a lot more than the vanilla CSS stuff that should be replacing it. Also, I am
starting to see that this is a change I’ll need to take in my future WebDev adventures.
Firefox on the brink? (Bryce Wray) [Programming]
Bryce Wray is warning (or predicting?) that Firefox may be in a very dangerous spot in its loss of
user share. This is really disappointing, given how evil Chrome continues to be. And it’s only going to
get more evil. Convince your friends to run Firefox!
Zachary Kanfer: Numberdle! (Zachary Kanfer) [Programming]
This is a really fun browser game for people who enjoy numbers more than words. Move over, Wordle!!
History
Finally, here’s something fun to share with the kids.
Org-mode is the reason that emacs won out over vim (though I do dabble in evil-mode) for me when it comes to IDEs.
In 2019 during my first batch at the Recurse Center, I peeked over Shapr's shoulder while he was deep in coding, and was like, "WHAT'S THAT??" I used Sublime Text 2 as my IDE at the time, but the moment I saw org mode, I dropped everything to learn emacs and I haven't looked back since. Why was it such a gamechanger? For me, it make it unbelievably easy to context switch and keep good notes. I previously kept a running daily log in markdown, but that was fairly unstructured outside a list of daily bullet points. I rely on my org file to keep my brag sheet up to date, track time and effort on tasks.
If your manager at any point questions your sprint velocity, you can use org-mode to build a report of your unticketed "glue" work time. When I say I use it for everything, I mean everything. Every ticket with a task list, writing all those tickets with task lists, every MR/PR review with comments, every meeting with meeting notes (also a good way to keep track of your meeting to focus-time ratio), even literate programming via code snippets, and slideshow presentations. All of these things can be captured and done in org-mode.
Like many emacs users, my personal config is cobbled together from others (I can name at least Sasha Chua, Shapr, and Ryan Prior as major influences).
My startup configuration within each .org file isn't that fancy:
One feature I particularly like about tags is their nested behavior (a child heading will inherit its parent's tags). I have a number of custom tags, as I used them to track effort and time spent on my last job hunt and make some nifty data visualizations. I also have some custom SEQ_TODO for a similar reason.
I use Doom emacs now, with the following org packages in my packages.el file.
Most of these are helper functions to allow things like drag and drop into org files, but a particular favorite is the org-modern package for getting really nice styling.
I also use some other variable customizations in config.el for styling my org files:
(setq org-modern-star '("◉" "○" "◈" "◇" "*"))
(setq
;; Edit settings
org-auto-align-tags nil
org-tags-column 0
org-catch-invisible-edits 'show-and-error
org-special-ctrl-a/e t
org-insert-heading-respect-content t
;; Org styling, hide markup etc.
org-hide-emphasis-markers t
org-pretty-entities t
org-ellipsis "…"
;; Agenda styling
org-agenda-tags-column 0
org-agenda-block-separator ?─
org-agenda-time-grid
'((daily today require-timed)
(800 1000 1200 1400 1600 1800 2000)
" ┄┄┄┄┄ " "┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄")
org-agenda-current-time-string
"⭠ now ─────────────────────────────────────────────────")
;; Global
(global-org-modern-mode)
Here are examples of some of my most often used features:
Thanks to org-babel, there is support for literate programming within emacs in many programming languages. So far, I've tried out Python, Javascript, Ruby, Rust, and SQL. The SQL snippets are something I use quite often with work, as it's a good way to keep track of query responses on different days by keeping them in notes with a date tag assigned.
*** Python
#+begin_src python :results output
from collections import defaultdict
class Graph:
def __init__(self):
self.nodes = defaultdict(list)
def add_edge(self, u, v):
self.nodes[u].append(v)
def DFSrecursion(self, v, visited):
visited.add(v)
print(v, end=" ")
for neighbor in self.nodes[v]:
if neighbor not in visited:
self.DFSrecursion(neighbor, visited)
def DFS(self, v):
visited = set()
self.DFSrecursion(v, visited)
def print_graph(self):
print(self.nodes)
def make_graph():
g = Graph()
g.add_edge(0, 1)
g.add_edge(0, 2)
g.add_edge(1, 2)
g.add_edge(2, 0)
g.add_edge(2, 3)
g.add_edge(3, 3)
print("DFS starting at Node 1")
g.DFS(1)
return
make_graph()
#+end_src
#+RESULTS:
: DFS starting at Node 1
: 1 2 0 3
** Business Logic Applies to Use Cases
#+begin_src sql :engine postgresql :dbhost localhost :dbport 5434
select album.title as album,
sum(milliseconds) * interval '1 ms' as duration
from album
join artist using(artist_id)
left join track using (album_id)
where artist.name = 'Red Hot Chili Peppers'
group by album
order by album;
#+end_src
#+RESULTS:
| album | duration |
|-----------------------+--------------|
| Blood Sugar Sex Magik | 01:13:57.073 |
| By The Way | 01:08:49.951 |
| Californication | 00:56:25.461 |
I've only recently investigated using org-roam for furthering my personal knowledge management system, and I'm eager to incorporate it into my workflow!
I am having a hard time keeping my git repositories clean: there are
just too many repositories, I counted 31 in total, and I have 5
computers where I work on them.
The consequence is that sometimes I get surprised at seeing a lot of
seemingly useful changes that are not committed to the git repo. I had
to stop whatever I was doing to just think about what to do with those
changes. It breaks the flow!
There are other occasions where I thought I fixed some bugs, but I
don’t have the patches on my laptop. It turned out I didn’t check in
to the cloud, so I have to log back to the right server to run a couple
of git commands, or if I don’t have access to the servers, I have
to fix the bugs from scratch again. It is inefficient!
It can happen a lot in active projects where I work on multiple
systems and multiple git repos or when I travel. I plan to revisit my
filesystem (which is inspired by Stephen Wolfram 1) and tech
setup to reduce the number of repos by merging them and keeping only
1 laptop, 1 workstation and 1 server. This is something for summer, it
can reduce the severity of the problem but can not eliminate it.
At the moment, I just have to become more disciplined in managing
files, e.g. to have an atomic habit of checking my git repo regularly,
or at least do it once at the end of the day, or as part of the
shutdown ritual after finishing a task2.
Emacs Lisp Helper
The 3rd Law of Behavior Change is make it easy.
James Clear, Atomic Habit
To facilitate the forming of this habit, I implemented a utility
function in Lisp to list the dirty git repo, and provide a clickable
link to the magit-status buffer of the git repo. With one click on
the hyperlink, I can start to run git commands via the mighty magit
package. I bind this action to keystroke F9-G.
(defunyt/git--find-unclean-repo(root-dir)"";; (interactive)(setqoutnil)(dolist(dir(directory-files-recursivelyroot-dir"\\.git$"t))(message"checking repo %s"dir)(let*((git-dir(file-name-parent-directorydir))(default-directorygit-dir))(unless(string=""(shell-command-to-string"git status --porcelain"))(pushgit-dirout))))out)(defunyt/dirty-git-repos(&optionalroot-dir)"list the dirty git repos, provides a clickable link to their
magit-status buffer."(interactive(list(read-directory-name"Where's the root directory?")))(let((buffer(get-buffer-create"*test-git-clean*"))(git-repos(yt/git--find-unclean-reporoot-dir)))(with-current-bufferbuffer(unless(eqmajor-mode'org-mode)(org-mode))(goto-char(point-min))(insert(format"Number of dirty git repos: %s "(lengthgit-repos)))(dolist(git-repogit-repos)(insert(format"\n[[elisp:(magit-status \"%s\")][%s]]"git-repogit-repo))))))
The workhorse is the git status --porcelain command: If the git repo
is clean, it returns nothing, otherwise, it outputs the file names
whose changes are not checked in, e.g. the first file is modified (M),
and the second file is not untracked (??).
M config/Dev-R.el
?? snippets/org-mode/metric
The rest of the code is for parsing the outputs and turning them
into a user-friendly format in Org-mode. What’s interesting is that
The org-mode provides a kind of hyperlink that evaluates Lisp
expressions, using the example below,
[elisp:(magit-status"/foo")]["Git Status of Repo /foo"]
The description of the hyperlink is “Git Status of Repo /foo” , after
I click it, it runs the expression (magit-status "/foo") which shows
the git status of /foo repo in a dedicated buffer.
Before executing it will ask for a confirmation. It can be a bit
annoying and inconvenienced at first which naturally leads to the
temptation of removing this behaviour by setting
org-link-elisp-confirm-function to nil. I discourage you from doing
so in case someone embeds funny codes, (for example rm -rf ~/) in
a hyperlink, so make sure to check that variable’s documentation
before changing it3!
Practise
It was fun to write the lisp functions. I learnt how to use the
optional function argument and interactive so that the function can
be used both interactively and pragmatically. I’m very much wanting to
spend more time in coding, to enhance it with some ideas I got from
reading Xu Chunyang’s osx-dictionary package4.
However, the effectiveness of those functions has little to do with
the extra features I had in mind but really depends on how I use
them. Solving the problems requires deliberate practise and changing
my behaviours so that cleaning git repos becomes a habit of mine, which
is always the hardest part.
One key indicator for this habit5 can be the number of check-ins
and see if there’s a substantial increase from today.
I haven't blogged in quite a while. Not because I haven't done anything in the past 9 months, but because I have too much to do!
This past year has been challenging in ways I've never expected. I'm still trying to figure out some kind of balance as a interdisciplinary person whose job title at any given time unfortunately puts me in one box. My breadth of experience has resulted in some really great short-term opportunities in the past year, but I'm still trying to figure out where that fits as a career.
In terms of professional "hobby" development, 2023 was the year of maps for me. I started out the year wanting to add mapping to Bridge.Watch, but given its status as a hobby project, I never wanted to commit any amount of money to a tiling service. Thanks to a fellow Recurser, Brandon Liu, I was tipped onto Maplibre and Protomaps and attended a bunch of Recurse meetups for mapping (appropriately called Maptime) to explore the world of open source mapping. I started with some smaller data sets, first looking at the soft-story retrofit program. After several building failures in NYC, I mapped all of the active DOB complaints as well.
This self-imposed mapping learning path culminated in giving a talk at the FOSS4G North America conference at the end of October, which was a nice way to meet a different segment of open source users and contributors. I also got to visit NYC for the first time since I left in 2019, and I met up with nearly 30 friends and family over the course of 10 days.
Unfortunately the highs of those two weeks came crashing down in November. I'm still trying to recover both physically and mentally, so hoping to move towards better habits and a better mindset in the new year. To that end, I'm hoping to blog more about my "extracurriculars" and live a lot more by shapr's words, "Do something; brag about it."
Expertise I want to continue to develop and new stuff I want to learn:
Continuing from my last post, the EPA provides a seamless interface when
working with GPG files in Emacs. But there are situations where I have
to work with GPG files using other programs (mostly Python) which EPA
cannot help.
For those cases, I have to decrypt the GPG files first before using
them (for example, calling pandas.read_csv).
Obviously, there’s no point in encrypting a file if there is a
decrypted version next to it. So I also need a function to delete all
the decrypted files.
Emacs Lisp Implementation
Of course, I run Python inside of Emacs, I wrote the Lisp functions to
decrypt GPG files and delete all the decrypted files.
(defunyt/gpg--decrypt-recursively(root-dir)"It decrypts all the files ends .gpg under the root-dir. The decrypted files have the same filename but without the .gpg extension.
It stops if the decryption fails.
"(interactive)(dolist(file(directory-files-recursivelyroot-dir"\\.gpg"));; the 2nd argument for epa-decrypt-file can only be the base filename without the directory.(let((default-directory(file-name-directoryfile)))(epa-decrypt-filefile(file-name-basefile)))))(defunyt/gpg--delete-decrypted-files(root-dir)"It deletes the decrypted files under the root-dir directory.
e.g. if there's a file foo.tar.gz.gpg, it attempts to remove the foo.tar.gz file.
"(interactive)(dolist(file(directory-files-recursivelyroot-dir"\\.gpg"))(delete-file(file-name-sans-extensionfile))))
A bit of explanation:
directory-files-recursively: searches for files with a
pattern. Here, it returns all the files ending with .gpg under the
given root-dir,
dolist: loops over the GPG files to process them one by one,
epa-decrypt-file: decrypts a GPG file into a new file.
delete-file: deletes a given filename.
It seems the epa-decrypt-file function does not like the new
filename with the directory in its path, so I have to set the default
directory (working directory) and use the base filename after removing
the directory as a workaround.
Bash Implementation
It would be useful to have those functionalities outside of the Emacs,
so I implemented their counterpart in Bash.
function decrypt_recursively(){# PS: this function is equivalent to `gpg --decrypt-files $1/**/*.gpg`for fn in$(find $1-iname"*.gpg")do
echo decrypt ${fn} to "${fn%.*}"
gpg -o"${fn%.*}"-d"${fn}"done}function remove_decrypted_files(){for fn in$(find $1-iname"*.gpg")do
echo removing "${fn%.*}"rm"${fn%.*}"done}
The interface is the same: given a root directory, it decrypts all the
GPG files or deletes the decrypted files.
A little bit of Bash:
$1: refers to the first function argument, $2 refers to the
second function argument and so on. This is the Bash way. When the
function is called, $1 will be replaced with the actual argument,
here it means the root directory.
$(find …): is a list of files returned by the find program. In
this context, it stands for all the files whose filename ends with
.gpg.
It can be achieved using ls program but it will be a lot
slower 1 and requires some configuration in MacOS 2.
${fn%.*}: removes the last file extension of the variable $fn$,
for example, foo.tar.gz.gpg becomes foo.tar.gz.
Another approach is using $(basename $fn .gpg) to remove the
.gpg extension explicitly.
for, do, done: loops through each file.
The Bash functions have the advantage of being easily incorporated
into the system, for example, call the remove_decrypted_files
function automatically prior to shutting down or after login.
the new year is always a nice refresh and a chance to work on new creative projects. I have been thinking about going out and taking more photos and pen-plotting more of them on my AxiDraw . I decided to try to get DrawingBotV3 to work again, a piece of software which can take images or animations and export SVGs suitable to be plotted. Last year I bought the "premium" version which adds CMYK and pen-color matching, and run a bunch of different path-finding algorithms to generate the pen paths which lets you do some really neat things, like this for example:
The last version that I used was distributed as a jar file which was nice since I could just add a wrapper program which would call java -jar with the right arguments but newer versions are distributed only as deb or rpm files on Linux. So I updated my DrawingBot V3 on NixOS package to extract the debian package. This is the first time I have done this and it came together pretty quickly. It took some time to get the buildInputs lined up, especially since the Java app does some JNI loading which autoPatchelfHook doesn't detect. But this was a pretty fun little experiment and you can look forward to seeing more pen plotter art on my Plotter Art page over on the Lion's Rear.
I have growing concerns about data security. It is not that I have
something to hide, it’s that I don’t like how my data is being
harvested in general by the big corporations for their own benefits,
which is mostly trying to sell me stuff that I don’t need or I
purchased already. Seeing the advertisements specifically targeting me
motivates me to do something.
Setting my personal cloud seems a bit too extreme, and I don’t have
the time for it anyway. So I did a little “off-the-grid” experiment
in which I exclusively used an offline Debian laptop for data
sensitivity work (password management, personal finance, diary
etc). It is absolutely secure for sure, but the problem is
accessibility: I can only work when I have access to the physical
hardware.
It becomes infeasible when I travel, and it gives me some headaches to
maintain one more system. Also, the laptop’s screen is only 720p, I
can literally see the pixels when I write; it feels criminal to not
use the MBP’s Retina display. Lastly, It cannot be off the grid
completely; at one point, I have to back it up to the cloud.
So I spent some time researching and learning. I just need a data
protection layer so that I don’t have to worry about leaking private
data accidentally by myself, or the cloud storage provider getting hacked.
The benefits include not only having peace of mind but also
encouraging myself to work on those types of projects with greater
convenience.
GNU Privacy Guard (GPG)
is the tool I settled with. It is a 24 years old software that enables
encrypting/decrypting files, emails or online communication in
general. It is part of the GNU project which weighs a lot to me.
There are two methods in GPG:
Symmetric method: The same password is used to both encrypt and decrypt
the file, thus the symmetric in its name.
Asymmetric method: It requires a public key to encrypt, and a
separate private key to decrypt.
There seems no clear winner in which method is better1. I choose
the asymmetric method simply for its ease of use. The symmetric method
requires typing the passwords twice whenever I save/encrypt the file
which seems too much.
The GPG command line interface is simple. Take the below snippet as an
example,
The first line encrypts the foo.org file using the public key identified as
“Bob”. It results in a file named foo.org.gpg.
The second line decrypts the foo.org.gpg file to foo2.org which will
be identical to foo.gpg.
EPA - Emacs Interface to GPG
Emacs provides a better interface to GPG: Its EPA package enables me
to encrypt/decrypt files in place. So I don’t have to keep jumping
between the decrypted file (foo.org) and the encrypted file
(foo.org.gpg) while working on it.
Below is the simple configuration that works well for me and its
explanation.
epa-file-enable: is called to add hooks to find-file so that
decrypting starts after opening a file in Emacs. It also ensures the
encrypting starts when saving a GPG file I believe.
To stop this behaviour, call (epa-file-disbale) function.
epa-file-encrypt-to: to choose the default key for
encryption.
This variable can be file specific, for example, to use the key
belonging to foo2@bar.com key, drop the following in the file
;; -*- epa-file-encrypt-to: ("foo2@bar.com") -*-
epg-pinentry-mode: should be set to loopback so that GPG reads
the password from Emacs’ minibuffer, otherwise, an external program
(pinentry if installed) is used.
Org-Agenda and Dired
That’s more benefits Emacs offers in working with GPG files. Once I
have the EPA configured, the org-agenda command works pretty well
with encrypted files with no extra effort.
In the simplified example below, I have two GPG files as
org-agenda-files. When the org-agenda is called, Emacs first try
to decrypt the foo.org.gpg file. It requires me to type the password
in a minibuffer.
The password will be cached by the GPG Agent and will be used to
decrypt the bar.org.gpg assuming the same key is used for both files. So I
only need to type the passphrase once.
After that, org-agenda works as if these GPG files are normal
unencrypted files; I can extract TODO lists, view the clock summary
report, search text and check schedules/deadlines etc.
The dired provides functions to encrypt (shortcut “:e”) and decrypt
(shortcut “:d”) multiple marked files in a dired buffer. Under the
hood, they call the epa-encrypt-file and epa-decrypt-file
functions.
Lisp to Close all GPG Files
It seems that once a buffer is decrypted upon opening or encrypted upon
saving in Emacs, it stays as decrypted forever. So I need a utility
function to close all the GPG buffers in Emacs to avoid leakage.
(defunyt/gpg--kill-gpg-buffers()"It attempts to close all the file visiting buffers whose filename ends with .gpg.
It will ask for confirmation if the buffer is modified but unsaved."(kill-matching-buffers"\\.gpg$"nilt))
Before I share my screens or start working in a coffee shop, I would
call this function to ensure I close all buffers with sensitive data.
My notes show I have had this issue since 9 months ago. I made another
attempt, but still could not find a solution!
Solution
I then switched to tidy up my Emacs configuration, and the variable
org-html-prefer-user-labels caught my eye.
its documentation says
By default, Org generates its own internal ID values during HTML
export.
When non-nil use user-defined names and ID over internal ones.
So “#org0238b9f” is generated by org-mode. They are randomly
generated; they change if I update the export file. It means every
time I update a blog post, it breaks the URLs. This was a problem I
wasn’t aware of.
Anyway, what’s important is that, in the end, it says
Independently of this variable, however, CUSTOM_ID are always
used as a reference.
That’s it, I just need to set CUSTOM_ID. That’s the solution to my
problem. It is hidden in the documentation of some variables…
Implementation
So I need a function to loop through each node, and set the CUSTOM_ID
property to its headline. The org-mode API provides three helpful
functions for working with org files:
org-entry-get: to get a textual property of a node. the headline
title is referenced as “ITEM”,
org-entry-put: to set a property of a node,
org-map-entries: to apply a function to each node.
I changed the final function a bit so it is used as an export hook
(org-export-before-processing-functions) as an experiment. With this
setup, it runs automatically whenever I export a blog post in org-mode
to Markdown. Also, it works on the exported file so it leaves the
original org file unchanged.
The code is listed below. It can also be found at my .emacs.d git repo
which includes many other useful Emacs configurations for Jekyll.
(defunyt/jekyll--create-or-update-custom_id-field()"so that the CUSTOM_ID property is the same as the headline and
the URL reflects the headline.
by default, the URL to a section will be a random number."(org-entry-putnil"CUSTOM_ID"(org-entry-getnil"ITEM")))(defunyt/jekyll--create-or-update-custom_id-field-buffer(backend)(when(eqbackend'jekyll-md)(org-map-entries'yt/jekyll--create-or-update-custom_id-field)))(add-hook'org-export-before-processing-functions'yt/jekyll--create-or-update-custom_id-field-buffer)
I’m the type of writer who writes first and comes up with the title
later. The title in the end is usually rather different to what I
started with. To change the title is straightforward - update the
title and date fields in the front matter.
However, doing so leads to discrepancies between the title and date
fields in front matter and the filename. In Jekyll, the filename
consists of the original date and title when the post is first
created.
This can be confusing sometimes in finding the file when I want to
update a post. I have to rely on grep/ack to find the right files. A
little bit of inefficiency is fine.
Recently, I realised that readers sometimes can be confused as well
because the URL apparently also depends on the filename.
For example, I have my previous post in a file named
2022-12-08-trx-3970x.md. It indicates that I started writing it on 08
Dec with the initial title “trx 3970x”. A couple of days later on 13
Dec, I published the post with the title “How Much Does Threadripper
3970x Help in Training LightGBM Models?”.
The URL is however yitang.uk/2022/12/13/trx-3970x. It has the correct
updated publish date, but the title is still the old one. This is just
how Jekyll works.
From that point, I decided to write a bit of Emacs Lisp code to help the
readers.
Emacs Lisp Time
The core functionality is updating the filename and front matter to
have the same publish date and title. It can breakdown into three
parts:
when called, it promotes a new title. The publish date is fixed to
whenever the function is called.
It renames the current blog post file with the new date and title.
It also updates the title and date fields in the front matter
accordingly.
It deletes the old file, closes the related buffer, and opens the
new file so I can continue to work on it.
My Emacs Lisp coding skill is rusty but I managed to get it working in
less than 2 hours. I won’t say it looks beautiful, but it does the
job!
I spent a bit of time debugging, it turns out the (org-show-all)
needs to be called first to flatten the org file, otherwise, editing
with some parts of the content hidden can lead to unexpected results.
I always found working with the filename/directory in vanilla Emacs
Lisp cumbersome, I wonder if is there any modern lisp library with a
better API, something like Python’s pathlib module?
Code
Here are the main functions in case someone needs something similar.
They are extracted from my Emacs configuration.
(defunyt/jekyll-update-post-name()"it update the post filename with a new title and today's date.
it also update the font matter."(interactive)(let*((title(read-string"new title: "))(ext(file-name-extension(buffer-file-name)));; as of now, the ext is always .org.;; the new filename is in the format of {date}-{new-title}.org(filename(concat(format-time-string"%Y-%m-%d-")(file-name-with-extension(jekyll-make-slugtitle)ext)));; normalise the filename. (filename(expand-file-namefilename));; keep the current point which we will go back to after editing.(old-point(point)))(rename-file(buffer-file-name)filename);; update the filename(kill-buffernil);; kill the current buffer, i.e. the old file.(find-filefilename);; open the new file.(set-window-point(selected-window)old-point);; set the cursor to where i was in the old file.;; udpate title field. ;; note jekyll-yaml-escape is called to ensure the title field is yaml friendly.(yt/jekyll-update-frontmatter--title(jekyll-yaml-escapetitle))))(defunyt/jekyll-update-frontmatter--title(title)"Update the title field in the front matter.
title case is used.
"(let*((old-point(point)));; ensure expand all the code/headers/drawers before editing.(org-show-all);; go to the first occurence of 'title:'.(goto-char(point-min))(search-forward"title: ");; update the title field with the new title.(beginning-of-line)(kill-line)(insert(format"title: %s"title));; ensure the title is in title case(xah-title-case-region-or-line(+(line-beginning-position)7)(line-end-position));; save and reset cursor back to where it started.(save-buffer)(goto-charold-point)))
Back in my Kaggle days, I always wondered how much my ranking could
improve with a better computer. I finally pulled the triggers (twice)
and got myself a 32-Cores Threadripper 3970x workstation.
Before I can tell if it helps my Kaggle competitions or not, I thought
it would be interesting to quantify how much benefits I can get from
upgrading the i5-13600k to 3970x in training LightGBM model.
The TLDR is:
The speedup is 3 times in training LightGBM using CPU.
To my surprise, it is 2 times faster using GTX 1080Ti GPU than
i5-13600k.
There are no obvious gains from GTX 1080Ti to RTX 3080.
Experiment Set up
I use the data in the Optiver - Trading At The Close
competition. There are about 500,000 rows and 100 features. I train a
3-fold (expanding window) LightGBM model. Repeating the same process
with varying numbers of cores used in the process to get a performance
graph like this:
Threadripper 3970x vs i5-13600k: Train LightGBM Models on CPU
i5-13600k - Efficient Cores Count
The i5-13600k has 6 performance cores and 8 efficient cores. In
practice, I never use more than 6 cores in training ML models. My
theory is mixing fast performance and slow efficient cores leads to a
worse performance than using the performance cores alone. By
specifying 6 cores, I assume the OS uses only performance cores.
The result shows that I was wrong - Using more than 6 cores can give
considerable performance gain. It reduces the runtime by 10 minutes
from 6 to 14 cores.
The only plausible explanation is that when training LightGBM with 6
cores, it is already mixed with efficient cores. Therefore I see an
increases in performance while adding more cores.
Regardless I will start to use 12 cores in practise.
3970x - Disappointing Results
I know the performance gain will not scale linearly with the number of
cores but I wasn’t expecting that adding more cores can slow down the
model training.
The graph shows the 3970x achieves its best performance at using 12
cores. After that, adding more cores increases the runtime.
This type of behaviour is usually observed in simple tasks where the
overhead of coordinating between cores outweighs the benefits of extra
cores bring in.
But training thousands of decision trees with half a million data
points is definitive not in this simple task category. So I don’t
understand why this is happening.
i5 vs. 3970x - Training in Parallel
For 6 cores, it took i5 51 minutes and 3970x 42 minutes, which is about
1.2 speedup which is not bad. The same speed boost is also observed at
using 10 and 12 cores.
I found this consistent speedup confusing because there’s a mix of
performance and efficient cores in i5, so in theory every performance
core I add in 3970x should increase the performance marginal when
compared to i5.
In general, because of the poor scalability with respect to the number
of cores, the best performance is achieved when training the model with
a small number of cores and running multiple training in parallel. This is
the trick I use to get the extra performance boost for CPU-bound
tasks.
Here’s the setup for each computer:
i5-13600: use 6 cores to train each model, and train 2 models in
parallel. They are 2 cores left for OS background activities.
3970x: also use 6 cores to train each model, but train 5 models in
parallel! It also leaves 2 cores for OS background activities.
After a little bit of maths, it takes 14 hours for 3970x to train 100
models, and 42.8 hours for i5, so the speedup is 3 times. This is just
based on my theory. It would be good to actually run the experiment
and see the actual numbers.
Table 1: Training 100 models in parallel setting.
CPU
Runtime of 1 model (S)
No. models in Parallel
No. Batches
Total Runtime (H)
13600k
3083
2
50
42.8
3970x
2523
5
20
14.0
So the most benefit I can get from 3970x is in running multiple
experiments in parallel!
CPU vs. GPU - Impressive Performance
I have a GTX 1080Ti in my i5 PC for running deep learning models and
CUDA code. I never use it for LightGBM because the GPU implementation
was slower than the CPU in 2019 when I tried it.
In summer Guolin Ke, the author LightGBM, promised a significant
improvement in GPU performance when he was looking for volunteers to
work on improving LightGBM’s GPU algorithm.
Since I have the experiments set up already, it took me little time to
repeat the same experiments using the GPU trainer. All I did was adding
device_type=’gpu’ in the configuration files.
Table 2: Runtime of training a single mdoel
# CPU Cores
i5-13600k
tr-3970x
GTX 1080ti
RTX 3080
6
3083
2523
1435
1256
10
2695
1940
1269
1147
The result shocks me: I can get 2 times speedup just by switching from
i5 to 1080Ti with one additional line in the config and it outperforms
the 3970x in training single model setting by a big margin!
Is the 3970x worth it?
I found myself asking this question after seeing the results. In the
context of this experiment, no, it makes no sense to spend £2,000 to
get 3 times speedup when I can simply switch to 1080Ti to get 2 times
speed up with no costs.
However, the reason I go for the Threadripper and the TRX40 platform
is the 128 PCIe 4.0 lanes. The workstation is capable of running 4
GPUs at the same time at full capability while as i5 can only run 1
GPU.
If I had 4 GTX 3080 installed, it would finish training 100 models in
just under 8 hours! That’s 5.25 speedup to i5 and 1.75 speedup to
3970x in parallel setting.
This calculation is not for just entertainment. It turns out that
utilising multiple GPU to train gradient boost tree can be a really
big thing!
For a long time My Custom Keyboard Layout has used xmodmap to put !? closer to my fingers under shifted ,..
Shift + comma produces an exclamation point
Shift + period produces a question mark
Shift + slash produces a backslash
Shift + 1 produces a less-than mark
Shift + backslash produces a greater-than mark
Doign this with xmodmap works well enough on login but every time I plug or unplug a USB keyboard or switch inputs on my display (thus re-routing its internal hub) I have to re-run a systemd user service to reinvoke xmodmap.
Previously, I'd tried to define this using xkb in My NixOS configuration but the way I'd tried to do it was causing everything that depended on Xorg to be compiled from source. Yikes! I gave up for a while, but re-approached this last week once I moved my primary machine to Plasma Wayland because xmodmap does not work in Wayland at all.
This was a bit of a pain because there's not a lot of examples of folks doing this on GitHub and it wasn't clear to me whether and how NixOS's services.xserver.xkb worked with KWin's Wayland compositor. Last time I tried to write an entire xkb_keymap file but I couldn't get the syntax to work right and frustratingly xkbcomp wouldn't output any errors even with -w 10 until I got everything lined up. And then once I got it working with a whole xkb_keymap file that would apply with xbkcomp the_file.xkb $DISPLAY using examples from the NixOS Wiki's Keyboard Layout Customization page, it didn't work in Wayland-native apps like my MOZ_ENABLE_WAYLAND=1Firefox . At least services.xserver.xkb will work in both X11 and Wayland, but the documentation I found both in NixOS and in Linux-land broadly is quite weak on this subject.
But hopefully My Custom Keyboard Layout can serve as a minimal example for customizing keyboard symbols.
This past weekend was EmacsConf 2023. It was held entirely online, which was great because I could attend in my pyjamas! Since it happened on Eastern Standard Time and I live on the Pacific coast, I found the pyjama-enablement to be quite conducive to an excellent conference experience. Here are some things I learned by waking up at 5:30 in the morning this weekend!
Before getting into the good stuff, I should add that I did not spend a lot of time taking notes during the conference. Instead, I usually jumped straight into my Emacs configuration and started tinkering and fooling around. This means that I’ve probably left out some great stuff.
Emacs Advocacy
There were a couple of talks about Emacs advocacy, but the one that I paid the most attention to was Mentoring VS-Coders as an Emacsian by Jeremy Friesen. I think the big takeaway was that he got more mileage out of showing the power of Emacs than he did by trying to argue with users of other editors. Emacs users tend to already understand the power and flexibility of the tool they are using; just show it off, let it do the work.
However, another topic that was brought up in a number of talks was that having all of your projects available in the same tool seems to enable a better, richer type of thinking. Being able to both write effectively and program efficiently in Emacs helps generate more complete thoughts. It also seems to spur creativity and help people conceptualize projects better. This may be a big part of advocacy, as very few other programming editors are also good at writing and note-taking, which leads me to …
Writing
I do a lot of writing in Emacs, and I’m not the only one. James Howell gave a talk about authoring and presenting university courses with Emacs. I have also written and presented lectures and classes using just Emacs. It has the power to create slides, lecture notes, and even handouts from the same source. There were a few more great talks about writing, including another one by Jeremy Friesen (he seems to do a lot of cool stuff).
Howard Abrams talked about playing table-top role playing games (TTRPGs) in Emacs. He focused on how the pandemic had encouraged him to look into solo TTRPGs, which led him to Ironsworn. Howard then created a full Ironsworn system in Emacs that allows him to play the game on its own. I am including this under the “Writing” section because solo TTRPG play is a lot like writing fiction, it’s just that you use rules and randomness to help you figure out where the story goes. One of the best things about this talk is that the pre-recorded video was produced by his son. It looks great!
Hyper Things (Hyperbole and Hyperdrive)
I watched a talk about Hyperbole again this year. This is an Emacs package that I would really love to figure out how to use, but it just doesn’t integrate cleanly into my configuration nor my workflow. I tried it last year and got kind of annoyed at it. I gave it another try during Rob Weiner’s talk, but this time I couldn’t even get Hyperbole to properly set itself up. Anyhow, I gave up on it again. Until next year, Hyperbole!
Now Hyperdrive, on the other hand, was something new and interesting. Joseph Turner and Protesilaos Stavrou gave a talk on hyperdrive.el: Peer-to-peer filesystem in Emacs. Set up was extremely simple; it just worked. I was quickly able to share hyperdrive files and links with others. It remains to be seen what my use-case for Hyperdrive will be, but I look forward to watching this project develop.
EMMS
Yoni Rabkin gave a talk on the Emacs Multi-Media System (EMMS). I like EMMS, but I don’t use it as regularly as I should. However, Rabkin’s talk encouraged me to give it another chance. I am currently listening to Information Society in EMMS while I write this blog post. I suspect that like most people, I don’t really dig into my old MP3 collection as much as I should. This is a good excuse! Rabkin was also happy to remind the audience that EMMS can handle streaming online music sources and playing pretty much every kind of media. It’s worth exploring EMMS again!
Overall Impression
I loved EmacsConf this year. I also loved it last year. The organizers and moderators did such a great job running this conference, and it seemed to have a lot of attendees. I followed along on IRC and on Mastodon, and there was a ton of great talk happening. In addition, the Q&A sessions that I saw were filled with enriching conversation. I’m really looking forward to next year. Maybe I’ll find something on which to present!
I was planning to make posts like this more regularly, but I entered into a period where I was thinking, huh, I am not collecting very many links. But I was wrong, I was collecting links. I collected too many. And now look at everything you have to read! I am sure these links will keep you occupied for a while.
How The Thompson Hack Worked (jcs) [Security]
Aaaah! This is an amazing look at Ken Thompson’s 1983 Turing Award lecture. How can we trust
software? Do we trust software? You’ll enjoy this one and it will tickle your inner computer nerd.
Emacs Line Wrapping (jcs) [Emacs]
For several years, I have been dealing with a line-wrapping annoyance in Emacs that I just couldn’t
figure out. It turns out it was filladapt, a package that apparently isn’t used often anymore, but
I hadn’t noticed. I disabled the package and everything just works the way I want it to. Arrrgh!
Kensett: A Free 19th Century Urban Cartography Brush Set for Fantasy Maps (K. M. Alexander)
K.M. Alexander shares a lot of cool brush sets for creating maps for fantasy games. I have been
trying to figure out how to use them, and this one really caught my eye. Check out all of their
brush sets!
Lovat’s Genesis: City of Darkness (K. M. Alexander)
I am once again running a D&D game, with a homebrewed campaign, so I think it will be fun to include
a few RPG-related posts every once in a while. Seeing how others get their inspiration is really
helpful when I need to find some of my own.
I'm an avid Emacs user, so it appears more than once in the list. The original list of categories doesn't include coding, for which I obviously also use Emacs when possible 😛. It is my default app for almost all text related tasks. But for many other categories, I really just use default apps.
So, here are the apps I use:
📨 Mail Client: Mail.app
📮 Mail Server: Fastmail
📝 Notes: Emacs (org-roam)
✅ To-Do: Emacs (org-mode)
📷 iPhone Photo Shooting: Lock-screen button
🟦 Photo Management: Photos.app
📆 Calendar: Calendar.app
📁 Cloud File Storage: Apple iCloud
📖 RSS: Emacs (elfeed)
🙍🏻♂️ Contacts: Contacts.app
🌐 Browser: Firefox on Desktop, Safari on Mobile
💬 Chat: Telegram
🔖 Bookmarks: Firefox, Safari
📑 Read It Later: Emacs (org-mode)
📜 Word Processing: Emacs (org-mode), Google Docs
📈 Spreadsheets: Google Sheets
📊 Presentations: Google Sheets
🛒 Shopping Lists: Apple Reminder
🍴 Meal Planning: N/A
💰 Budgeting and Personal Finance: N/A
📰 News: N/A
🎵 Music: Spotify
🎤 Podcasts: Apple Podcasts
🔐 Password Management: 1Password
And some extra categories not mentioned in the podcast:
(ed: It was pointed out to me today that my last article in this feed had an update time far in the future, my apologies for fat-fingering that. Today I learned that lua's os.time will convert 93 days in to three months and change!)
Today I set up one of the final services in the long migration to my NixOS based Homelab Build , The Wobserver .
It's the "graveyard for web articles i'll never read" known more commonly as Wallabag .
wallabag is a web application allowing you to save web pages for later reading. Click, save and read it when you want. It extracts content so that you won't be distracted by pop-ups and cie.
Wallabag is a PHP application which is packaged in nixpkgs , but it is not trivial to enable it in NixOS as with many other services which are packaged therein. I did find a year-old NixOS module in dwarfmaster/home-nix which was a good starting place, so I copied that in to my system and set to work making it work with the current version of Wallabag, 2.6.6. It's nicely written but needed some work to make it match the current configuration format, and other small changes.
I then customized it so that it is easy to configure and use like standard NixOS modules:
If you want to use this, it should be straightforward to integrate. I don't think it's high enough quality to try to contribute it directly to nixpkgs right now, but if someone is brave enough to shephard that I surely wouldn't mind. 😊
paperless-ngx is a cool little document management system |
When I agreed to be the treasurer and bookkeeper for the Blue Cliff Zen Center I bought a Brother DCP-L2550DW printer/scanner/copier, an affordable, functional and reliable inkjet printer that doesn't mess around with you like similarly priced printers from HP etc do. It's fine and basically works with minimal configuration, and I can scan over the network using Skanlite. All nice and easy.
But taking this a step forward and managing my personal paper detritus has been a long-term goal; I have been broadly aware that there are decent open source OCR toolkits like tesseract for a while and I wanted to build a pipeline for generating OCR'd PDFs from a directory of scanned documents, and I never bothered to figure out how to do this myself.
I stumbled recently on Paperless-ngx , though, and found that it was packaged in nixpkgs , with a NixOS module to easily setup and configure it. So I did that.
Importantly, the full-text search works pretty well on printed documents. On hand-written stuff it'll struggle, I wonder if I can tune it against my own handwriting, but for now this is pretty nice:
It also attempts to do some amount of auto-categorization, though with only a couple dozen documents brought in so far, it's a bit too stupid to trust, and I spent about two hours after the first batch scan job to clear out the INBOX tag and manually sort things out. It also had a habit of parsing dates as D/M/Y instead of Americanese M/D/Y dates which I need to figure out how to fix.
Setting up the printer to do "Scan to FTP" was a bit of a pain , for some reason US models have the functionality disabled; I blame CISA. There is some BS you can do to go in to a maintenance menu to change the locale, reconnect it to the wifi, and then configure a Scan to FTP profile in the web UI but this feature is silently disabled by default.
Anyways, I got through a year worth of personal docs in a few hours and have a bigger shred pile than I would like, but I can shred them without feeling too badly now. I'll have encrypted backups of these documents on Backblaze B2 forever now, alongside a sqlite DB that I can full-text search. I'll probably upload my "Important Docs" directory in to this thing sooner or later, but for now it'll be able to handle my mail and the Zendo's documents. It also has a Progressive Web App manifest so you can "install" the management app on your phone to search docs on the go.
As with all of my NixOS code, it's documented and exposed on the Paperless-ngx page.
This is the second of my “posts of interest” posts. This week, I have also included some interesting Mastodon posts, because the Emacs community on Mastodon is thriving like crazy. It is really a blast to see so much interest in Emacs and so much activity.
Text showdown: Gap Buffers vs Ropes[Programming, Rust]
Troy Hinckley has been working on building the core of Emacs in Rust. This sounds like a very
difficult project, and it is informative and interesting to follow along. The latest entry in his
saga involves various ways of storing and working with text buffers, along with many benchmarks.
This is an interesting read!
ELPA and Emacs Zine (September 2023)
The new ELPA and Emacs Zine has released its latest issue, with some pretty interesting stuff about
the current state of tree-sitter and how development is progressing.
Emacs Macros[Mastodon]
Emacs macros remain kind of a mystery to me, but they were presented in an interesting way recently:
these are ways to provide a high level of automation in Emacs without learning how to program Emacs
Lisp. Well, they are worth checking out, then!
Taking advantage of tree-sitter[Mastodon]
This is a really cool Emacs function that takes advantage of tree-sitter to copy the current
function. I think it might need something to detect if tree-sitter is active, and error out if not.
XMPP in Emacs[Mastodon]
Fabio Natali on Mastodon reports that his XMPP usage would be more consistent if Emacs supported it
better. In particular, he misses E2E encryption support. I’ve also had a lot of difficulty with
moving to XMPP.
Alex Schroeder: Posting to Oddµ from Emacs (Alex Schroeder)
Alex Schroeder provides a simple, straightforward way to use the url package in Emacs to post
stuff to another service. In this example, he uses Oddμ.
Security (4)
C-suite weighs in on generative AI and security (Chris McCurdy)
More on the adoption of generative AI and security risks. 96% of business leaders say adopting generative AI makes a security breach likely in their organization within the next three
years! That’s certainly something to think about.
10 years in review: Cost of a Data Breach (Jonathan Reed)
Data breaches are dang expensive! We all know that. This piece explores some of the most important
factors in preventing and mitigating data breaches. There have been some changes in recent years,
some of which are caused by the rise of importance in AI. That means you should probably read the article.
The fraud was in the code (Molly White)
In the SBF court case, they actually used a code review to show fraud.
Bounty to Recover NIST’s Elliptic Curve Seeds (Bruce Schneier)
Here is a delightful story about the history of NIST elliptic curve cryptography and how things came
to be. Also, a cryptographic puzzle about where they may be going!
This is my first attempt at using elfeed-curate to collect interesting blog posts and share them. I have also attempted to subscribe to the RSS feed for the #Emacs hashtag on Mastodon, but that doesn’t seem to be working correctly yet. I’ve seen other blogs do similar things, but is this useful for my blog? I don’t know! Let me know what you think.
What to know about new generative AI tools for criminals (Mike Elgan) [Security]
Generative AI is still a minor concern for security professionals, but the threat is rising! This is
an interesting look at the state of the art and current means for addressing the threat.
Alex Schroeder: 2022-03-20 Torchbearers and bodyguards (Alex Schroeder) [Emacs]
Alex Schroeder continues his exploration of running tabletop RPGs using Emacs. Check out the other
blog posts in the series—they are a lot of fun. I have been playing with Emacs for running TTRPGs,
as well. Someday I’ll explain my methods.
Elfeed-curate (jcs) [Emacs]
Elfeed-curate sounds like a really neat package. This is my first annotation using that package,
which I will soon attempt to export.
Portland (1)
Tour of Untimely Departures – SOLD OUT! (lfadmin)
The Tour of Untimely Departures is an annual event at Lone Fir Cemetery in Portland. But guess what?
It’s already sold out! It sold out a while back, but I didn’t notice because my RSS feed for the
Friends of Lone Fir was broken.
In 2020 I wanted to create annotated charts showing events influencing the
COVID-19 numbers. I looked around and didn't find great ways to do it, so of
course, as a software engineer, I started building a thing:
contextualize.ai.
Fast forward many weekends, and lots and lots of hours around my 9-5s, I get the
project to a level where I'm not absolutely embarrassed by around a month ago
and start building and sharing charts with it. Some were really popular on
Hacker News, Reddit, and other social media.
And then yesterday something new happened: I got my first customer! Someone who
is not my mom or a good friend is willing to pay from their hard-earned money to
use this thing I materialized into the world out of sheer will and persistence.
The feeling is a mix of gratitude and cautious relief. Going from zero to one is
an important milestone, but it's just the first step in the uncertain path
towards "making something people want".
My goal from the start was creating a business that can sustain me and my
family. That hasn't changed, and I'll continue doing the work: marketing,
talking to users, doing my best to build something that makes their lives
better.
I haven't officially launched yet, but I can now say this with confidence:
contextualize.ai is the easiest way to build
and share a beautiful and engaging annotated chart.
P.S.: I'll start sharing more on Twitter (X?) if you're interested in following
along: @mpereira.
Now that I have installed Emacs 29.1, I needed to get it set up for Go development for a project. I was interested in taking advantage of both the new Tree-Sitter integration, and the new Eglot language server client. However, they were mildly tricky to set up! Here is what I did.
Configuring Tree-Sitter for Go
If you follow the excellent How to Get Started with Tree-Sitter instructions from Mickey Peterson, you will have a great head-start on getting Tree-Sitter working for most of your favorite languages (and probably Java, too). However, those instructions didn’t cover everything I needed for Go. When I tried running M-x go-ts-mode, Emacs complained about a missing gomod module. Baffling!
I couldn’t find any information in the Emacs documentation about where to find this missing module. I looked around on the net and found Camden Cheek’s tree-sitter-go-mod, and added that to my list of recipes. My treesit-language-source-alist then looked like this:
Note that the package is named go-mod but go-ts-mode expects it to be named gomod. I wish this were documented somewhere! In any case, I was then able to use M-x treesit-install-language-grammar for both go and gomod. Finally, M-x go-ts-mode worked!
After going through this process, I found Robert Enzmann’s post about automatically using Tree-Sitter. He has created the treesit-auto package, now available on MELPA, that does most of this work for you. It is a much faster way of solving the gomod mystery, so give it a shot!
Configuring Eglot for Go
I’d been using lsp-mode for ages, but with Emacs 29.1 including Eglot, I decided to make the switch.
In my Go project, I ran M-x eglot and was immediately met with an error:
[eglot] Server reports (type=1): Error loading workspace folders (expected 1, got 0)
failed to load view for file:///path/to/my/project: err: go command required, not found: exec: "go": executable file not found in $PATH: stderr:
I’ve got Go installed in /usr/local/go, and /usr/local/go/bin is definitely in my exec-path variable in Emacs. It looked like Eglot wasn’t propagating exec-path down to its subprocesses. How annoying! I did a quick search through the list of Eglot-related variables and the Eglot documentation and no solution seemed immediately forthcoming.
So I took the cheap way out and made a symlink. In my shell, I ran:
sudo ln-sf /usr/local/go/bin/go /usr/local/bin/go
It is a dumb trick, and I am sure there is a better way to solve it. Do you know of one? Please comment and let me know!
What Else?
My exploration has revealed that there’s a lot of work left to do in the Emacs Tree-Sitter world. There are plenty of languages major modes that don’t yet have a ts-mode equivalent, and plenty of others that still need a lot of work.
It’s too soon for me to say if this setup is preferrable to my previous configuration. But I am really looking forward to playing around with Eglot’s features and exploring the capabilities of Tree-Sitter.
You have probably heard by now, but Emacs 29.1 has been released! Here are some reasons to upgrade and how to do so right away if you are running Ubuntu 22.04 LTS.
What’s New with Emacs?
One of the most exciting new features for me is TreeSitter support, which provides incremental parsing capabilities for programming languages and other formatted files. This means that programming support will be getting faster, more comprehensive, and even better in future Emacs packages.
Other new features include native WebP image support, pure GTK support, and the ability to use emacs -x in the first line of a script, which could lead to some fun applications of Emacs Lisp. Read all of the release notes here!
Installing on Ubuntu 22.04 LTS
I am too impatient to wait for somebody to release a packaged version, so I installed Emacs 29.1 from source. It is fairly straightforward, though you will need to do a couple of special things to get all of the features you want.
Installing the Requirements
Note: All of the commands in this post assume you are using bash.
Most of the extra libraries and packages you will need to build Emacs 29.1 are covered in the build dependencies for the stock Emacs package. However, to take advantage of some of the cool extra features now included in Emacs, you’ll need to install a few more things.
In order to get native compilation (a feature added in Emacs 28) working correctly, you will need to make sure your shell instructs the build system to use gcc-10.
Download the Emacs 29.1 source code from a nearby GNU mirror and then extract it! Then follow the instructions below. Note that you might want to take a closer look at the options to ./configure. If you want native compilation, but don’t want to use the “ahead of time” option because it’s slow, you can remove the =aot. You might also want to stick Emacs in a different location using the --prefix option. For example, I used --prefix=/opt/emacs29.
cd emacs-29.1
./autogen.sh
./configure --with-native-compilation=aot --with-imagemagick--with-json\--with-tree-sitter--with-xft
make -j$(nproc)
The build might take a while. With the “ahead of time” compilation, I think my build took ten or fifteen minutes. But once it is complete, try running your new Emacs binary to make sure it works.
./src/emacs -Q
If it works, install it!
make install
What’s Next
You’ve got Emacs 29.1 installed and running! This is exciting. Which new features do you want to try first?
Three weeks on the trot. Definitely calling that a win.
Also, Good Omens 2 is a delight. Still enough of Terry’s character hanging
around it, and the new writers help it not feel too Neil-y.
Wednesday
After a bit of fiddling, I’ve worked out how to add helpers to the Emacs `C-x 8` keymap, so now I have shortcuts for typing ‘λ’, ‘🙂’ and various other characters that I type more or less frequently. Beats the crap out of doing `C-x 8 <ret>` and then typing out the name of the character I’m looking for.
You can no doubt use define-key as well, but I find `general` or `bind-keys` to be much nicer to work with. The latter has the advantage that it’s included in Emacs as part of `use-package` and plays nice with `which-key`, so I might go and redo my key bindings and get rid of `general`, nice as it is, since the real selling point of that library is how easy it is to bind stuff in `evil-mode` states.
Sunday
I still miss Twitch Sings. It’s how I started streaming—long before the Friday night Song Swaps and folk streams. I’d be happily belting out Lady Gaga’s Bad Romance, hamming it up to You Spin Me Round or giving it my best Johnny Cash
Not a particularly good impression. I can’t get that low!
on Hurt. It was just huge fun and a great way to make friends on Twitch.
Twitch ended up pulling the plug because it was a free app and… well, free apps and sync rights really don’t play well together.
You’ll still find people doing Karaoke on Twitch though, many of them the same faces I met back in Twitch Sings days. This morning, I woke up early and spotted some friends Karaoke-ing it up on a Discord, so I pulled on pyjamas and went and joined ’em for a few songs. These days, I just use Loopy Pro rather than searching YouTube for backing tracks. It’s great fun though, and definitely makes for a more enjoyable way to spend the occasional hour or so of early morning insomnia.
Singing in company, even virtual company is still the best thing you can do in public with your clothes on. I encourage you all to sing more. What’s the worst that could happen?
Small victory of the week: Actually got off my arse and did something about selling off my old Magic the Gathering cards. For my next trick, I hope to do the same with my collection of [mostly card] magic books.
Tuesday
Made a capture template for adding a week note. Support functions are currently not the prettiest, and don’t deal with a bunch of corner cases, but they seem to work for my case, so I’ll leave ’em be for the time being. I plan to write it up in a longer post, and that will no doubt tweak my coder pride enough to make things suck a little less.
Oh god, once I start fiddling with my Emacs configuration, it’s impossible to stop!
Wednesday
Nipped over to Mum and Dad’s for lunch at Zini’s, and to borrow dad’s drills for my on going cigar box MIDI controller project. Managed to get eight holes accurately placed enough that I only had to drill 7.8mm holes for the M7 threaded rotary encoders I’d soldered to my stripboard. I’m calling that a win! Next trick, get the microcontroller wired up and appropriate software written.
Also discussed making PID controller I promised to make dad for his heat treatment setup a while back. A Pi Pico and one of its mini displays looks like it should do the job nicely. The plan is to make an extension cable with an SSR as a separate bit of kit, then control that from the prototype controller. Once they’re working as separate parts, we can work out how to bring it all into one container. I shall wuss out of making the kind of thing I saw in a commercial radio controlled plug, which powered the control circuit with a very simple capacitor based power supply, with the slightly worrying wrinkle that the controller’s 0V line was floating at around 5V below mains Vmax. Clever, sure, but scarier than I’m prepared to work with.
Friday
Holy crap, but old Magic the Gathering cards are getting horrifically pricy. According to the buy list of the shop I just took my cards in to, I should be expecting about £400 for just four of my cards. And probably another couple of hundred for the two dual lands (assuming they’re not from the Unlimited set, in which case they’re worth a lot more). All being well that’s covered the cost of getting my grandfather’s old recliner reupholstered and fixed.
If I could be arsed with it, I could probably get a lot more by selling direct on eBay, but I was already losing the will to live just sorting things out to take in to the shop.
Do not ask me about the Tabernacle at Pendrell Vale and Black Lotus that I sold far too early, because that might make me grumpy.
In this post I announce that Democratize can now import shortdoc-based examples, mention that the hundreds of examples from shortdoc.el and treesit.el are already available, show some stats, explain advantages of using Democratize to see shortdoc examples in Helpful (or regular Help) buffers, and invite you to suggest other libraries that provide documentation in shortdoc format.
Democratize can extract thousands of usage examples (aka "demos") from some of your favorite Emacs Lisp libraries and, among other things, insert them into Help or Helpful buffers when you look up a function.
I show how all sorts of key–value collections can be let-bound with dotted symbols in Emacs Lisp: lists, vectors, lines, key–value lines, cons pairs, alists, plists, JSONs, Org tables, lists of lists, TSVs, CSVs, SSVs, and hash tables.
Last year, I wrote about converting Org to DOCX with pandoc. Well, that particular method has needed some improvements. I needed to also support converting Markdown files, and more vitally, I needed to support the new-ish org-cite citation format.
For citations to work, you need to have a Citation Style Language (CSL) file. Zotero comes with a ton of them, so check your Zotero installation for examples.
In the updated fish shell function below, you will want to update both refdoc and csldoc to point to your reference DOCX file and your CSL file, respectively.
function org2docx --description 'Generate a DOCX file using a custom reference document'
set -l refdoc "$PATH_TO_REFERENCE_DOCX"
set -l csldoc "$PATH_TO_CSL"
set -l fromfmt (string match -r '(?:org|md)$' $argv)
set -l base (basename -s .$fromfmt $argv)
echo Generating $base.docx ...
pandoc --from "$fromfmt"+citations \
--citeproc --csl $csldoc \
--reference-doc $refdoc -o $base.docx $argv
end
And there you have it! Now you can convert both Org files and Markdown files to DOCX. And I am sorry that you have to use DOCX!
A recent project involves delivering a finished product as a collection of Jupyter Notebooks. This process involves using Emacs for writing, Git for version control, and a slightly tricky process for enabling non-Jupyter, non-Emacs users to perform document review.
Writing—just like programming—ideally includes a review process before anything is delivered to the client. Even first drafts need at least two readers before delivery. I’ve previously discussed how I use Org Mode and Pandoc to deliver DOCX files, and DOCX or ODF files unfortunately remain the easiest way to track changes and edits among word processor users.
Since Jupyter Notebooks are basically JSON documents, the best way to keep track of changes and revisions is using some kind of version control. Here is one process using Git and GitHub.
When my notebook files are ready for review, converting them to DOCX files is pretty straightforward.
First, I use the jupyter command line tool to convert to Markdown, like this:
jupyter nbconvert --to markdown *.ipynb
Next, I use Pandoc to convert to DOCX using a reference link.
for file in *.md
pandoc --reference-doc $path_to_refdoc -o $file.docx $file
end
Renaming files with Dired
At this point, I needed to rename all of the DOCX files and move them to the proper shared folder, so my reviewer could get to them and know what’s going on. We have a naming format for filenames that helps us track project and versions, so all of the files needed to have at least a v1 in them.
Emacs has a file manager called Dired, which contains powerful features that allow you to modify directory contents just like any other buffer. I now had a bunch of files that ended in .md.docx that needed to instead end in -v1.docx. Here is the process I used to easily rename them.
In Emacs, use M-x dired to open the directory.
Use C-x C-q to run dired-toggle-read-only.
Use M-% to run query-replace, and replace .md.docx with -v1.docx.
Finish “writing” the directory with C-c C-c.
All done! It was nice and simple. The DOCX files were finally properly named and ready for review.
This static blog is built using Jekyll in 2014. It survived after 7
years which is a success when it comes to personal blogging. Part of
the reason is having a good blogging workflow: write posts in Org
Mode, export to HTML with a front matter, build the site using Jekyll,
send the folder to an Amazon S3 bucket, and that’s it. All done in
Emacs of course.
Technical Debt
I added a few things to the workflow to enhance the reading experience
including code highlights, centred images with caption, table of
content etc. There are more features I want to add but at the same
time, I want to be able to just write.
With that mindset, whenever there are issues, I apply quick fixes
without a deep understanding of the actual causes. It seems efficient
until recently some fixes become counter-productive.
I started seeing underscore (_) is exported as \_ and <p> tag
appears in code snippets. It all sounds like quick fix, but I just
couldn’t get it correct after few hours. For the last few posts, I had
to manually fix them for each of the read-edit-export-fix iteration.
Revisit the Tech Stack
I have an ambitious goal for this blog. So it is time to go sweep the
carpet. I studied the technologies used for this blog, Jekyll, AWS and
Org Mode exporting. It was a good chance to practise Org-roam for
taking atomic notes. The time is well spent as I learnt a lot.
I was impressed I got the whole thing up and running 7 years ago. I
don’t think I have the willpower to do it now.
Still, there are a lot of things that I do not have a good understand,
e.g. the Liquid templates, HTML and CSS tags etc. The syntax just
puts me off.
Long Ride with Jekyll
I prefer a simple format like Org Mode or Markdown and don’t have to
deal with HTML/CSS at all. There are a couple of occasions when I
cannot resist the temptation to look for an alternative to
Jekyll. There’s no luck in the search. It seems HTML is the only way
because it is native to the web.
So the plan is to stick with Jekyll for at least a few years. In the
next couple of weeks, I’d try to fix all the issues, after that,
gradually add more features to enhance the writing and reading
experience.
I hope people who also uses the similar tech stack (Org-mode, Emacs,
Jekyll, AWS) can benefit my work.
I switched to MacOS last year for editing home gym videos. I was and
am still amazed by how fast the M1 chip is for exporting 4K
videos. The MacOS also enriched the Emacs experience which makes it
deserve another blog post.
So I have been slowly adapting my Emacs configuration and workflow to
MacOS. One of the changes is the Emacs server.
The goal is to have fully loaded Emacs instances running all the time
so I can use them at any time and anywhere, in Terminal or Spotlight. They are
initiated upon login. In cases of Emacs crashes (it is rare but more
often than I like) or I have to stop them because I messed up the
configuration, they restart automatically.
It is an extension of Emacs Plus’ plist file. I made a few changes for
running two Emacs servers: one for work (data sciences, research) and
one for personal usage (GTD, books). Taking the “work” server as an
example, the important attributes of the plist configuration file are:
Line 5: The unique service name to launchctl
Line 8: The full path to the Emacs program. In my case, it is
/opt/homebrew/opt/emacs-plus@31/bin/emacs
Line 9: The “–fg-daemon” option set the Emacs server name to
“work”. Later I can connect to this server by specifying “-s=work”
option to emacsclient
Line 13: The KeepAlive is set to true so it keeps trying to
restart the server in case of failures
Line 16 and 18: The location of standard output and error
files. They are used to debug. Occasionally I have to check those
files to see why Emacs servers stopped working, usually because of
me introducing bugs in my .emacs.d.
With the updated plist files in place, I start the Emacs servers with
The launchctl list | grep -i emacs is a handy snippet that lists the
status of the services whose name includes “emacs”. The output I have
right now is
PID
Exit Code
Server ID
1757
0
emacs_org
56696
0
emacs_work
It shows both Emacs servers are running fine with exit code 0.
Launch Emacs GUI in Terminal
I can now open a Emacs GUI and connect it to the “work” Emacs server
by running emacsclient -c -s work &. The -c option
Launch Emacs GUI in Spotlight
In MacOS, I found it is natural to open applications using Spotlight,
for example, type ⌘ + space to invoke Spotlight, put “work” in the
search bar, it narrows the search down to “emacs_work” application,
and hit return to finalise the search. It achieves the same thing as
the command above but can be used anywhere.
I uploaded a demo video on YouTube to show it in action. You might want
to watch it at 0.5x speed because I typed so fast…
To implement this shortcut, open “Automator” application, start a new
“Application”, select “Run Shell Script”, and paste the following bash
code
/opt/homebrew/opt/emacs-plus@31/bin/emacsclient \--no-wait\--quiet\--suppress-output\--create-frame-s work \"$@"
and save it as emacsclient_work in the ~/Application
folder.
Essentially, the bash script above is wrapped up as a MacOS
application, named emacsclient_work and the Spotlight searches the
application folder by default.
I’m working on replicating the (Re-)Imag(in)ing Price Trends paper -
the idea is to train a Convolutional Neutral Network (CNN) "trader" to
predict the stocks' return. What makes this paper interesting is the
model uses images of the pricing data, not in the traditional
time-series format. It takes financial charts like the one below
and tries to mimic the traders' behaviours to buy and sell stocks to
optimise future returns.
Alphabet 5-days Bar Chart Shows OHLC Price and Volume Data
To train the model, the price and volume data are transformed into
black-white images which is just a 2D matrix with 0s and 1s. For just
around 100 stocks' pricing history, there are around 1.2 million
images in total.
I used the on-the-fly imaging process during training: in each batch,
it loads pricing data for a given stock, sample one day in the
history, slice a chunk of pricing data, and then convert it to an image. It
takes about 0.2 milliseconds (ms) to do all that, so in total it takes 4
minutes to loop through all the 1.2 million images.
1.92 ms ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
To train 10 epochs, that's 40 minutes in loading data. To train one
epoch on the full dataset with 5,000 stocks, that's 200 minutes in
loading data alone!
PyToch utilises multiple processing in loading the data using CPU
while training using GPU. So the problem is less severe, but I'm using
the needle, the deep learning framework we developed during the
course, it does have this functionality yet.
During training using needle, the GPU utilisation is only around
50%. After all the components in the end-to-end are almost completed,
it is time to train with more data, go deeper (larger/more complicated
morel), try hyper-parameters tuning etc.
But before moving to the next stage, I need to improve the IO.
Scipy Sparse Matrix
In the image above, there are a lot of black pixels or zeros in the data
matrix. In general only 5%-10% of pixels are white in this dataset.
So my first attempt was to use scipy's spare matrix instead of numpy's
dense matrix: I save the sparse matrix, loaded it, and then convert it
back to a dense matrix for training CNN model.
967 µs ± 4.99 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
It reduces the IO time to 1ms, so about half of the time, not bad,
but I was expecting a lot more as the sparseness is high.
Numpy Bites
Then I realised the data behind images is just 0 and 1, in fact, a lot
of zeros, and only some are 1. So I can ignore the 0s and only need to
save those 1s, then reconstruct the images using those 1.
It is so simple that numpy has functions for this type of data
processing already. The numpy.packbites function converts the image
matrix of 0 and 1 into a 1D array whose values indicate where the 1s
are. Then the numpy.unpackbits does the inverse: it reconstructs the
image matrix by using the 1D location array.
This process reduces the time of loading one image to 0.2
milliseconds, that's 10 times faster than the on-the-fly method with
only a few lines of code.
194 µs ± 3.95 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
Another benefit is the file size is much smaller: it is 188 bytes
compared to 1104 bytes using sparse matrix. So it takes only 226MB of
disk space to save 1.2 million images!
It takes a couple of minutes to generate 1.2 million files on my Debian
machine. It is so quick! But then I release this approach is not
scalable without modification because there's a limited number of
files the OS can accommodate. The technical term is Inode. According
to this StackExchange question, once the filesystem is created, one
cannot increase the limit (Yes, I was there).
Without going down to the database route, one quick workaround is to
bundle the images together, for example, 256 images in one file. So
later in training, load 256 images in one go, then split them into
chunks. Just ensure the number of images is a multiple of the batch
size used in training so I don't have to deal with unequal batch
sizes. Since those bundled images are trained together, it reduces the
randomness of SGD, so I won't bundle too many images together, 256
sounds about right.
The LSP and other tools can cause problems when they are monitoring
folders with a large number of files. Moving them out of the project
folder is the way to go so Emacs won't complain or freeze.
I have been working on the Deep Learning System course. It is the
hardest course I ever studied after university. I would never thought
that I need CI for a personal study project. It just shows how
complex this course is.
Here is the setup: the goal is to develop a pytorch-like DL library
that supports ndarray ops, autograd, and to implement DL models, LSTM
for example, from scratch. That's the exciting math part. The tricky
part is it supports both CPU devices with C++11 and GPU devices with
Cuda. On the user front, the interface is written in Python. I worked
on my M1 laptop most of the time, and switch to my Debian desktop for
Cuda implementation.
It was a fine Saturday afternoon, I made a breakthrough in implementing
the gradient of Convolution Ops in Python after couple of hours of
tinkering in a local coffee shop. I rushed home, boosted up Debian
to test the Cuda backend, only to find "illegible memory access"
error!
It took me a few cycles of rolling back to the previous change in git to
find where the problems are. It made me think about the needs of
CI. In the ideal scenario, I would have a CI that automatically runs
the tests on the CPU and Cuda devices to ensure one bug-fix on CPU
side doesn't introduce new bugs on the Cuda, and vice versa. But I
don't have this setup at home.
Two Components of PoorMan CI
So I implemented what I call PoorMan CI. It is a semi-automated
process that gives me some benefits of the full CI. I tried hard to
refrain from doing anything fancy because I don't have
time. The final homework is due in a few days. The outcome is simple yet
powerful.
The PoorMan CI consists of two parts:
a bunch of bash functions that I can call to run the tests, capture
the outputs, save them in a file, and version control it
For example, wrap the below snippet in a single function
pytest -l-v-k"not training and cuda"\> test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log
git add test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log
a log file where I keep track of the code changes, and if the new
change fixes anything, or breaks anything.
In the example below, I have a bullet point for each change committed
to git with a short summary, and a link to the test results. The
fce5edb and f43d7ab are the git commit hash values.
- fix grid setup, from (M, N) to (P, M)!
[[file:test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log]]
- ensure all data/parameters are in the right device. cpu and cuda, all pass! milestone.
[[file:test_results/2022_12_11_13_51_22__f43d7ab__fast_and_cuda.log]]
As you can see, it is very simple!
Benefits
It changed my development cycle a bit: each time before I can claim
something is done or fixed, I run this process which takes about 2
mins for two fast runs. I would use this time to reflect on what I've
done so far, write down a short summary about what's got fixed and
what's broken, check in the test results to git, update the test log
file etc.
It sounds tedious, but I found myself enjoying doing it, it
gives me confidence and reassurance about the progress I'm making. The
time in reflecting also gives my brain a break and provides clarity on
where to go next.
During my few hours of using it, it amazes me how easy it is to
introduce new issues while fixing existing ones.
Implement in Org-mode
I don't have to use Org-mode for this, but I don't want to leave Emacs
:) Plus, Org-mode shines in literate programming where code and
documentation are put together.
This is actually how I implemented it in the first place. This section
is dedicated to showing how to do it in Org-mode. I'm sure I will come
back to this shortly, so it serves as documentation for myself.
Here is what I did: I have a file called poorman_ci.org, a full
example can be found at this gist. An extract is demonstrated below.
I group all the tests logistically together into "fast and cpu", "fast
and cuda", "slow and cuda", "slow and cuda". I have a top level header
named group tests, Each group has their 2nd-level header.
The top header has a property drawer where I specify the shell session
within which the tests are run so that
* grouped tests
:PROPERTIES:
:CREATED: [2022-12-10 Sat 11:32]
:header-args:sh: :session *hw4_test_runner* :async :results output :eval no
:END:
it is persistent. I can switch to the shell buffer named
hw4_test_runner and do something if needed
it runs asynchronically on the background
All the shell code block under the grouped tests inherits those
attributes.
The first code block defines variables that used to create a run
id. It uses the timestamp and the git commit hash value. The run id is
used for all the code blocks.
#+begin_src sh :eval nowd="./test_results/"ts=$(date +"%Y_%m_%d_%H_%M_%S")git_hash=$(git rev-parse --verify--short HEAD)echo"run id: "${ts}__${git_hash}$#+end_src
To run the code block, move the cursor inside the code block, and hit C-c
C-c (control c control c).
Then I define the first code block to run all the tests on CPU except
language model training. I name this batch of tests "fast and cpu".
#+begin_src sh :var fname="fast_and_cpu.log"fname_full=${wd}/${ts}__${git_hash}__${fname}
pytest -l-v-k"not language_training and cpu"\
2>&1 | tee${fname_full}#+end_src
It creates the full path of the test results. The fname variable
is set at the code clock header, this is a nice feature of
Org-mode.
pytest provides an intuitive interface for filtering tests, here
I use "not language_training and cpu".
The tee program is used to show the outputs and errors and at the
same time save them to a file.
Similarly, I define code blocks for "fast and cuda", "slow and cpu",
"slow and cuda".
So at the end of the development cycle, I open the poorman_ci.org
file, run the code blocks sequentially, and manually update the change
log. That's all.
Functional programming offers a bunch of really cool programming patterns. Two that I really enjoy are tail recursion and pattern matching, especially how they are implemented in OCaml. However, I spend a lot of time writing Emacs Lisp now, and I was wondering if I could find a way to use these patterns in that language.
It turns out that it is possible, thanks to named-let and pcase. It isn’t as pretty and elegant as OCaml, but at least I get to keep excercising those parts of my programming brain. Maybe next I’ll try to figure out currying in Emacs Lisp.
Note that this blog post includes some really dumb examples, because that’s usually how I learn these things best.
Pattern Matching with pcase
Most programmers will be familiar with the granddaddy of pattern matching, the switch/case construct. This is present in many, many programming languages, especially those in the ALGOL family.
However, pattern matching can be so much more! Instead of simple equality checks, pattern matching extends the switch/case concept to include testing for all kinds of patterns and conditionals.
Lisp programmers will already be familiar with cond, which can be used to sequentially test for conditionals. However, functional language programmers have probably fallen in love with a more mature and sophisticated form of pattern matching that cond doesn’t totally satisfy.
Fortunately, Emacs Lisp has pcase, the pattern-matching conditional. Here is an example of its use to duplicate car, which is the dumbest possible example I could think of.
You can see that pcase has a backquote syntax for matching various constructs, such as with the `(,head . ,_) piece. This matches a cons cell and binds the CAR to head while ignoring the CDR.
The next case is just _, which is a catch-all matching operator.
Tail call optimization (TCO) is the programming language feature that allows efficient tail recursion without overflowing your stack. It is increasingly common in languages today, though from what I’ve seen, it always involves caveats.
In Emacs Lisp, the easiest way to use TCO that I’ve come across is the named-let macro. With it, you define a function that can get “unrolled” inside another function. For example, here is a simple function that calculates a factorial using tail recursion.
In this example, you will notice that rec-fact is the locally named function that gets called at the end of the first pcase pattern. This is a tail call! It will get optimized.
You can check this out by running something like (ela/fact 5) and getting 120 as the result. Try using a ridiculously big number and see if you get a stack overflow! You shouldn’t.
Another Example: Summing a List
This is just a nostalgic example, since it’s probably the first tail recursive pattern matching function I ever wrote when learning OCaml a zillion years ago. This function will take a list of numbers and then add them all together. There are much better ways to write this in Emacs Lisp, like with apply.
And you will end up with exactly the result you expect. I was amused to see that the documentation page for named-let has a different implementation of this function that doesn’t use pcase.
Oh heck, let’s get fancy and rewrite apply using this approach.
(defunela/apply(fn&restarguments)"Apply FN to each element of ARGUMENTS and return the accumulated result.";; Set up accumulator to the right type.(let*((arguments-flat(flatten-listarguments))(initial-value(pcase(cararguments-flat)((predintegerp)0)((predstringp)"")(_nil))))(named-letapply-rec((accumulatorinitial-value)(input-listarguments-flat))(pcaseinput-list(`(,head.,tail)(apply-rec(funcallfnaccumulatorhead)tail))(_accumulator)))))
I am certain this version of apply has bugs, but it works for + and concat, so that’s good enough for a simple example. And it uses pcase twice!
Hopefully this has been a useful blog post for somebody out there. Let me know in the comments if there are other fun things you have done with TCO and pattern matching in Emacs Lisp!
Once you have the ddate package installed, you can use it to add the Discordian date to your dashboard with code like
this:
(defunela/dashboard-insert-ddate(list-size)"Insert the Discordian date into the dashboard."(let((ddate-string(ddate-pretty)))(dashboard-center-lineddate-string)(insertddate-string)))(use-packagedashboard:init(dashboard-setup-startup-hook):config;; Add the ddate item provider to the list.(add-to-list'dashboard-item-generators'(ddate.ela/dashboard-insert-ddate));; Set up your items with ddate at the top.(setqdashboard-items'((ddate)(recents.5)(bookmarks.5)(registers.5))))
Freelance writers need to deliver documents in the format requested by clients. However, frequently
the requested format is not the writer’s preferred working format. I like to write in Org Mode, but
many clients prefer delivery in Microsoft Word’s DOCX format.
This is how I generate DOCX files for my clients.
What is Org Mode?
Org Mode is an Emacs package for writing and working with Org files. Org
files are highly structured plain text files that may appear to be a text outline, but can do so
much more. Org Mode is incredibly versatile, and can be used to track projects, manage schedules,
write outlines, and even create documents.
Choosing between Org Export and Pandoc
Org Export runs inside Emacs and is capable of converting Org files to a variety of other
formats. While it is very powerful, it also has its
idiosyncrasies. For instance, when converting Org files to DOCX files, it uses its own style names
such as “Org Title” and “Org Heading 1”.
A second option for converting Org files to DOCX files is Pandoc. Pandoc
prides itself on being the Swiss army knife of document format conversion. It handles an impressive
variety of document formats and handles a dizzying collection of configuration options.
Since the DOCX files that I create need to be shared with other writers, editors, and reviewers, I
need to make sure that they are easy to work with. This influenced my decision. Since Pandoc uses
more standard style names, I decided to use
it for Org conversion.
Setting up Pandoc
To use Pandoc to generate nice looking DOCX files, you will need to configure a template
document. The recommended method for doing this is to generate a default template using Pandoc, and
then edit it in Word. I used LibreOffice Writer for this, and it
worked just fine.
Open reference.docx in your word processor and edit the styles so they meet your needs.
Converting from Org to DOCX
One option for running the conversion is to take advantage of the
ox-pandoc package for Emacs. If you will always be
using the same configuration for your exports, this is a great option.
However, I need to use a number of different configurations for converting documents, so I tend to
run Pandoc from the command line. Recent versions of ox-pandoc support passing options via Org
headers, but I still haven’t
bothered to set that up. It should be very easy to template this using
Yasnippet, though.
I use a custom fish shell function that looks like this:
function org2docx --description 'Generate a DOCX file using a custom reference document'
set -l refdoc "$PATH_TO_REFERENCE_DOCX"
set -l base (basename -s .org $argv)
echo Generating $base.docx ...
pandoc --reference-doc $refdoc -o $base.docx $argv
end
From my fish shell command line, I can then just run org2docx whatever.org to generate
whatever.docx.
I have not found a level of automation that makes my converted DOCX files completely perfect,
unfortunately. After conversion, I always open the new file in my word processor to make final
tweaks and fixes.
Have fun converting files!
The method I’ve outline in this blog post is straightforward and fits my needs. There are definitely
improvements to be made, such as using templates to pass the proper options to Pandoc. Switching to
ox-pandoc would mean one fewer reason to leave Emacs, after all.
In recent years, more and more clients are asking for files to be delivered via Google Docs. So far,
I have yet to find a good conversion pipeline to get Org files into Google Docs easily. My method
right now takes too many manual steps. That’s a problem I would love to solve.
Do you have a conversion pipeline for documents that works for you? Leave me a comment and let me
know!
As a freelance writer, I need to be ready to deliver high quality copy in a timely fashion. My editor of choice for writing is
Emacs. I have found that Yasnippet templates have streamlined my writing process.
As a templating system for Emacs, Yasnippet is well known by programmers. It can quickly expand
function definitions, control structures, and other templates into blocks of source code. Source code is just text.
Since most of my writing originates in text format, templates are great for speeding up document
creation and avoiding common errors and omissions.
Installing Yasnippet
With use-package, you can get started quickly with Yasnippet by including the following in your
Emacs init file.
This blog is primarily written in Markdown for Jekyll, which means that each file needs a YAML block at the top with specific
information. I have a template that looks like this:
Not only does this template save me from needing to remember the format of the YAML block, but it ensures that frequently forgetten items
are included, such as the permalink setting and the <!--more--> tag.
My Org Mode files are more complex. Many writing clients expect deliveries in Microsoft Word files, but
I also frequently find myself needing to produce OpenDocument files, HTML, and even Markdown. The headers for these files need to support
all of these options.
I use the following template to support all of the Org Mode configuration and options I require.
I also use Org Mode to publish to WordPress websites using org2blog. For one of these websites, I
write show notes for podcast episodes. I use a template that looks like this.
This template reminds me of important items I need to include in all show notes, like links, credits, and a Patreon link.
More Bonus Snippets
The official snippet collection also contains a bunch of Org Mode snippets. Here is the whole
list. Since Org Mode contains so many complicated
structures like source blocks and optional keywords, these are great time savers. There are also snippets for supporting
org-reveal.
Writing Faster is Writing Smarter
Don’t let your tools get in the way of your writing. Frequently, Emacs is portrayed as cumbersome and filled with obscure keybindings and
weird commands. However, I have found that it is an excellent tool for writing. Yasnippet templates let me get started on new documents
quickly without fretting over different syntaxes for configuration.
And Yasnippet is just one of the tools I use when writing with Emacs. I will discuss more of these tools in the future.
As a long-time Linux user, I am pretty comfortable with command line interfaces. However, as I started learning more about automation and how important it is to get your computer to do more work for you, I leaned toward wanting my command line shell to do more. Eventually I switched to fish shell, a very user-friendly shell with excellent scripting capabilities that is far more readable and less obscure than bash.
Plugins for fish
There have been a few plugin managers over the years, but the one that seems to be the best maintained and most usable is fisher, created by Jorge Bucaran. It allows you to easily install plugins straight from GitHub repositories, while updates can be executed with a simple fisher update on the command line.
z
One of the greatest command-line time savers ever is the z command. Originally a bash script, it has been ported to fish and, frankly, I love it.
What does it do?z provides shorthand for visiting directories based on a combination of frequency and recency—or "frecency".
If you frequently visit the /opt/calibre directory, a z calib command will allow you to jump straight there.
If you already have fisher installed, just use fisher install jethrokuan/z and start playing around with z.
Your IDE has been handling your parenthesis for ages. Now your shell can do the same! The pisces plugin has automatic matching symbol management for parenthesis, braces, quotes, and others.
Install it with fisher using the command fisher install laughedelic/pisces and be matching parens in seconds!
Automation can help you interact with your editor from your shell. The Emacs plugin for fish was originally written for an unmaintained plugin manager called Oh My Fish. However, I wanted the plugin to work with fisher, so I made my own fork.
How does it work? This plugin adds easy commands like ef and ed that allow you to open files and directories in Emacs quickly.
Install it quickly using the command fisher install pymander/plugin-emacs and read more about it here.
How does it work? Inside a vterm session, this plugin adds fish commands like vf and vd to open files and directories quickly. It mirrors the Emacs plugin.
Install it using the command fisher install pymander/vfish and read more about it here. Note that it will require some code to be added to your Emacs startup file.
apt
For Debian and Ubuntu users, the Oh My Fish apt plugin works seamlessly with fisher.
What does it do? The apt plugin adds a wrapper around the apt and apt-get commands that simplify package management from the command line.
Install this plugin quickly with fisher install oh-my-fish/plugin-apt and then read more about it.
What's your favorite fisher plugin?
I love finding plugins that can help me automate more of my everyday tasks and workflows. What's your favorite fisher plugin?
Leave a comment below and let me know! Be sure to include a link.
For machine learning projects, I tweaked my workflow so the
interaction with remote server is kept as less as possible. I prefer
to do everything locally on my laptop (M1 Pro) where I have all the
tools for the job to do data analysis, visualisation, debugging etc
and I can do all those without lagging or WI-FI.
The only usage of servers is running computation extensive tasks like
recursive feature selection, hyperparameter tuning etc. For that I ssh
to the server, start tmux, git pull to update the codebase, run a
bash script that I prepared locally to fire hundreds of
experiments. All done in Emacs of course thanks to Lukas Fürmetz’s
vterm.
The only thing left is getting the experiment results back to my
laptop. I used two approaches for copying the data to local: file
manager GUI and rsync tool in CLI.
Recently I discovered dired-rsync that works like a charm - it
combines the two approaches above, providing a interactive way of
running rsync tool in Emacs. What’s more, it is integrated
seamlessly into my current workflow.
They all have their own use case. In this post, I brief describe those
three approaches for coping files with a focus on dired-rsync in
terms of how to use it, how to setup, and my thoughts on how to
enhance it.
Note the RL stands for remote location, i.e. a folder a in remote
server, and LL stands for local location, the RL’s counterpart. The
action in discussion is how to efficiently copying files from RL to
LL.
File Manager GUI
This is the simplest approach requires little technical skills. The RL
is mounted in the file manager which acts as an access point so it can
be used just like a local folder.
I usually have two tabs open side by side, one for RL, and one for LL,
compare the differences, and then copy what are useful and exists in
RL but not in LL.
I used this approach on my Windows work laptop where rsync is not
available so I have to copy files manually.
Rsync Tool in CLI
The rsync tool is similar to cp and scp but it is much more
power:
It copies files incrementally so it can stop at anytime without
losing progress
The output shows what files are copied, what are remaining, copying
speed, overall progress etc
Files and folders can be included/excluded by specifying
patterns
I have a bash function in the project’s script folder as a shorthand
like this
copy_from_debian_to_laptop (){# first argument to this functionfolder_to_sync=$1# define where the RL is remote_project_dir=debian:~/Projects/2022-May
# define where the LL is local_project_dir=~/Projects/2022-May
rsync -avh--progress\${remote_project_dir}/${folder_to_sync}/ \${local_project_dir}/${folder_to_sync}}
To use it, I firstly cd (change directory) to the project directory
in terminal, call copy_from_debian_to_laptop function, and use the
TAB completion to quickly get the directory I want to copy, for
example
This function is called more often from a org-mode file where I kept
track of all the experiments.
Emacs’ Way: dired-rsync
This approach is a blend of the previous two, enable user to enjoy the
benefits of GUI for exploring and the power of rsync.
What’s more, it integrates so well into the current workflow by simply
switching from calling dired-copy to calling dired-rsync, or
pressing r key instead of C key by using the configuration in this
post.
To those who are not familiar with copying files using dired in
Emacs, here is the step by step process:
Open two dired buffer, one at RL and one at LL, either manually
or using bookmarks
Mark the files/folders to copy in the RL dired buffer
Press r key to invoke dired-rsync
It asks for what to copy to. The default destination is LL so press
Enter to confirm.
After that, a unique process buffer, named *rsync with a timestamp
suffix, is created to show the rsync output. I can stop the copying by
killing the process buffer.
Setup for dired=rsync
The dired-rsync-options control the output shown in the process
buffer. It defaults to “-az –info=progress2”. It shows the overall
progress in one-line, clean and neat (not in MacOS though, see Issue
36). Sometimes I prefer “-azh –progress” so I can see exactly which
files are copied.
There are other options for showing progress in modeline
(dired-rsync-modeline-status), hooks for sending notifications on
failure/success (dired-rsync-failed-hook and
dired-rsync-success-hook).
Overall the library is well designed, and the default options work for
me, so I can have a bare-minimal configuration as below (borrowed from
ispinfx):
There are two more things to do on the system side:
In macOS, the default rsync is a 2010 version. It does not work
with the latest rsync I have on Debian server so I upgrade it using
brew install rsync.
There no way of typing password as a limitation of using process
buffer so I have to ensure I can rsync without remote server asking
for password. It sounds complicated but fortunately it takes few
steps to do as in Setup Rsync Between Two Servers Without Password.
Enhance dired-rsync with compilation mode
It’s such a great library that makes my life much easier. It can be
improved further to provide greater user experience, for example, keep
the process buffer alive as a log after the coping finished because
the user might want to have a look later.
At the moment, there’s no easy way of changing the arguments send to
rsync. I might want to test a dry-run (adding -n argument) so I can
see exactly what files are going to be copied before running, or I
need to exclude certain files/folders, or rerun the coping if there’s
new files generated on RL.
If you used compilation buffer before, you know where I am
going. That’s right, I am thinking of turning the rsync process buffer
into compilation mode, then it would inherit these two features:
Press g to rerun the rsync command when I know there are new
files generated on the RL
Press C-u g (g with prefix) to change the rsync arguments before
running it for dry-run, inclusion or exclusion
I don’t have much experience in elisp but I had a quick look at source
code, it seems there’s no easy of implementing this idea so something
to add to my ever-growing Emacs wish-list.
The good thing about Emacs is that you can always tweak it to suit
your needs. For years I’ve been doing it for productivity reasons. Now
for the first time, I’m doing it for health reasons.
Life can be sht sometimes, when I was in my mid 20s, I was reshaping
every aspects of my life for good. But optician told me my vision can
only get worse. I wasn’t paying much attention, busy with my first
job and learning.
Last month, I was told my right eye’s vision got whole point worse,
whatever that means. Now I’m wearing a new pair of glasses, seeing the
world in 4K using both eyes, noticing so much details. It makes the
world so vibrate and exciting. It comes with a price though, my eyes
get tired quickly, and it become so easy to get annoyed by little
things.
One of them is switching windows in Emacs. Even though I am in the
period of calibrating to the new glasses, I decided to take some
actions.
Ace-Window
Depends on the complexity of the tasks, I usually have about 4-8
windows laid on my 32 inch monitor. If that’s not enough, I would have
an additional frame of similar windows layout, doubling the number of
windows to 8-16.
So I found myself switching between windows all the time. The action
itself is straightforward with ace-window.
The process can be breakdwon into five steps:
Invoke ace-window command by pressing F2 key,
The Emacs buffers fade-in,
A red number pops-up at the top left corner of each window,
I press the number key to switch the window it associates with,
After that, the content in each Emacs buffer are brought back.
This approach depends on visual feedback - I have to look at the
corner of the window to see the number. Also, the screen flashes
twice during the process.
I tried removing the background dimming, increase the font size of the
number to make it easier to see, and bunch of other tweaks.
In the end, my eyes were not satisfied.
Windmove
So I started looking for alternative approaches and found windmove
which is built-in.
The idea is simple - keep move to the adjacent window by move left,
right, up, or down until it arrives at the window I want.
So it uses the relative location between windows instead of assigning
each window a unique number and then using the number for switching.
Is it really better? Well with this approach, I use my eyes a lot less
as I do not have to look for the number. Plus, I feel this is more
nature as I do not need to work out the directions, somehow I just
know I need to move right twice or whatever to get to the destination.
The only issue I had so far is the conflicts with org-mode’s
calendar. I like the keybinding in org-mode, so I disabled windmove
in org-mode’s calendar with the help from this stackoverflow question.
The following five lines of code is all I need to use windmove.
I created an git branch for switching from ace-window to
windmove. I would try it for a month before merge it into master
branch.
Back to where it started
After using it for few days, I realised this is the very package I
used for switch windows back in 2014 when I started learning Emacs. I
later then switched to ace-window because it looks pretty cool.
Life is changing, my perspectives are changing, so is my Emacs
configuration. This time, it goes back to where I started 8 years ago.
Introduction of my new Emacs package XHT and comments about the advantages of using hash tables in Emacs Lisp, their speed, and possible barriers to a more widespread adoption.
My intention with this library is that: (1) dealing with hash tables in Emacs Lisp be pleasant, (2) hash tables become your go-to choice for most key–value operations in the language, and (3) for almost everything hash-table–related that you might want to do in Emacs Lisp, this library will have functions for it — or at least close enough that you can do it by composing a few of them.
I've about how it is viable to use a custom Emacs setup, but I didn't go too far into the details about what it's like to actually take that route. In this, the first post in a series on updating my custom Emacs setup, I'll talk about my workflow for upgrading packages to newer versions.
OrgReadme-fy helps you create a README.org for your Emacs projects. Features: (1) A readme-template.org so you don't have to start from scratch. This file will, upon creation, fetch some metadata from your package file (the main .el). It will have a skeleton of common sections. (2) A summary table and/or descriptive subtrees that feed readme-template.org to create the final README.org. This table and/or subtrees, in turn, are automatically generated from your package file and an optional examples.el file, providing a list of functions, their signature, their docstring, and examples of usage.
Exemplify-ERT helps you write clean-looking examples that double as regression tests. (1) Easier to write tests. (2) Easier to read and understand tests at a glance. (3) Tests easily exportable as they are to be shown as usage examples.
layout: announcement
title: Taking A Short Break
date: 2022-03-16 23:05:01
—
You may have noticed that there wasn’t an Emacs meetup this past
month. You may have even noticed that there wasn’t one this year.
We’re not gone! We’re taking a break.
The Emacs meetup has been running for eight years now. That’s wild to
think!
A lot has happened in those past eight years and in some ways a lot
has stayed the same.
We’ve experimented a bit with different formats and have found some
success. We learned early that relying on big flashy speakers hasn’t
been the most successful. Another thing that we learned is that it’s a
lot of work and we haven’t given it the attention it deserves.
Finally, this pandemic has been hard and that difficulty has only
exacerbated the struggles that have come with running this group.
We want to give this community the attention it deserves. So we’re
taking some time to rethink a bit and try to make something really
powerful for the community.
We don’t have a timeline for what we are hoping, but we don’t expect
to go any later than June this year.
We will keep everyone updated as much as possible with what’s going
on. Thank you for being part of this community and happy hacking.
After 3 months using my brand new MacBook Pro 14 M1 Pro, one of the
USB-C port stopped working. I will have to send it back, not sure what
Apple will do with it but I can't bear the risk of losing data. So I
need a backup.
In fact, I need to backup regularly for situation like this so that's
why I worked on it.
Wireless Backup Solution
The easiest solution is to get a USB-C portable SSD, plug it into my
laptop and open Time Machine to start back up, do it once a week and
call it a day.
But I'm reluctant to add more devices to my already cluttered home
lab. There are a few hard drives in the drawers, it would be good to
utilise them.
So I decided to set up a Time Machine backup solution using on my
Raspberry Pi 4. The benefits are
no additional costs, save me about £50-£100
no need to buy new stuff, so fewer things to care of
wireless backup to keep my desk clean
Later I realised the benefits of having a wireless backup is
overlooked. It can backup anytime and anywhere in my house. Also,
because of convenience, I can have more granular backups - instead of
weekly backup, I have hourly backup without getting the cables and
hard drives. I do less but get more value out of it.
The only concern I had was the speed. It turns out with SAMBA 3
protocol, I can get 55 MB/s write speed and 40 MB/s read speed from
laptop to Raspberry Pi. So in theory, it would take around 2.5 hours
to backup my 500 GB laptop. It might be a lot but only for the first
backup, the subsequent incremental backup would be much simpler and
faster, for example, as of now, the Time Machine completed a new
backup within 3 minutes in the background without my notice.
A portal USB-C SSD can finish the backup within minutes but it's an
overkill for an ordinary user like me and it's inconvenient.
So I'm satisfied with the current solution.
Set Up Raspberry Pi
I read a few guides on setting up Raspberry Pi for Time Machine, and I
found this guide most accurate and useful.
One thing I noticed is the AFP (Apple File Protocol) is deprecated, so
make sure you use SAMBA as the protocol.
Additionally, I followed this stack overflow answer to auto-mount the
SAMBA server so that every time I reboot my laptop, the Time Machine
will be ready to back up.
Time Machine Backup frequency
By default, Time Machine does hourly backup.
If you feel hourly backup is not necessary, you can change it by
updating this file
for example, to change the frequency from hourly to daily backup,
change the interval value from 3600 to 43200.
In the end, I left it with the default hourly backup so it does many
small backup hourly instead of one big backup daily.
Backup for Backups
After couple of hours of work, I managed to get a wireless backup
solution for my laptop so I won't have to worry about data loss. Plus
I can time-travel files at hourly intervals.
One concern that occurred to me was the backup sits on my local hard
drive. If the hard drive died, I would lose all my backups.
To solve that problem, I will have to go through the rabbit hole of
doing backup for backups, or backup to a remote location or cloud, or
setup a Raspberry Pi RAID.
At the moment, I'm not very concerned - I have Apple iCloud to back up
my photos, videos, notes etc and I use GitHub to host my org-files and
code. So having a backup for backups is not necessary for me for now.
This year, I'll review a really cool package I've found. It's not new, I just didn't manage to
stumble upon it over the years. So it's new to me, and might be a useful pointer to others as well.
Emmet-mode
Emmet-mode is a wonderful Emacs package which greatly
improves HTML code generation. It's based on Emmet, so you can read up on the
syntax here. I've discovered it quite recently after
reading Arjen Wiersma's
22-years-of-emacs.
The gist is that ul#nav>li.item$*4>a{Item $} will expand to:
And as a cherry on top, when used in clojure-mode, these snippets will produce hiccup.
Getting emmet-mode for free
As I wrote earlier (feels like ages ago), I like to
get functionality for free, i.e. without having to sacrifice any old key bindings or learning new
key bindings. So I've integrated the code of emmet-expand-line with C-o (ora-open-line).
Here is the relevant cond branch:
We check if emmet-mode has detected a valid snippet at point. In that case, expand it. This allows
us to chain multiple snippet engines (like yasnippet, auto-yasnippet, tiny, emmet-mode), and
even provide a hydra as a fallback for even more easy to reach commands on C-o!
Personal
Our son, Lev, was born on May 7, 2021. He is our love, happiness, our shining light in 2021.
And also the reason why I didn't have too much free time to do Emacs stuff.
Big thanks to Basil L. Contovounesios for helping me maintain Ivy!
I'm hoping to be much more active on Github in 2022.
In other personal news, I'm currently looking for a new job as a Software Engineer. If you'd like
me to work with you, please get in touch via LinkedIn or
Twitter. Also, a big thanks to everyone
donating to support my open-source work. It means a lot.
Outro
I wish everyone reading this good health in 2022! And happy hacking!
is an exquisitely powerful, free game engine. One of my favorite aspects of Godot is that its editor was created with Godot itself! And indeed, it's an excellent showcase of how the engine could be used for building a great GUI program. But as excellent as the editor is, including the code editor for the bespoke scripting language GDScript, as an Emacs user it doesn't quite fit into my workflow. Thankfully, Godot has support for the , so I'm able to use Emacs in my game development workflow. In this post I'll describe what's required to do this, and how it compares to the stock Godot editor.
Org mode, the task management and document markup system for Emacs,
includes a tool called Babel which may be used for literate
programming. In this talk I will explain literate programming, discuss
how Org mode and Babel enable it, and go over an example using the
slide deck itself. I will also cover some real-world experiences
writing literate programs in Emacs and the pros and cons of doing so.
Josh has made his
slides
available, as well as their
source
Vi is old news and the new game in town VSCode. It’s taking everyone
by storm! Stand down Sublime Text! Did anyone even use Atom? Good bye,
PyCharm, GoLand, or other JetBrains editors! You’ve all been replaced
by an editor that implements the Language Server Protocol and is
developed by Microsoft. I hear they <3 open source.
Is VSCode all that it’s cracked up to be? Should we be concerned that
this means we’ll lose good development on Emacs?
Join us as we discuss why VSCode is better! Or is it?
Back in 2015, I took a course in being an ally for local Native American communities from the
Portland Underground Grad School (PUGS). One suggested action was
learning the local language, but it proved difficult to find opportunities. When the pandemic forced
school closures, though, Lane Community College began offering classes online. I found out about
this thanks to the Kaltash Wawa blog, and this fall I signed up to take a remote Chinuk
Wawa class through Lane Community College.
What is Chinuk Wawa?
Chinuk Wawa originated as a type of creole or pidgin trade language in the Pacific Northwest,
incorporating loan words from more than a dozen indigenous and European languages. Though it has a
small lexicon, it is an interesting language with an impressive array of new phonemes for me to
learn. There are 12 different variations on “k”, for instance! As you can imagine, with this
variety, it is pretty difficult to type Chinuk Wawa using the standard Latin alphabet. I am learning
the Grand Ronde dialect, which uses an IPA-based alphabet.
Keyman for Linux
Under Linux, the easiest way to add an IPA-based input method is using the IPA (SIL) keyboard for
Keyman. This integrates nicely under GNOME 3 with Ubuntu 20.04. However, I’m a Dvorak typist,
so having to switch back to a QWERTY-style keyboard is frustrating, and was slowing me
down. Instead, I came up with this method for inputing Chinuk Wawa compatible IPA using Emacs.
Emacs Input Methods
Emacs has an easy method for switching between input methods, which you can read more about
in the Emacs manual. It comes with a number of IPA input methods, and after taking a look at all
of them, I chose ipa-x-sampa, as it seemed to have the best coverage of symbols needed by
Chinuk Wawa. However, it was missing a way to input the character “x̣”, which was
needed by one of the first words I learned, “yax̣al”!
I added the following code to my startup file to remedy this. This adds the ability to type x_.
to get x̣ and ?/ to get ?, both of which make typing Chinuk Wawa much easier for me.
(defunchinuk-wawa-quail-rules()"Add Chinuk Wawa rules to the `ipa-x-sampa` input map."(interactive)(if(string-equal"ipa-x-sampa"(quail-name))(quail-define-rules((append.t))("_."#x0323);; allows for x with dot beneath("?/""?"));; allows for ? character ))(add-hook'quail-activate-hook'chinuk-wawa-quail-rules)
As you can see, this method allows plenty of additional characters to be added to ipa-x-sampa,
so if it turns out I missed anything, I can expand on it later.
The biggest downside of this is that I can only input Chinuk Wawa easily in Emacs, but since I do
most of my work there—including writing this blog post—it doesn’t feel that bad to me. It’s
definitely better than typing QWERTY!
We look forward to any talk you want to give that is Emacs or Emacs
adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 5 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
During this discussion we’ll be focusing on collaboration software and techniques. Living in this remote world and even before collaboration has been invaluable. Having a good setup can make a huge difference.
To help get us prepped for this a previous guest Qiantan Hong will be giving us a brief workshop on crdt.el, a project he debuted last year.
Join us and learn about a new Emacs specific way to collaborate and share with us your own techniques and woes.
A talk by Eric Collins, Founder/Organizer of EmacsNYC and Senior
Engineer/Engineering Manager at Parachute Health.
You really enjoy doing everything in Emacs, editing text, managing all
of your tasks, playing games like 2048, or building presentations, but
do you read your email there? There are lots of options to choose
from, but we’ll focus on setting up mu4e for seamless indexing and
reading. This will include setting up supporting software that makes
this all work together seamlessly to process your email.
Bringing email into familiar keybindings dramatically improves your
ability to read and process messages. It’s so helpful, you might just
very well be able to achieve inbox zero.
Original Etherpad source: https://etherpad.wikimedia.org/p/emacsnyc_may_2021_notes
Ben Bass - Beorg (iOS Org Mode App)
Org interface from iOS.
Scripted in biwascheme (https://www.biwascheme.org/ )
Uses different capture templates from Org(maybe?)
Can sync via Dropbox, iCloud, WebDAV, Box
References Getting Things Done methodology(https://gettingthingsdone.com/what-is-gtd/)
Zachary Kanfer - Transient Key Maps
Maintain a certain state
After calling set-transient-map something
Any keypress that is within the map will call the function associated
It will work until a non keymap key is pressed
I live in Emacs entirely apart from using browser for googling. Having
an Emacs server running on the background makes Emacs available all
the time. So I won't worry about closing it accidental.
It is not hard to do that, just run
emacs --daemon
in command line to start the Emacs server. It will load user
configuration file as usual. Then run
emacsclient -c &
to open an Emacs GUI instance that uses the Emacs server. That's
how I have been doing for a while.
An better approach is using systemd. It is the services manager of
Linux. Whenever my Debian 11 laptop boot up, systemd would start a
bunch of services in parallel, for example, Networking manager
connects WIFI, Bluetooth connects wireless keyboard so everything
would be ready after I login. And I want Emacs to be ready as well.
I can achieve that by simply having an shell script automatically
running after login. But there are benefits of using systemd. It
has bunch of sub-commands for managing services, for example,
checking logs, status etc.
It's a nice tool to have, I can use it for example Jupyter Notebook
server.
That's why I pulled the trigger and spent 2 hours in implementing
and testing it. Here's the technical bit.
How to Implement As Systemd Service
In order to use systemd to manage Emacs server, I firstly need a
configuration file (which is called unit file). Debian Wiki
provides a short description of the syntax and parameter of unit
file.
I found an simple one in Emacs Wiki. It looks like this
service, in this case it runs /usr/bin/emacs --daemon command.
ExecStop
it tells systemd what to do when shutting down Emacs
service, in this case it runs /usr/bin/emacsclient --eval
"(kill-emacs)" command.
If you are using an Emacs built in a difference directory, you have
to change /usr/bin/emacs to wherever your Emacs is located.
Then save the configuration file as
~.config/systemd/user/emacs.service/.
After that run
systemctl enable--user emacs
so systemd would copy the configuration file into central places
and it would start Emacs service at boot time.
To run Emacs service right now, use
systemctl start --user emacs
This is what I see in my console
emacs.service - Emacs text editor
Loaded: loaded (/home/yitang/.config/systemd/user/emacs.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-06-14 09:12:26 BST; 24h ago
Docs: info:emacs
man:emacs(1)
https://gnu.org/software/emacs/
Main PID: 5222 (emacs)
Tasks: 5 (limit: 19027)
Memory: 154.7M
CPU: 3min 25.049s
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/emacs.service
├─ 5222 /usr/bin/emacs --daemon
└─16086 /usr/bin/aspell -a -m -d en_GB -p /home/yitang/git/.emacs.d/local/ispell-dict --encoding=utf-8
Jun 14 09:11:57 7270 emacs[5222]: No event to add
Jun 14 09:11:57 7270 emacs[5222]: Package dash-functional is obsolete; use dash 2.18.0 instead
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/org-mode.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/load_config.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Starting Emacs daemon.
Jun 14 09:12:26 7270 systemd[4589]: Started Emacs text editor.
Enhance User Experience
For far, I have the following two tweaks to make the usage of systemd
more pleasant.
sudo Privilege
The Emacs server is started using my own account, so it doesn't have
the sudo privilege. In order to edit files that requires sudo
permission, simple open the file in Emacs, or in command line with
emascclient -c FILENAME
then type M-x sudo inside Emacs, type the sudo password. If the
password is correct, I can edit and save the file as sudo user.
Environment Variables
The customised shell configuration in .bashrc are loaded when
opening an interactive shell session. So the Emacs server managed by
systemd would not have the environment variables, alias, functions or
whatever defined in .bashrc.
This stackoverflow post provides the rationale and how to tweak
the unit file so systemd would load .bashrc.
This problem can solved a lot easier on the Emacs side, by using
exec-path-from-shell package. It will ensure the environment
variables inside Emacs are the same as in the user's interactive
shell.
Simply put the following in your .emacs would do the trick.
(exec-path-from-shell-initialize)
Start Emacs Server Before Login?
The systemd services under my account would only start after I
login. Because I have tons of Emacs configuration, I still have to
wait few seconds before Emacs server is ready. So it would be awesome
to have the Emacs server starting to load before I login.
This doesn't seems to be simple to implement, because technically,
it would require the Emacs server to be defined on system level,
but it will load files in my personal home drive without me being
logged in. It might be still okay since I'm the sole user of my
laptop, but I have to tweak the permissions and would probably end
up with non-secure permission setting.
Fold and Focus helps you navigate Org, Markdown, and Emacs Lisp files with focused attention: one thing at a time. You can navigate by heading (or, in Elisp, by defun), optionally narrowing to it as you go.
With Org B64 you can easily Base64-encode files into an org file. This makes possible the easy embedding as plain text of binaries such as images, audio, and compressed files for later extraction or sharing with friends.
Org Reflect allows you to mirror the contents of source files into a target org file. When the contents are code, they are automatically wrapped in an Org Src block (this behavior can be disabled, if you want). Before being displayed, filters can be applied, such as restriction of line ranges and the processing of the text through Emacs Lisp or Shell script filters of your choice (write any functions you want). Therefore, it embeds data from other files into your Org document. This is convenient, and can be instantly updated.
After Emacs Lisp, I tackled the phones-to-words challenge in Bash. Here I describe my different approaches and how these solutions compare to each other. All of them were terribly slow to run, but the code itself is quite concise, with a bit more than half the number of lines of code of my elisp version.
I decided to tackle an old programming challenge where the digits in a list of phone numbers are converted to letters according to rules and a given dictionary file. The results of the original challenge suggested that Lisp would be a potentially superior alternative to Java, since Lisper participants were able to produce solutions in, on average, fewer lines of code and less time than Java programmers. Here I tackle the problem using Emacs Lisp and share my solution.
Some time after having first seen it, I sit down and write some solutions to Ben Hoyt's word-counting exercise. Solutions are in Unix Shell and Emacs Lisp. Here I introduce the problem and prepare input files.
We look forward to any talk you want to give that is Emacs or Emacs
adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 5 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
It’s been one year since we’ve gone online and it’s been seven years
since we started this meetup for you lambda enthusiasts.
In this discussion session, I think it’d be helpful to have a
retrospective on what has been going well and what could be
better. That said, we’re thinking also about having some lightning
discussions about various topics. Ideas are welcome, if not we have
several to go from.
It seems to be widely accepted that creating a powerful, useful Emacs setup "by hand" is just too much trouble, and you should choose a "distro" like Doom Emacs. But is it really all so bad? If you go the route of "hand-made", will you suffer through endless nights of fixing your setup? The answer is: probably not, but read on for more details!
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
Managing Email with mu4e and Other Software
A talk by Eric Collins, Founder/Organizer of EmacsNYC and Senior
Engineer/Engineering Manager at Parachute Health.
You really enjoy doing everything in Emacs, editing text, managing all
of your tasks, playing games like 2048, or building presentations, but
do you read your email there? There are lots of options to choose
from, but we’ll focus on setting up mu4e for seamless indexing and
reading. This will include setting up supporting software that makes
this all work together seamlessly to process your email.
Bringing email into familiar keybindings dramatically improves your
ability to read and process messages. It’s so helpful, you might just
very well be able to achieve inbox zero.
I think one of the things that really separates us from the high primates is
that we're tool builders. I read a study that measured the efficiency of
locomotion for various species on the planet. The condor used the least energy
to move a kilometer. Humans came in with a rather unimpressive showing about a
third of the way down the list. It was not too proud a showing for the crown
of creation. So, that didn't look so good.
But then, somebody at Scientific American had the insight to test the
efficiency of locomotion for a man on a bicycle. And, a man on a bicycle, a
human on a bicycle, blew the condor away, completely off the top of the
charts.
And that's what a computer is to me. What a computer is to me is it's the most
remarkable tool that we've ever come up with.
No one knows
when or how we, the human species, started talking to each other. It is likely a
natural progression from gesturing, but we can only speculate about it.
Language allowed us to break out of our brains and reveal the inner workings of
our consciousness to others.
Language is the vessel that carried us from the stone age through the
agricultural revolution, the development of written language, the scientific and
industrial revolutions, and now, the digital age.
Writing allowed us to offload memories to the physical world—outside of our
brains. Through our collective and external memories, each generation has a head
start on the previous one. Little by little, standing on the shoulders of taller
and taller giants, we accumulate knowledge about ourselves and everything around
us.
We've been for long using tools to help us think: notebooks help us calculate
formulas, reason geometrically and preserve our ideas. With computers, our
thinking is now occurring outside of our brains.
Computers are extensions of our minds in that they allow us to store, process,
and retrieve information from them. With the advent of the internet we now have
immediate access to not only almost all of the information ever produced by
humankind but also to reproducible thinking encoded into these machines:
algorithms.
Our brain is still a much more impressive device than any of today's computers. Computers learn mostly by finding patterns in massive quantities of examples given by us. Teaching a young kid about cars—how to recognize one, what they are, what their purpose is, and how they're related to other things—requires little supervision. Noam Chomsky talks about it in this interview.
Each of these processes—storing, processing and retrieving information—have
concrete effects on the physical world: if I'm in Munich, saying "show route to
Hamburg" to my phone will immediately show me the distance, ETAs and paths for
different types of transport to reach my destination. Not only do I now suddenly
know how to navigate across the country to reach another city, I'm also able to
follow through the exact path via GPS—a sixth sense giving me perfect
geolocation!
These things that we created—computers, and the internet—are literally
rewiring our brains,
right now, shaping how we think, and engage in social relationships, changing
not only our individual selves but the societies we live in.
They started as mechanical machines that filled entire laboratories, turned into
beige boxes in our homes and places of work, and are now sleek slabs of plastic,
metal and glass in everyone's pockets. Step by step they get closer to our
bodies, their interfaces more intuitive and natural.
The way we communicate with them is changing: before, we could only interact
with them by speaking their language. We have now taught them ours. The torch of
progress blazes on: it's a matter of time until they're connected directly
with our brains—which is equally terrifying and awe-inspiring.
Brain-computer interfaces present a monumental scientific and engineering
challenge, and brain-to-brain, a whole other category of difficulty.
First, we have no idea how information is encoded in the brain. That needs to be
understood. Second, even assuming we're able to take a perfect snapshot of a
piece of information in someone's brain—for example, how a particular movie
scene makes them feel—we still need to be able to encode it in a way that
includes the full context of their subjective experiences. Maybe the scene
evokes unique memories of their childhood or is somehow entangled with the smell
of a particular cinema's leather seats. Third, we need to figure out how to
safely write this perfect snapshot into someone else's brain in a way that can
be perceived identically.
Which is to say, it's a difficult problem. But a worthwhile one: imagine having
the capability to suddenly become aware of answers for questions you just
thought about. To expertly control
truly integrated prosthetics giving you
superhuman abilities. To give movement to the paralized, sound to the deaf, and
sight to the blind.
What would be the impacts on society if we were able to communicate an order of
magnitude more effectively? What if everyone was equipped with the same
undisputed basic knowledge of history and science?
There are internal thoughts that we can attempt to describe with a thousand
words, but ultimately fail to capture in a way that's precise, much less
comprehensible by someone else. Words and sentences are an incomplete
representation of our internal thoughts. In the same way that 3D objects cast 2D
shadows (and 4D, 3D)
communicating through language doesn't carry all of our cultural and
developmental context—transmitting all of that along with every phrase would be
impractical. Language is in this sense, lossily compressed thought.
Inert strings of words of ink and paper take a life of their own inside our
heads. It's why the exact same information can be interpreted completely
differently by different people.
Before language, fire and cooking technology allowed us to reallocate energy
usage from the digestive system to the brain by outsourcing digestion to outside
of our bodies, making macronutrients more efficiently absorbable. Almost all of
a cooked meal is metabolized by the body, whereas raw foods yield less than half
of their nutrients.
Cooking is an extension of our digestive system, and enabled us to develop
large, calorie-hungry brains. It also gave us time to think: our primate cousins
spend half of their days chewing raw food to consume enough calories to stay
alive.
Brains can be seen as survival machines, locked inside dark skulls, constantly
building a model of the outside world by predicting and learning through senses
and memory. The biological human brain evolved to have the necessary
sophistication to not only expertly navigate and understand the brute physical
reality but also to construct social reality. Democracy, religion, money: all
made up by us, for us.
We remember the past so that we can predict the future, and by doing so, we
thrive.
We create technology, which functions as a non-biological extra layer to our
brains and bodies, augmenting, complementing, and sometimes replacing our
natural capabilities.
The wheel… is an extension of the foot.
The book… is an extension of the eye…
Clothing, an extension of the skin…
Electric circuitry, an extension of the central nervous system.
To be human is to have the ability to change the world around us. The shift from
hunting and gathering to farming allowed us to spend less energy to acquire food
while giving us a predictable calorie supply.
The resulting food surplus made it possible for populations to settle down and
grow quickly while supporting people not being directly involved in the
production of food—before agriculture that was everyone's job. For one, it
allowed some to specialize and focus on developing better farming tools and more
resistant crops, starting a vicious cycle of improvement and consumption that
continues until today.
The transition from active foraging to a more sedentary lifestyle resulted in
worse health for the general population. The average farmer worked harder than
the average forager and got a worse diet in return. Our teeth, bones and joints
became more fragile, and we became afflicted by novel diseases coming from newly
domesticated animals, carriers of pathogens that incubated in our new densely
populated cities.
Owning land suddenly became really important. Agriculture and the concept of
private property reinforced each other and grew together, allowing us to create
value and secure the fruits of our labor. It also created the circumstances for
slavery to arise, and wars to be waged.
The groups of people growing the first crops could not have anticipated all of
the collateral effects of their breakthrough. They just wanted more food.
If the past has taught us anything is that we have to be mindful of the
consequences of our progress. In an increasingly connected world, change is
often nonlinear
and unpredictable. Cars didn't just replace horses—they forever changed the
entire outlook of every city. Did Tim Berners-Lee anticipate his invention
adding to forces pulling whole countries apart?
Our progress will continue to bring us previously unimaginable challenges.
Against an unknowable future, it doesn't hurt to keep improving our capabilities
to adapt and, more difficultly, to cooperate—especially at scale.
Computers are getting pretty good at driving cars—even in the most difficult
situations—and can already instantly diagnose some diseases better than human
doctors. Technology has a way to
reveal
the potential of our environment, and ourselves. We have to be careful not to
look at what surrounds us as mere raw materials to be consumed for the purposes
we conceive—sooner or later we'll start calling humans resources too...
It serves us well to leverage technology to give us time. Time to create and
enjoy art, follow the trail of our curiosities and passions, be fully present
with loved ones, or even just appreciate the freedom to idle and ponder about
the inconsequential—the stuff that seems to make us, us.
We are born with incomplete brains that get imbued with language and the
accumulated collective knowledge of our previous generation. Knowledge,
roughly
defined as a justified, true belief, doesn't fit the bill of much of the waves
of man-made information hitting the shores of our eyes and ears these days.
Acquiring it requires many things, and passively consuming content curated by
profit-maximizing algorithms is not one of them.
We attempt to transfer our gathered knowledge to machines and we specify the
rules for their learning, and by doing that we're inherently encoding our own
biases and limitations in algorithms that will be making life-altering choices.
Should your out-of-control self-driving car
automatically swerve to avoid
running over kids on the street, and by doing so put your own life at
considerable risk? Should you be able to opt-out of this behavior with a
checkbox?
We need to be careful about what we teach machines, and prevent them from making
the same mistakes we do, because they will do them orders of magnitude more
efficiently and at scale. Human judgment is both fallible and (still)
indispensable, especially when the stakes are higher.
Our learning machines already exhibit emergent behaviors that go beyond human understanding and could be interpreted as creativity, like AlphaGo's 37th move on the second match against Lee Sedol in 2016.
To be human is to have the ability to change oneself. Through open source and
hardware hacking, people with type 1 diabetes—who need to continuously measure
and manipulate their glucose and insulin levels to stay alive—took it upon
themselves to build an
artificial pancreas
and hook it up to their own bodies. The technology they created not only removed
an enormous cognitive burden from their lives but also decreased the likelihood
of physical complications and increased their lifespans.
What wouldn't you give to free up a large part of your brain processing
power and at the same time considerably improve all of your health
indicators?
Empowered by knowledge and technology, they didn't have to wait for the world
around them to change—they went ahead and changed it themselves. And in the
process, they changed their lives.
Arunachalam Muruganantham, who grew up in poverty and dropped out of school at
14 to support his single mother, also didn't wait for the world around him to
change. Going against conservative rural India—who treats sex education as
taboo—he provided underpriviledged women with
affordable sanitary pads by
creating a set of pad-producing machines. Poor menstrual hygiene cause women to
miss school, risk infections, and die from cervical cancer. The industry around
his invention provides women with income, gives them dignity, and saves their
lives.
Technology and the tools we create drastically accelerate our progress. Matt
Taylor compellingly puts it in perspective in
"Humanity 2.0". In it he presents
the chart below, which seems to show life's steady and even progress from
unicellular organisms to us, human beings, building
artificial suns, taking pictures of
black holes,
and unlocking the
mysteries of life itself.
The scale of this chart could be initially misleading: the visual distance
between the birth of life and the first eukaryotic cells—2 billion years—is
represented identically as the distance between the industrial revolution and
the personal computer—200 years.
The chart below tells the same story on a scale more easily digestible by us.
The mostly horizontal line depicts the slow process of biological evolution,
which eventually—out of only randomness and constraints—brought our neomammalian
brains into existence, kick-starting the journey of fire and language towards
modern civilization. It's been a long ride, and in the relative time scale of
biology our evolution through technology is happening fast and only seems to be
accelerating.
Each technological advance builds upon the last, creating a positive feedback
loop of progress.
We are inevitably shaped by what we create, and fundamentally driven by our
deeply human essence: the anticipation of discovery, the satisfaction of
attainment, and the joy in relationships we cultivate along the way.
The technology we create will survive us, and its impact will be unevenly
felt—the fruits of progress aren't unconditionally good.
We've come a long way
towards improving our lives, and we can still go so much further. There are so
many problems to solve.
In these unprecedented times of tremendous individual potential, it's good to
keep our values in check and constantly revisit the question: are we building
the right things?
History is a metaphorical pathway, and just like physical ones, it's built
purposefully by us, based on the topography and constraints of the environment.
Unlike physical ones, it sometimes takes us by surprise.
The future is still not determined, and "the best way to predict it is to invent
it". Informed by our knowledge and empowered by our technology, it is up to us
to lay the bricks.
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
March is mail month! It’s not totally clear what’s going to be
happening this month, but it will have something to do with email in
Emacs!
I started using
Emacs
almost exactly four years ago, after almost a decade of
Vim. I made the switch cold turkey. I
vividly remember being extremely frustrated by unbearable slowness while
editing a Clojure file at work. With no sane way of debugging it, just moving
the cursor up and down would result in so much lag that I had to step away from
the computer to breathe for a while. When I came back I quit Vim (I knew how at
that point), opened Emacs, and started building my configuration.
More recently, I've been very put off by the performance and stability (or lack
thereof) of building large scale software via Tramp. This has been sufficient to
have me looking out again. On a whim, I installed VSCode for the first time and
tried its "remote development" capabilities and holy smokes are they good.
Getting up and running was trivial and the performance was great. Saving files
was snappy and LSP worked out of the box. What a different experience from my
carefully-put-together, half-working, slow Emacs setup.
💭
My common denominator for rage-quitting software seems to be consistent: bad performance.
There has recently been more discussion than usual regarding "modernizing"
Emacs, by making keybindings more consistent with other applications and using
more attractive color schemes and visuals, with the end goal of attracting more
users and by extension more contributors.
In my view improving these aspects of user experience wouldn't hurt. The way I
see it, though, is that for Emacs to attract more users it needs to be
objectively better than the alternatives. And the way to do it is for Emacs to
become even more like Emacs.
💭
I see Emacs as being fundamentally two things: a programmable runtime, and a beacon for free software. I'm talking more about the former.
It needs to be a more robust, more efficient, and more integrated platform
with a more powerful extension language, to empower its users to build their
own environment.
Getting LSP integrated pervasively in Emacs in a way that it reliably just
works and performs well out of the box, would go a long way towards making
Emacs more attractive not just to new users, but to existing ones too. Imagine
an experience similar to VSCode's:
Open Emacs for the first time
Open a source code file
Emacs asks if you want it to configure itself for the programming language
of that source file
Saying "yes" automatically sets up Emacs to have a modern programming
environment for that programming language with smart code completion,
navigation, and refactoring, rich hover information, highlighting, automatic
formatting, snippets, etc. Maybe even open a side window with a buffer with
a short "getting started" tutorial showing the available keybindings.
Given enough users, opinionated community-built Emacs "distributions" like
Spacemacs, Doom, and Prelude will do the job of making it easier for newcomers
to get started with typical contemporary tasks: building software with popular
programming languages, writing documents, managing machines, etc.
Building and maintaining these "distributions" also becomes much easier given a
more robust, more efficient, and more integrated platform with a more powerful
extension language.
Having a
wizard
showing up in new Emacs installations might be a great low-hanging fruit way of
making Emacs more accessible. Assuming buy-in from core maintainers, the wizard
could even directly reference popular Emacs "distributions" like the ones
mentioned above, so that new users can kickstart their lives in Emacs.
The way to attract contributors can also be stated simply: directly improve
the contribution process.
💭
Easier said than done.
Many have created their Emacs
wishlists. This is mine.
For the past one and a half years,
Andrea Corallo, a compiler engineer, has been
working on adding native compilation
capabilities to the Emacs Lisp interpreter. His work is available in a branch in
the official Emacs repository. Folks have been trying it out, and according to
the reports I'm hearing, the results are staggeringly positive. I am very
excited about Andrea's work, which seems to bring enough improvement to the
"code execution speed" side of the equation to make it a non-issue for now.
Andrea's work will also allow for more of Emacs to be implemented in Emacs Lisp
itself (instead of C), which is what most contributors are used to. This is a
great win for maintainability and extensibility: incrementally having more and
more of Emacs be implemented in the language with which it's extended.
The garbage collector is still in much need of improvement. Many resort to hacks
to ameliorate frequent and sometimes long pauses that seem to be unavoidable
while working on large git repositories, fast-scrolling font-locked Eshell
buffers, displaying dynamically updating child frames, navigating big Org files,
and many other tasks.
💡️
Also, try this out:(setq garbage-collection-messages t)
2. Improved display efficiency and rendering engine
The display implementation in Emacs core is... less than ideal.
GNU Emacs is an old-school C program emulating a 1980s Symbolics Lisp Machine
emulating an old-fashioned Motif-style Xt toolkit emulating a 1970s text
terminal emulating a 1960s teletype. Compiling Emacs is a challenge. Adding
modern rendering features to the redisplay engine is a miracle.
It would be great if Emacs did like Neovim and decoupled the editor runtime from
the display engine. This would make it possible for the community to build
powerful GUIs without having to change
Emacs core,
possibly
using technologynot fully sanctioned
by core maintainers.
Take a look at the screenshots of these Neovim GUIs:
They're powerful, look great, perform well, and more importantly, are based on
industry standard, cross-platform graphics APIs (Vulkan and WebGL respectively)
that get lots of personpower contributions from companies and individuals alike.
The Onivim and
Xi text editors could also be sources
of inspiration:
Separating the core runtime from the user interface
Ropes for faster
incremental changes and parallelization of text operations
Emacs does not support parallel code execution via multi-core processing. Code
execution happening on any buffer will freeze the whole program, preventing not
only user interaction but other cooperative threads of execution from making
progress as well.
Adding parallelism to Emacs in a way that automatically makes existing code run
in parallel is about as close to impossible as it can get. What would be more
feasible is including new primitives for parallel execution that new code could
leverage, to build more powerful extensions to Emacs.
Emacs-ng is a recent effort that
implements just that: an additive layer over Emacs that brings not only
parallelism, but also asynchronous I/O capabilities via an embedded
Deno runtime, and GPU-based rendering via
WebRender. I am super excited about the
very fast progress from the folks working on emacs-ng, and I think the project
holds great promise for the future of Emacs itself.
There also seems to be advances in the area of immutable data structures that
could be leveraged by the Emacs core, as seen in
"Persistence for the Masses: RRB-Vectors in a Systems Language".
Persistent data structures would make building thread-safe parallel code much
easier.
4. Enhanced stability
It is very easy to either freeze Emacs or cause it to run very slowly. Multiple
times a day I have to hit C-g incessantly to bring it back from being frozen.
When that fails, I am sometimes able to get it back with pkill -SIGUSR2 Emacs.
At least once per week I have to pkill -9 Emacs because it turned completely
unresponsive. I suspect doing more work outside of the main thread might help
with this?
There are many hacks to ameliorate issues caused by long lines, but they're
still fundamentally there. Advancements in the "display efficiency and rendering
engine" effort would help with this too.
I recently tried a package that displays pretty icons on completion prompts, and
noticed that it made scrolling through candidates really slow. Profiling showed
that the package was creating thousands of
timers,
which were somehow causing the issue. There are lots of cases like this, where
folks attempt to create something nice, but inevitably have to resort to
hacks to either
achieve acceptable performance, or to be able to implement the thing at all.
Having a more robust/efficient/integrated core with a more powerful extension
language would help here.
Impressive efforts from folks like Lars Ingebrigtsen who routinely comes in and
obliterates 10% of all reported Emacs bugs
also have a sizable impact. We users should follow the lead and do a better job
not only creating good bug reports but also dipping in our toes and helping out:
fixing bugs, writing tests, and documentation.
Yuan Fu recently wrote a
nice guide for
contributing to Emacs.
5. Emacs Lisp improvements
Emacs Lisp is a much better language than Vimscript. Unfortunately, that's not
saying much. It's not a particularly good Lisp and has lots of room for
improvement.
For example, if you want to use a map, you have three choices: you can use
alists, plists or hash maps. There are no namespaces in Emacs Lisp, so for
each of the three data types you get a bunch of functions with weird names.
For alists get is assoc and set is add-to-list, for hash maps get is
gethash and set is puthash, for plists get is plist-get and set is
plist-put. For each of those types it is easy to find basic use cases that
are not covered by the standard library, plus it is easy to run into
performance pitfalls, so you end up rewriting everything several times to get
something working. The experience is the same across the board, when working
with files, working with strings, running external processes etc. There are
3rd party libraries for all those things now because using the builtins is so
painful.
Emacs Lisp APIs evolved incrementally while maintaining backwards compatibility
over a long period of time. This is good: code written more than a decade ago
still runs. These increments came about via decentralized volunteer efforts, and
it shows: there are many inconsistencies and conflicts between and within
libraries, which feel like having evolved without an overarching design.
Programmers used to languages that did go through
careful, deliberate design brought some
of it to Emacs Lisp:
It would be great to have more and more of these influencing and being
incorporated to the Emacs Lisp standard library and made to be very performant.
map.el and seq.el seem to already be in thanks to Nicolas Petton!
Assuming a multi-core future for Emacs, it will also be critical to have good
ergonomics for writing concurrent code. It should be easy to do the right thing
(writing thread-safe code), and hard to do the wrong thing. I believe Clojure
can also be a
source of inspiration.
How easy it is to just say these things! Easy to do the right thing, hard to do the wrong thing!
Other than that, a great module system, possibly one that allows different
versions of libraries to coexist, would also be a great addition. Andrea Corallo
seems to be trying out some
new ideas in this space
(discussion).
6. Dealing with non-text
It is currently possible to browse the web in Emacs in an embedded
fully-featured WebKit widget. We need to go further—I want to have the same
experience of the likes of Nyxt and
vimperator, integrated to Emacs:
switch between tabs with fuzzy completion (ivy, helm, etc.)
navigate via link hinting
Isearch web pages
easily copy text content from web pages, paste it elsewhere
create macros to repeat actions on web pages
I can kinda do some of these things right now with the existing WebKit widget
along with some clever hacks. There are
other more
adventurous hacks
which work around Emacs to create a full graphical interface. It would be
great if this type of functionality was deeply integrated to Emacs. It's a
difficult thing to do because of the existing display engine implementation and
Emacs Lisp limitations.
I believe doing like Neovim (and others) and separating the
core from
display would help here. But, it would
likely bring its
own problems.
There are unfortunately no silver bullets.
Less importantly but still desirable: email. Even though I write most of my
email messages in Emacs, I read them mostly outside of it. I prefer to exchange
plain text email, but sometimes I receive HTML email. When I do, I'd prefer to
visualize it as the author intended. This is currently technically possible, but
suffers from the same challenges as web browsing.
7. Improved contribution and development process
Contributing to Emacs core and packages in the official repository requires
assigning copyright to the FSF. Employed software developers need to get
paperwork signed by their employers' legal departments, a process that takes
many days. Copyright assignment is likely not going away—can it be made more
convenient?
It would be great to move to a forge style of contribution. It is honestly
incredible to me how people keep track of patches flying around in email
threads. Unless something like sourcehut or Patchwork is being used there's no
automated CI making sure individual patches and overall contributions are in a
good state. Hopefully the Emacs
GitLab instance starts being more actively
used and becomes the official way to contribute.
Copyright assignment and mailing-list driven development are definitely
off-putting to folks who just want to contribute, and aren't looking forward to
having to sign paperwork or learn a special way to contribute to every project
they work with. The GitHub generation of open source developers are used to a
standardized, powerful and convenient platform—anything other than that just
feels not worth it.
Emacs will likely always have a niche of users, but it could grow to not have
developers. Having large parts of the core implementation be in C makes it not
very approachable to anyone outside the handful of contributors who do feel
confident to change it.
It probably makes sense to continuously look for functionality implemented in
the C core that could be replaced with focused libraries, like Neovim did by
replacing almost all of their hacky, platform-specific code with
libuv.
Also, would it make sense to start an Emacs
Open Collective to fund work on Emacs?
Last "small" thing
Improve Tramp performance to match the experience of using terminal Emacs via
SSH, or VSCode's
Remote Development.
* * *
Talking is easy. Accomplishing any of these would require lots of work. It may
not seem like it but text editors are a
hardproblem. And
people, an even harder one.
I wonder if Emacs will stick around long enough and grow the necessary
functionality for us to someday run M-x neuralink-mode and evaluate Lisp in
the brain?
With corporations like Microsoft, Oracle, and Google truly reinventing
themselves to adapt to an open source world, and typical open source projects
moving towards—oftentimes centralized—governance models, the
Cathedral-Bazaar
dichotomy feels increasingly less relevant.
While being an entertaining piece of history with useful takeaways, its most
important achievement was arguably helping create a sense of identity for hacker
culture via the revolutionary Open Source movement, and promoting the value of
the Internet for software development.
In the Cathedral-Bazaar continuum, contemporary projects like Kubernetes,
Chromium, and VSCode are fusion reactors.
They have the backing of heavily invested companies with virtually infinite
capital, who are able to staff highly competent teams that not only work
full-time on these projects, but also have enough personpower to maximally
leverage the benefits brought by a gigantic user base.
Like with fusion power, they seem to be able to leverage a high amount of energy
to generate even more.
Sometimes, their user base includes other organizations with endless resources: by means of its success, VSCode is getting sizable contributions from Facebook, for example.
In contrast, the vast majority of open source projects depend almost exclusively
on decentralized volunteer efforts from people sacrificing time out of their
busy schedules and lives to move things forward.
And yet, projects following this style of development can end up becoming
backbones of modern computing:
The OpenSSL project has been around since 1998. Since the project is open
source, it is an informal group comprised primarily of about a dozen members
throughout the world, most of whom have day jobs, and some of whom work on a
volunteer basis. Being open source, the OpenSSL project's code has always been
public facing. Any person could download it and modify it or implement it in
their own software.
[...]
The fascinating, mind-boggling fact here is that you have this critical piece
of network infrastructure that really runs a large part of the internet, and
there's basically one guy working on it full time.
Heartbleed is a symptom of an ever-existing problem: corporations profit
massively by leveraging typically under-resourced open source projects while not
giving back proportionally, or (most commonly) at all; either with money or
people.
After Heartbleed the Core Infrastructure Initiative was created to support software essential to the "functioning of the Internet and other major information systems".
Less critical software like Emacs also follows this decentralized development
style, and similarly, lacks resources.
So if Emacs wants to compete with these tools then it has to have seamless,
context aware code completion and refactoring support, and GNU tools has to
provide Emacs the necessary information to implement these features.
I agree. But to have that, the only way is to have motivated volunteers step
forward and work on these features. Otherwise we will never have them.
Right now, no one is working on that, though everyone is talking. [T]he same
as with weather.
One way to incentivize contributions is by funding developers. Some (most?) open
source contributors would gladly take income to fund their work.
Long-term my big dream has always been to accumulate enough backing to be able
to work full-time on open-source projects, but whether I'll achieve this dream
or not is entirely up to you.
While others feel that accepting funding would degrade their
intrinsic motivation
to contribute.
I can't speak for all FLOSS developers, but I can speak for myself: I don't
want monetary rewarding from users. Mainly for the reason, that I don't want
to change the relationship with my users. Currently it is mostly a team
attitude, we're working together to solve the problem. And there is also no
legal obligation for me to work on something I don't like.
If I accepted contributions I think many users would get a "but I paid for
that, so do what I want" attitude. I definitely don't want that. I do FLOSS in
my free time to do something that matters, and for my personal fulfillment,
not for money.
It will also get harder to do the right thing (in contrary to doing what the
users want) since the users can stop the payments.
Which is fair: with compensation comes responsibility, timelines, expectations,
and things can get complicated. From the point of view of the project, there
doesn't seem to be a conflict: capture funding to enable those who need (or
want) it, while still empowering those who don't, to contribute on their own
terms.
Not everyone is in a position to spend unpaid time on open source, especially
consistently. "Free" time
isn't free—it
costs life. By funding work, a project might get to see contributions from
talented folks passionate about it who wouldn't be able to volunteer their time.
And sometimes, the reason why your PR isn't getting immediate attention is that the maintainer is busy literally fighting a revolution.
For Emacs specifically, one problem is that there's no clear way of funding
"Emacs". Sending money to the FSF doesn't guarantee that it will fund Emacs
development. Even if there was a way of "funding Emacs", the Emacs
community—like many things in life—seems to be roughly divided in two sides:
those who prioritize
freedom, and those who
prioritize progress. So one would potentially want to fund one side or the
other, depending on their values.
In the excellent "Working in Public:
The Making and Maintenance of Open Source Software", Nadia Eghbal calls
attention to a shift in how individuals support open source. Similarly to
platforms like Twitch, more and more people are funding creators directly
instead of projects, as a way of "incentivizing the ongoing creation of creative
work" from developers who produce things that are in their interests.
Maintainers of popular Emacs packages, for example, have their own
separate streams of patronage,
which receive varying levels of support depending on their popularity and the
value added by what they create.
For efforts that involve multiple people, like maintaining and evolving Emacs
itself, services like Open Collective could be of
great assistance by helping on three fronts:
providing a legal banking entity
recurrently collecting funds from individuals and companies
distributing funds to contributors
Funds can be transparently collected, and dispersed for specific contract work,
infrastructure costs, and even developer salaries. Take for example the
Babel project, which draws in enough
recurrent income to finance multiple contractors and a full-time developer
earning a San Francisco salary. An Emacs Open Collective could not only be an
answer to "how can I fund Emacs development?" but also a way to financially
support developers working on it.
The Clojure community seems to be doing an excellent job at not only funding efforts that are making the whole Clojure ecosystem better but also at surveying and responding accordingly to user feedback.
GitHub Sponsors is another great example of
developer empowerment. With it, not only does GitHub equip people to:
be more productive by providing great code hosting, bug tracking, wiki, code
reviewing and merging, project management, continuous integration,
documentation, artifact hosting, etc.
have a broader impact by giving projects more visibility and standardized
workflows that are familiar to others already on the platform
It also makes it possible for its
40 million users to frictionlessly fund work on
open source, and for a large number of maintainers to be conveniently
compensated for their labor. I just started sponsoring someone with literally
two clicks!
The power of platforms can't be understated.
Whether you like GitHub or not, it's undeniable that it has, and continues to
revolutionize Open Source, simply by providing a significantly better and
unified experience for all aspects of building software.
When there are people making over 100k/year on GitHub Sponsors, you better have a great reason to not try to take advantage of it.
In the case of Emacs, the reason is freedom.
I wouldn't mind if Emacs development moved to GitHub, but
I don't think it's ever going to happen.
Maybe for good reason: GitHub is backed by a for-profit corporation and is far
from perfect, both in
moral
and
technical
terms. It might be a great tool today, but being a proprietary platform, its
users are at their complete mercy.
I should point out that from my perspective GitHub has been for the most part a force for good.
It would be great if main development at least moved from a mailing-list-driven
process to a modern forge style of contribution. It
seems that it might, but whether or not it
will is still unclear.
In the same way that corporations extract value out of open source, open source
projects should as much as possible leverage "energy" generated by corporations.
In this new open source world, companies have their workforce contributing
millions of person-hours to projects that benefit everyone. LLVM equips people
to build programming languages. LSP gives people potent software development
capabilities. Rails empowers people to build powerful web applications.
More than 3,000 people have committed man-decades, maybe even man-centuries,
of work for free. Buying all that effort at market rates would have been
hundreds of millions of dollars. Who would have been able to afford funding
that?
That's a monumental achievement of humanity! Thousands, collaborating for a
decade, to produce an astoundingly accomplished framework and ecosystem
available to anyone at the cost of zero. Take a second to ponder the magnitude
of that success. Not just for Rails, of course, but for many other, and
larger, open source projects out there with an even longer lineage and
success.
In many cases, the
ideology ingrained in Emacs prevents it from leveraging value generated by
efforts not totally compatible with the goals of the Free Software movement.
Still, there are many non-conflicting opportunities for improvement.
Maybe Emacs doesn't need to be a fusion reactor. I only hope it continues to
generate energy for many years to come.
We look forward to any talk you want to give that is Emacs or Emacs
adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 5 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
In 2018 Bryan Cantrill gave a brilliant talk where he shared his recent
experiences with the Rust programming language. More profoundly, he explored a
facet of software that is oftentimes overlooked: the values of the software we
use. To paraphrase him slightly:
Values are defined as expressions of relative importance. Two things that
we're comparing could both be good attributes. The real question is, when you
have to make a choice between two of them, what do you choose? That choice
that you make, reflects your core values.
He goes ahead to contrast the core values of some programming languages with the
core values we demand from systems software, like operating system kernels, file
systems, microprocessors, and so on. It is a really good talk and you should
watch it.
It is important to think about values because they are core to the decisions
that we make.
Unlike systems software, the values demanded from text editors or IDEs vary
greatly depending on who you ask. These are much more personal tools and make
room for a diverse set of desires.
The following listing enumerates values that could be attributed to development
tools.
Value
Commentary
Approachability
Ease of getting started with for typical tasks, and contribution friendliness
Doing one thing well
Unix philosophy, fitting into an ecosystem
Editing efficiency
Fewer interactions, mnemonics, composable keystrokes, etc.
Extensibility
The degree to which behavior and appearance can be changed
Cohesive core and concerted third-party functionality
Introspectability
Capable of being understood and inspected ad-hoc
Keyboard centrism
Focus on keyboard interactions
Maintainability
The degree to which it can be modified without introducing faults
Progressiveness
A measure of eagerness to make progress and leverage modern technology
Stability
Things that worked before continue to work the same way
Text centrism
Text as a universal interface
Velocity
Short and focused release cycles, aligned personpower, leveraging the community effectively
💭
Before we go any further, I'd like to point that out if you care about any of the topics discussed ahead you will likely strongly disagree with something or the other.
That's fine! We probably just have different values.
In my view, Emacs has the following core values:
Emacs
Extensibility
Freedom
Introspectability
Keyboard centrism
Stability
Text centrism
We can feel the clasp of stability in the following—rather poetic—exchange in
the Emacs development mailing list, which also provides useful historical
perspectives.
Emacs is older than the operating systems people use today. (It is almost as
old as the first Unix, which barely resembled the Unix of later decades.) It
is much older than Linux, the kernel.
The oldest design elements were not designed for the uses we make of them
today. And since we wrote those, people have developed other areas of software
which don't fit Emacs very well. So there are good reasons to redesign some of
them.
However, people actually use Emacs, so a greatly incompatible change in Emacs
is as unthinkable as a greatly incompatible change in the New York City
subway.
We have to build new lines through the maze of underground pipes and cables.
The following exchange reifies freedom and stability while demonstrating a
disinclination to progressiveness. Which is neither good nor bad; it's just
what it is.
If Emacs was to become a "modern" app tomorrow, an editor extended in Lisp
still only has appeal for a minority of programmers, much like the Lisp
language itself. Most programmers looking for easy and modern experiences will
likely stick with Atom and Sublime.
Most of the push for a "modern look" comes from the desire for Emacs to play
more nicely with proprietary platforms. Rather, the goal of Emacs is to
support platforms like GNU/Linux. Platforms that respect your freedom, and
also do not push a corporate UI/UX vision of "modernity".
(Perhaps if we do move forward with modernization, we should think of
modernization in the context of something like GNOME rather than MacOS or
Windows. Surely Emacs could be a better citizen of GNOME.)
Given that many of the people complaining about "how Emacs looks" are not
submitting patches to fix the problem themselves, resources would be diverted
from actual functionality to "modernity".
By the time we do major code refactoring "modernizing" Emacs on the major
proprietary platforms, what is "modern" has now once again changed, and our
resources were put towards a project with a poor return on investment.
Basically, I don't see a "modernizing" project playing out well. We will spend
extensive time and energy on a moving target, and even if we succeed, our
Lisp-based vision still has limited appeal. Additionally, I don't think
"modernizing" Emacs advances the cause of free software, given that there are
other more popular casual libre tools for text editing that individuals can
use.
Core values are self-reinforcing. They attract like-minded people, who will
then defend them.
I'm an Emacs user, and reading the Emacs mailing lists serves to remind me that
my values are very different from the values held by maintainers and core
contributors. I don't value freedom or stability nearly as strongly and have
an inner affinity for progressiveness and velocity.
💭
One part of valuing progressiveness is constantly re-evaluating: is our current process or technology as good as it could be? What could be improved? How do we measure improvement? How are others solving these problems? Were there any advances in our area that we could leverage?
* * *
Now let's talk about Vim. I see Vim as intersecting with a few of Emacs' values,
but ultimately diverging radically with its narrow focus on providing really
efficient editing capabilities.
Vim
Doing one thing well
Editing efficiency
Keyboard centrism
Stability
Text centrism
One might notice that extensibility is not in the list. That's intentional.
Vim is certainly extensible to a degree, but it just does not compare to Emacs.
Vim has a "plugin system", while Emacs is the system. Your code becomes part
of it the moment it's evaluated. Since I'm sticking to yes/no indicators for
values I'm giving it a no.
Stability emanates from communications with the primary maintainer.
Vim development is slow, it's quite stable and still there are plenty of bugs
to fix. Adding a new feature always means new bugs, thus hardly any new
features are going to be added now. I did add a few for Vim 7.3, and that did
introduce quite a few new problems. Even though several people said the patch
worked fine.
At the end of 2013, a few folks were trying to get new concurrency primitives
merged into Vim. This would empower plugin authors to create entirely new types
of functionality and by extension, make Vim better.
This is what one of them had to say about the process:
The author of Neovim (Thiago de Arruda) tried to add support for
multi-threaded plugins to Vim and has been stymied.
I'm not sure how to get a patch merged into Vim. Bram Moolenar is the only
person with commit access, and he's not a fan of most changes beyond bug
fixes. My co-founder and I tried to add setTimeout & setInterval to vimscript.
Even six weeks of full-time effort and bending over backwards wasn't enough.
Eventually we were just ignored.
I've contributed to a lot of open source projects, and the Vim community has
been the most difficult to work with. I've been writing C for almost two
decades, and the Vim codebase is the worst C I've ever seen. The project is
definitely showing its age, and I'd love for something new to replace it.
While they understood that some of their values were ultimately incompatible
with the values of the Vim maintainers—who prioritized stability—they still
tried to push for a change, because they treasured the idea of Vim, embodied
by some of its values.
It didn't happen, so a Vim fork came to life: Neovim.
The vision was grand, and is summarized in a statement of its values:
Neovim is a Vim-based text editor engineered for extensibility and
usability, to encourage new applications and contributions.
improving testing, tooling, and CI to simplify maintenance, make aggressive
refactorings possible, and greatly reduce contributor friction
decoupling the core from the UI, making it possible to embed the Vim core into
browsers or IDEs (or any computer program really), also making way for more
powerful and diverse GUIs
embedding a Lua runtime and providing concurrency primitives to open the doors
for smoother, more efficient, and powerful plugins
extensive refactoring: bringing C code to modern standards (C99, leveraging
new compiler features), replacing platform-specific IO code with libuv,
removing support for legacy systems and compilers, including automatic
formatting, and fixing static analysis warnings and errors
creating a scriptable terminal emulator
And they delivered it.
In a very short amount of time they were able to, and I don't use this word
lightly, revolutionize Vim. The impact can be seen in Vim development, which
picked up considerably as Neovim gained ground, with features and processes
ending up being reimplemented in Vim.
💡️
And they aren't stopping there. Current plans include:
translating all Vimscript to Lua under the hood, increasing execution performance due to leveraging LuaJIT, a very, very fast runtime
shipping a built-in LSP client
Neovim builds upon Vim, and the way I see it, holds the following core values:
Neovim
Approachability
Editing efficiency
Extensibility
Keyboard centrism
Progressiveness
Text centrism
Velocity
As I see it, it also currently has a better story than Emacs on:
user interface: all UI-related code for every single platform was removed
from the core, replaced by a consistent
API (both TUIs and GUIs use it) that
made a diversity of decoupled
display implementations possible (UIs
are literally plugins!)
out of the box experience: better defaults, built-in LSP client
💡️
This article provides interesting perspectives on mailing-list-driven-development (and conveniently aligns with my own thinking).
These items are the outcome of massive change that came about through consistent
hard work from a few individuals who shared a vision and a set of values.
Crucially, it included aggressively improving the human side of software:
raising money to support development, lowering contribution friction, unblocking
contributors, reconciling and combining efforts, documenting processes. In other
words, the type of invaluable work non-software engineers do in technology
companies. To our detriment, in open source these tasks are often neglected.
Code is the easy part of building software.
It's hard to contest that Neovim's achievement happened because of its
approachable development process focused on maintainability and velocity,
while in contrast, it could be argued that current progress in Emacs happens
despite its development process.
For example, because Emacs highly values freedom, contributing to Emacs core
(or to packages in the official repository) requires
assigning copyright
to the FSF. To incorporate packages into the main repository, everyone who
committed to the project needs to have gone through that procedure. Even in the
case of a very willing, actual core Emacs maintainer, of an uncontroversially
valuable package used by virtually everyone, this process can take
years.
💡️
It also makes it impossible for some to contribute to Emacs. Check out this lively discussion about Emacs copyright assignment on Reddit for more context.
It is also not hard to find
criticism coming from
folks who have already and continue to give so much to the community and
ecosystem.
It all comes down to core values.
* * *
Let's now address the 800-pound gorilla in the room: VSCode.
It provides a powerful, refined, cohesive out of the box experience with great
performance.
It has immense leverage by building on top of Electron, NodeJS, and Chromium,
projects that receive contributions in the millions of person-hours of work,
from both the open source community and heavily invested corporations.
Here's how I see its values.
VSCode
Approachability
Integration
Maintainability
Progressiveness
Velocity
We can now put it all together in this very uncontroversial table.
Value
Emacs
Vim
Neovim
VSCode
Approachability
✓
✓
Doing one thing well
✓
Editing efficiency
✓
✓
Extensibility
✓
✓
Freedom
✓
Integration
✓
Introspectability
✓
Keyboard centrism
✓
✓
✓
Maintainability
✓
Progressiveness
✓
✓
Stability
✓
✓
Text centrism
✓
✓
✓
Velocity
✓
✓
Irrespective of values, VSCode is looking more and more as an acceptable Emacs
replacement.
It is somewhat extensible and very configurable
It can be mostly driven from a keyboard
It has a great extension language, TypeScript (which is in my opinion superior
to Emacs Lisp in terms of maintainability for non-trivial projects)
It also shines in areas where Emacs doesn't: if you're a programmer working on
typical contemporary projects, mostly just wanting to get stuff done, things
usually... just work. You install VSCode, open a source code file, get asked
to install the extension for that particular language, and that's it. You get
smart completion, static analysis, linting, advanced debugging, refactoring
tools, deep integration with git, and on top of that, great performance and a
cohesive user experience.
💡️
Ironically, LSP (originally developed by Microsoft for VSCode) is one of the main things bringing not only progressiveness and approachability but also integration to Emacs.
This type of experience is the selling point of
Doom and
Spacemacs, two initiatives driven by
relentless maintainers. These projects bring approachability and integration
to Emacs, and are in my view, along with
LSP,
Magit,
and Org, the biggest reasons drawing people to Emacs
nowadays. It is however clear from looking at their issue trackers just how
difficult it is to provide this cohesive experience by combining parts from the
ecosystem.
💡️
Since Emacs is so malleable, it is very easy for packages to interfere with one another, depend on functionality from other packages that get deprecated, changed in incompatible ways, or removed. There's currently no way for a package to depend on a specific version of another package, or for multiple versions of a single package to be loaded at the same time, for example.
With Neovim also shaping up as a worthwhile up-and-comer, this is probably the
first time Emacs has actual competition in its own turf.
In
"Emacs is my "favourite Emacs package""
Protesilaos Stavrou talks about the importance of Emacs, the platform. While
Emacs packages can be valuable in isolation, combined, they amplify the platform
that made them possible. The whole becomes greater than the sum of its parts.
It wouldn't be a stretch to say that Org represents 10% of my cognitive
function. Magit really is "Git at the speed of thought", and I have yet to see a
more integrated and rich interactive shell than Eshell.
And yet, even being a very enthusiastic Emacs user, I have a hard time
recommending it to folks who mostly just want to get stuff done. Some will argue
that those who aren't willing to build their computing environment from scratch
shouldn't be using a "power tool" like Emacs anyway. I don't see a fundamental
reason for that to be the case, and believe that not having young folks trying
out, using, and contributing to Emacs, is not a good thing for Emacs.
This existential threat seems to be acknowledged by maintainers.
One of the gravest problems I see for the future of Emacs development is that
we slowly but steadily lose old-timers who know a lot about the Emacs
internals and have lots of experience hacking them, whereas the (welcome)
newcomers mostly prefer working on application-level code in Lisp. If this
tendency continues, we will soon lose the ability to make deep infrastructure
changes, i.e. will be unable to add new features that need non-trivial changes
on the C level.
For better or worse, Emacs overfits to the needs and priorities of its
maintainers, and contributors who overcome its barriers to entry. Being a
decentralized, volunteer-based project, people will commonly scratch their own
itches or work on
whatever they find interesting. Which is only fair: they could be doing
literally anything else, and yet they choose to sacrifice their time and do
their best to advance Emacs according to their values. They owe no one anything
and deserve gratitude.
* * *
In a 2015 keynote, while laying out an argument for why the Go programming
language is open source at all, Russ Cox portrayed an active and intentional
effort to lower barriers to entry and deliberately improve the human side of Go,
so that as many people as possible used and contributed to it.
The core values of Go are incidentally made apparent through the talk:
Approachability
Developer productivity
Large-scale development
Performance
Simplicity
Go was created to make Google's developers more productive and give the company
a competitive advantage by being able to build products faster and maintain them
more easily. Why share it with the world?
Russ argues that the business justification for it is that it is the only way
that Go can succeed.
A language needs large, broad communities.
A language needs lots of people writing lots of software, so that when you
need a particular tool or library, there's a good chance it has already been
written, by someone who knows the topic better than you, and who spent more
time than you have to make it great.
A language needs lots of people reporting bugs, so that problems are
identified and fixed quickly. Because of the much larger user base, the Go
compilers are much more robust and spec-compliant than the Plan 9 C compilers
they're loosely based on ever were.
A language needs lots of people using it for lots of different purposes, so
that the language doesn't overfit to one use case and end up useless when the
technology landscape changes.
A language needs lots of people who want to learn it, so that there is a
market for people to write books or teach courses, or run conferences like
this one.
None of this could have happened if Go had stayed within Google. Go would have
suffocated inside Google, or inside any single company or closed environment.
Fundamentally, Go must be open, and Go needs you. Go can't succeed without all
of you, without all the people using Go for all different kinds of projects
all over the world.
The parallel to Emacs isn't direct, but it's clear.
Emacs evolved greatly since its inception in 1976 as a collection of macros for
the programmable TECO text editor, which is itself from 1962. Take a look at
this example TECO program, from Wikipedia:
0uz
<j 0aua l
<0aub
qa-qb"g xa k -l ga -1uz '
qbua
l .-z;>
qz;>
It was a different world then. Updating the display text in real-time as users
typed into the keyboard was a recent innovation.
Since then, Emacs Lisp was created, Emacs forks came and went, a graphical UI
was added, lexical scope was implemented, rudimentary networking and concurrency
primitives were introduced, and more.
It can be argued that stability and incremental evolution are the reasons why
Emacs survived and is still thriving. Stability though is necessarily
antithetical to progressiveness. The very thing that likely made it succeed is
what slows it down.
It doesn't, however, affect progressiveness as much as freedom. Because of
freedom, when faced with a question of using technology that is
Non-free, but objectively better
Free, but objectively worse
the latter
will always be picked. Given that most technological progress happens
through the mechanisms of capitalism,
"free" alternatives commonly lag behind to a large
degree.
Freedom has a price.
It can be seen clearly and succinctly in this exchange:
[...] I think freedom is more important than technical progress. Proprietary
software offers plenty of technical "progress", but since I won't surrender my
freedom to use it, as far as I'm concerned it is no progress at all.
If I had valued technical advances over freedom in 1984, instead of developing
GNU Emacs and GCC and GDB I would have gone to work for AT&T and improved its
nonfree software. What a big head start I could have got!
Agree with him or not, RMS has a point. The ability to inspect and change
software running on our computing machines gives us control.
Apple for example seems to be growing increasingly antagonistic to the privacy
of its users (and
bullish to its developers).
As I write this, it's been discovered that newer versions of macOS include
anti-malware functionality that transmits tracking information almost every time
any program is run. This information is sent unencrypted via a third-party
CDN, so not only this could be seen as a privacy violation but also a dangerous
data breach: anyone listening on the network can
roughly know which applications
you use, how often you use them, when do you use them, and from where.
There are measures that can still be taken to ameliorate this situation and
others, but it's ultimately outside of our control. macOS is a proprietary
operating system and can easily prevent users from taking these steps in the
future.
I choose to pay the price of compromising my freedom by tolerating invasions of
my privacy so that I can have a computer that mostly just works and allows me
to be productive towards achieving my life goals. We have to pick our battles,
and the hills we die on depend strongly on our values.
💭
Becoming aware of these facts is still not enough to make me switch back to GNU/Linux for my personal computing needs. At least for now...
We are increasingly finding ourselves in a world where we have to choose between
extremes: do you want a computing machine that respects your privacy, or a
modern, powerful one?
* * *
Back to Emacs.
Maintainers and core contributors likely use it in very different ways than the
majority of users and casual contributors and therefore have very different
priorities. For example, until it got stolen in 2012,
RMS used a 9-inch netbook
because "it could run with free software at the BIOS level" (these days he uses
an 11-year-old T400s). The recent
Emacs User Survey 2020 might help identify
prevailing usage patterns, and hopefully have an impact on the direction of
Emacs.
Emacs doesn't need a Neoemacs as much as Vim needed Neovim. Unlike Vim, Emacs
always had a rich ecosystem of active contributors, and maintainers who—to some
degree—listen to user feedback. There is still tension, rooted in ideology and
values, which throughout Emacs history materialized as forks: Lucid/XEmacs,
Guile Emacs, Aquamacs, Mac port, Remacs.
Forking is incredibly difficult to pull off. A successful one requires not only
an initial momentum and enthusiasm, but also unrelenting, sustained hard work
from a group of individuals, not to mention buy-in from a critical mass of
users. As someone said to me, you have to be a "special kind of crazy" to start
an Emacs fork.
I hope that Emacs doesn't find itself becoming "perfectly suited for a world
that no longer exists". Still, I understand and appreciate the difficulty of the
situation. People have diverging values and are
highly fallible.
Everything requires so much effort. So is our condition.
Ultimately, building software is a complex and deeply human activity. Everything
is contextual and there are rarely easy answers. Most meaningful progress
happens through consensus, compromise, luck, and lots of hard work.
In the end, a lot can be understood through the lens of values.
At work I contribute to a moderately-sized monorepo at 70 thousand files,
8-digit lines of code and hundreds of PRs merged every day. One day I opened a
remote buffer at that repository and ran M-x find-file.
💡️
find-file is an interactive function that shows a narrowed list
of files in the current directory, prompts the user to filter and scroll
through candidates, and for a file to open.
Emacs froze for 5 seconds before showing me the find-file prompt. Which
isn't great, because when writing software, opening files is actually something
one needs to do all the time.
Luckily, Emacs is "the extensible, customizable, self-documenting real-time
display editor", and comes with profiling capabilities: M-x profiler-start
starts a profile and M-x profiler-report displays a call tree showing how much
CPU cycles are spent in each function call after starting the profile. Starting
a profile and running M-x find-file showed that all time was being spent in a
function called ffap-guess-file-name-at-point, which was being called by
file-name-at-point-functions, an
abnormal hook
run when find-file is called.
💡️
If you're familiar with Vim you can think of Emacs hooks as Vim
autocommands, only with much better ergonomics.
I checked the documentation for ffap-guess-file-name-at-point with
M-x describe-function ffap-guess-file-name-at-point and it didn't seem to be
something essential, so I removed the hook by running M-x eval-expression,
writing the form below, and pressing RET.
This solved the immediate problem of Emacs blocking for 5 seconds every time I
ran find-file, with no noticeable drawbacks.
As I write this I attempt to reproduce the issue by re-adding
ffap-guess-file-name-at-point
to
file-name-at-point-functions.
I can't reproduce it anymore. The initial issue might have been
caused by having manually mutated the Emacs environment via ad-hoc code
evaluation (drifting from the state defined in configuration)
caused by settings or packages that aren't in my configuration anymore
fixed by settings or packages that were recently added to my configuration
fixed by some recent package upgrade
Or some combination of the above. I have no idea exactly what. Which is to say:
maintaining Emacs configurations is complicated.
I could now navigate around and open files. The next thing I tried in this
remote git repository was searching through project files. The great
projectile package provides the projectile-find-file function for that, but
I had previously given up making projectile perform well with remote buffers;
given how things are currently implemented it seems to be
impractical. So I installed
the find-file-in-project package for use on remote projects exclusively:
M-x package-install find-file-in-project.
💡️
Most Emacs commands are accessible via key combinations, with defaults that can be customized to be anything you want. I'll stick to referencing command names themselves instead of their default keybindings.
Both projectile-find-file and find-file-in-project (aliased as ffip):
show a narrowed list of all project files in the minibuffer
prompt the user to filter and scroll through candidates
open a file when RET is pressed on a candidate.
To disable projectile on remote buffers I had the following form in my
configuration.
Which causes the projectile-project-root function to not run its usual
implementation on remote buffers, but instead return nil unconditionally.
projectile-project-root is used as a way to either get the project root for a
given buffer (remote or not), or as a boolean predicate to test if the buffer is
in a project (e.g., a git repository directory). Having it return nil on
remote buffers effectively disables projectile on remote buffers.
💡️
Emacs advices are a way of modifying the behavior of existing functions without having to redefine them. They serve a similar purpose as hooks, but are more flexible.
I then wrote a function that falls back to ffip when projectile is disabled
and bound it to the keybinding I had for projectile-find-file, so that I could
press the same keybinding whenever I wanted to search for projects files, and
not have to think about whether I'm on a remote buffer or not:
(apply 'max '(12))
(defun maybe-projectile-find-file ()
"Run `projectile-find-file' if in a project buffer, `ffip' otherwise."
(interactive)
(if (projectile-project-p)
(projectile-find-file)
(ffip)))
Emacs froze for 30 seconds. After that, it showed the prompt with the
narrowed list of files in the project. 30 seconds! What was it doing during the
whole time? Let's try out the profiler again.
Start a new profile:
M-x profiler-start
Call the function to be profiled:
M-x maybe-projectile-find-file (it freezes Emacs again for 30 seconds)
And display the report:
M-x profiler-report
Which showed:
Function CPU samples %
+ ... 21027 98%
+ command-execute 361 1%
This tells us that 98% of the CPU time was spent in whatever ... is. Pressing
TAB on a line will expand it by showing its child function calls.
Expanding ... shows that Emacs spent 64% of CPU time in
ivy--insert-minibuffer and 9% of the time—roughly 3 whole seconds!—garbage
collecting. I had garbage-collection-messages set to t so I could already
tell that Emacs was GCing a lot; enabling this setting makes a message be
displayed in the echo area whenever Emacs garbage collects. I could also see the
Emacs process consuming 100% of one CPU core while it was frozen and
unresponsive to input.
The profiler package implements a sampling profiler. The elp package can be used for getting actual wall clock times.
Drilling down on #<compiled 0x131f715d2b6fa0a8> shows that cycles there (17%
of CPU time) were spent on Emacs waiting for user input, so we can ignore it for
now.
As I get deep in drilling down on ivy--insert-minibuffer, names in the
"Function" column start getting truncated because the column is too narrow. A
quick Google search (via M-x google-this emacs profiler report width) shows me
how to make it wider:
Describing those variables with M-x describe-variable shows that the default
values are 50.
From the profiler report buffer I run M-x eval-expression, paste the form
above with C-y and press RET. I also persist this form to my configuration.
Pressing c in the profiler report buffer (bound to
profiler-report-render-calltree) redraws it, now with a wider column, allowing
me to see the function names.
Here is the abbreviated expanded relevant portion of the call stack.
A couple of things to unpack here. From lines 8-11 it could deduced that ffip
maps a lambda that calls expand-file-name over all completion candidates,
which in this case are around 70 thousand file names. Running
M-x find-function ffip-project-search and narrowing to the relevant region in
the function shows
exactly that:
💡️
find-function shows the definition of a given function, in its source file.
On line 11 of the profiler report we can see that 60% of 30 seconds (18 seconds)
was spent on expand-file-name calls. By dividing 18 seconds by 70000 we get
that expand-file-name calls took 250µs on average. 250µs is how long a modern
computer takes to
read 1MB sequentially from RAM! Why
would my computer need to do that amount of work 70000 times just to display a
narrowed list of files?
Let's see if the function documentation for expand-file-name provides any
clarity.
M-x describe-function expand-file-name
expand-file-name is a function defined in C source code.
Signature
(expand-file-name NAME &optional DEFAULT-DIRECTORY)
Documentation
Convert filename NAME to absolute, and canonicalize it.
Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
(does not start with slash or tilde); both the directory name and
a directory's file name are accepted. If DEFAULT-DIRECTORY is nil or
missing, the current buffer's value of default-directory is used.
NAME should be a string that is a valid file name for the underlying
filesystem.
Ok, so it sounds like expand-file-name essentially transforms a file path into
an absolute path, based on either the current buffer's directory or optionally,
a directory passed in as an additional argument. Let's try evaluating some forms
with M-x eval-expression both on a local and a remote buffer to get a sense of
what it does.
In a local dired buffer at my local home directory:
The expand-file-name call in ffip-project-search doesn't specify a
DEFAULT-DIRECTORY (the optional second parameter to expand-file-name) so
like in the examples above it defaults to the current buffer's directory, which
in the profiled case is a remote path like in the second example above.
With a better understanding of what expand-file-namedoes, let's now try to
understand how it performs. We can benchmark it with benchmark-run in local
and remote buffers, and compare their runtimes.
M-x describe-function benchmark-run
benchmark-run is an autoloaded macro defined in benchmark.el.gz.
Signature
(benchmark-run &optional REPETITIONS &rest FORMS)
Documentation
Time execution of FORMS.
If REPETITIONS is supplied as a number, run forms that many times,
accounting for the overhead of the resulting loop. Otherwise run
FORMS once.
Return a list of the total elapsed time for execution, the number of
garbage collections that ran, and the time taken by garbage collection.
Benchmarking it in a local dired buffer at my local home directory
showed that it took 0.3 seconds to run expand-file-name 70 thousand times on a
local buffer, and 30 seconds to do so on a remote buffer: two orders of
magnitude slower. 30 seconds is more than what we observed in the profiler
report (18 seconds), and I'll attribute this discrepancy to unknowns; maybe the
ffip execution took advantage of byte-compiled code evaluation, or there's
some overhead associated with benchmark-run, or something else entirely.
Nevertheless, this experiment clearly corroborates the profiler report results.
So! Back to ffip. Looking again at the previous screenshot, it seems that the
list of displayed files doesn't even show absolute file paths. Why is
expand-file-name being called at all? Maybe calling it isn't too important...
Let's remove the expand-file-name call by
visiting the ffip-project-search function in the library file with
M-x find-function ffip-project-search
"raising" file in the lambda
re-evaluating ffip-project-search with M-x eval-defun
It's faster. This change alone reduces the time for ffip to show the candidate
list from 30 seconds to 8 seconds with no noticeable drawbacks. Which is
better, but still not even close to acceptable.
Profiling the changed function shows that now most of the time is spent in
sorting candidates with ivy-prescient-sort-function, and garbage collection.
Automatic sorting of candidates based on selection recency comes from the
excellent ivy and ivy-prescient packages, which I had installed and
configured. Disabling ivy-prescient with M-x ivy-prescient-mode and
re-running my function reduces the time further from 8 seconds to 4 seconds.
Another thing I notice is that ffip allows
fd to be used as a backend instead of GNU
find. fd claims to have better performance, so I install it on the remote host
and configure ffip to use it. I evaluate the form below like before, but I
could also have used the very handy M-x counsel-set-variable, which shows a
narrowed list of candidates of all variables in Emacs (in my setup there's
around 20 thousand) along with a snippet of their docstrings, and on selection
allows the variable value to be set. Convenient!
(setq ffip-use-rust-fd t)
Which brings my function's runtime to a little over 2 seconds—a 15x performance
improvement overall—achieved via:
Manually evaluating a modified function from an installed library file
Installing a program on the remote host and configuring ffip to use it
The last point is not really an issue, but the whole situation is not ideal.
Even putting aside all of the above points, I don't want to wait for over 2
seconds every time I search for files in this project.
Let's see if we can do better than that.
So far we've been mostly configuring and introspecting Emacs. Let's now
extend it with new functionality that satisfies our needs.
We want a function that:
Based on a remote buffer's directory, figures out its remote project root
directory
Runs fd on the remote project root directory
Presents the output from fd as a narrowed list of candidate files, with it
being possible to filter, scroll, and select a candidate from the list
Has good performance and is responsive even on large, remote projects
Let's see if there's anything in find-file-in-project that we could reuse. I
know that ffip is figuring out project roots and running shell commands
somehow. By checking out its library file with
M-x find-library find-file-in-project (which opens a buffer with the
installed find-file-in-project.el package file) I can see that the
shell-command-to-string function (included with Emacs) is being used for
running shell commands, and that there's a function named ffip-project-root
that sounds a lot like what we need.
I have a keybinding that shows the documentation for the thing under the cursor.
I use it to inspect the two functions:
ffip-project-root
ffip-project-root is an autoloaded function defined in
find-file-in-project.el.
Signature
(ffip-project-root)
Documentation
Return project root or default-directory.
shell-command-to-string
shell-command-to-string is a compiled function defined in
simple.el.gz.
Signature
(shell-command-to-string COMMAND)
Documentation
Execute shell command COMMAND and return its output as a string.
Perfect. We should be able to reuse them.
I also know that the ivy-read function provided by ivy should take care of
displaying the narrowed list of files. Looks like we won't need to write a lot
of code.
To verify that our code will work on remote buffers we'll need to evaluate forms
in the context of one. The with-current-buffer macro can be used for that.
M-x describe-function with-current-buffer
with-current-buffer is a macro defined in subr.el.gz.
Signature
(with-current-buffer BUFFER-OR-NAME &rest BODY)
Documentation
Execute the forms in BODY with BUFFER-OR-NAME temporarily current.
BUFFER-OR-NAME must be a buffer or the name of an existing buffer.
The value returned is the value of the last form in BODY. See
also with-temp-buffer.
For writing our function, instead of evaluating forms ad-hoc with
M-x eval-expression, we'll open a scratch buffer and write and evaluate
forms directly from there, which should be more convenient.
I have a clone of the Linux git repository
on my remote host. Let's assign a remote buffer for the officially funniest file
in the Linux kernel, jiffies.c—
Notice that the buffer is just a value, and can be passed around to functions.
We'll use it further ahead to emulate evaluating forms as if we had that
buffer opened, with the with-current-buffer macro.
Let's start exploring by writing to the *scratch* buffer and continuing to
evaluate forms one by one with eval-defun.
fd tells us that there are 28 C files in /home/mpereira/linux/kernel/time.
Let's see if we can get the project root, which would be /home/mpereira/linux.
Let's now play with default-directory. This is a buffer-local variable that
holds a buffer's working directory. By evaluating forms with a redefined
default-directory it's possible to emulate being in another directory, which
could even be on a remote host. The code block below is an example of that—the
second form redefines default-directory to be the project root.
Since it was declared (interactive) we can also to call it via
M-x my-project-find-file.
Going back to the large remote project and running my-project-find-file a few
times shows that it now runs in a little over a second—a 30x improvement
compared with what we started with.
This is still not good enough, so I went ahead and evolved the function we
were working on to most of the time show something on screen immediately and
redraw it asynchronously. You can check out the code at
fast-project-find-file.el.
As an aside: having the whole text editor block for over a second while I wait
for it to show something so simple is unacceptable. Through desensitization
and acquiescence, we, users of software have come to expect that it will
either not work at all, not work consistently, or exhibit poor or
unpredictable performance.
Did you notice how the function implementation came almost naturally from
exploration? The immediate feedback from evaluating forms and modifying a live
system—even though old news to Lisp programmers—is incredibly powerful. Combine
it with an "extensible, customizable, self-documenting" environment and you have
a very satisfying and productive means of creation.
One of the oldest pieces of software still in use was recently described as
A sort of hybrid between Windows Notepad, a monolithic-kernel operating
system, and the International Space Station.
Of course, they were talking about Emacs. And yes, it is kinda true.
I've been using Emacs for a while and had opportunities to use it to work on
projects in remote machines. There are a few quirks, but after changing a
setting here and there,
Tramp
is mostly usable. Tramp makes it possible to transparently treat remote
directories, files, commands, and more, as if they were local. Opening a
remote buffer via M-x find-file /ssh:user@remote-host:/some/project and
having tools like Magit and Eshell work out of the box feels like magic.
I'm not aware of similar functionality in Vim, but VSCode users recently got something pretty close.
Emacs runs either as a standalone graphical application (GUI) or in a terminal
emulator (TUI). In terms of features, GUI Emacs can be seen as a superset of TUI
Emacs. Among many other things, it makes it possible to display:
TUI Emacs has the terminal emulator (and also commonly tmux) limiting and conflicting with the clipboard, keybindings, colors, and more. None of these limitations are present in GUI Emacs.
For these and other reasons I use GUI Emacs and Tramp to work on remote projects
instead of TUI Emacs via SSH. Some do the opposite, and that's fine too! Even
with the limitations shown above, TUI Emacs is very powerful, and its
performance in an SSH session is still superior to Tramp.
Even so, one might think displaying images in Emacs is not a big deal. Just
open image.png and get an actual image viewer application to render it, right?
Things get interesting in a remote Tramp buffer: M-x find-file image.png
will display the remote image right there in your local Emacs instance. Or
in case you're in a remote dired buffer, pressing RET on an image file will
do the same. It's a non-disruptive workflow that allows one to remain in the
comfort of Emacs.
dired is a built-in file manager.
If you're not an Emacs user, chances are the previous sentence doesn't carry
much weight. What's good about staying inside Emacs?
It is safe to assume that most of your development, or more broadly, your
computing environment is represented below:
operating system (macOS, GNU/Linux distribution, Windows, etc.)
window manager (the one provided by your OS, i3, Openbox, XMonad, etc.)
text editor or IDE (Vim, VSCode, Xcode, IntelliJ IDEA, etc.)
email, web browsing, multimedia, and communications
Each item above is a discrete computer program. They're extensible in
disconnected ways, and to varying degrees. They're islands. Most of the time
integrating them is an uphill battle, and commonly just plain impractical.
Having used a computing environment centered around TUI Vim and tmux for a
decade, I know the pain of trying to make the shell, the terminal, and every
command line application running in it have the same look, feel, and behavior of
being inside Vim. It is impossible.
The emergence of the web browser as a software platform is in part an answer to this disconnectedness.
In contrast, Emacs unifies and equalizes the computing experience as much as one
desires, so that it happens—and is extended—in a single, cohesive environment.
My computing environment.
People commonly
use Emacs
to process their email inbox, communicate in chat, navigate the web, write code,
and prose. At their core, these activities are one and the same: text-editing.
Being able to perform them using the same keybindings and functionality for
movement, search, text manipulation, completion, undo-redo, copy-paste, is
already a big deal, but it's not all: their integration into a single
environment unlocks pleasant and efficient workflows that would be much less
convenient elsewhere.
Creating a to-do item for something your partner just asked you (while you were
concentrated on a task) that will show up automatically on your agenda tomorrow?
Just a few keystrokes and you're back to the task. Converting some text notes
you quickly scribbled down into a beautiful PDF via LaTeX, uploading it to S3,
and referencing the URL in a reworded existing git commit message? If you're an
experienced Emacs user, chances are you can visualize effortlessly and
efficiently executing these actions without ever leaving Emacs.
This interconnectedness is part of the value of computing environments like the
one provided by Emacs. The whole becomes greater than the sum of its parts.
Programmability gives it a further dimension of value: parts can be combined as
you see fit, in arbitrary and possibly unforeseen ways.
Now, back to the displaying an image from a remote host use case. For that, I
can think of some alternatives:
scp the image file between hosts and open it locally
Mount the remote filesystem and open it locally
ssh into the remote host from a modern terminal emulator capable of
rendering images and use a program like
icat
I've recently joined a company that for security reasons doesn't allow their
source code on laptops. Development happens strictly on workstations inside
their private networks, with some using their text editor's support for remote
file editing and others running editors on those machines via SSH.
Adapting to this situation has indirectly led me into a bit of a rabbit hole,
forcing me to acknowledge my core values, better understand the relation between
progress and human flourishing, and ponder about the question: why technology?
One of the oldest pieces of software still in use was recently described as
A sort of hybrid between Windows Notepad, a monolithic-kernel operating
system, and the International Space Station.
Of course, they were talking about Emacs. And yes, it is kinda true.
I've been using Emacs for a while and had opportunities to use it to work on
projects in remote machines. There are a few quirks, but after changing a
setting here and there,
Tramp
is mostly usable. Tramp makes it possible to transparently treat remote
directories, files, commands, and more, as if they were local. Opening a
remote buffer via M-x find-file /ssh:user@remote-host:/some/project and
having tools like Magit and Eshell work out of the box feels like magic.
I'm not aware of similar functionality in Vim, but VSCode users recently got something pretty close.
Emacs runs either as a standalone graphical application (GUI) or in a terminal
emulator (TUI). In terms of features, GUI Emacs can be seen as a superset of TUI
Emacs. Among many other things, it makes it possible to display:
TUI Emacs has the terminal emulator (and also commonly tmux) limiting and conflicting with the clipboard, keybindings, colors, and more. None of these limitations are present in GUI Emacs.
For these and other reasons I use GUI Emacs and Tramp to work on remote projects
instead of TUI Emacs via SSH. Some do the opposite, and that's fine too! Even
with the limitations shown above, TUI Emacs is very powerful, and its
performance in an SSH session is still superior to Tramp.
Even so, one might think displaying images in Emacs is not a big deal. Just
open image.png and get an actual image viewer application to render it, right?
Things get interesting in a remote Tramp buffer: M-x find-file image.png
will display the remote image right there in your local Emacs instance. Or
in case you're in a remote dired buffer, pressing RET on an image file will
do the same. It's a non-disruptive workflow that allows one to remain in the
comfort of Emacs.
dired is a built-in file manager.
If you're not an Emacs user, chances are the previous sentence doesn't carry
much weight. What's good about staying inside Emacs?
It is safe to assume that most of your development, or more broadly, your
computing environment is represented below:
operating system (macOS, GNU/Linux distribution, Windows, etc.)
window manager (the one provided by your OS, i3, Openbox, XMonad, etc.)
text editor or IDE (Vim, VSCode, Xcode, IntelliJ IDEA, etc.)
email, web browsing, multimedia, and communications
Each item above is a discrete computer program. They're extensible in
disconnected ways, and to varying degrees. They're islands. Most of the time
integrating them is an uphill battle, and commonly just plain impractical.
Having used a computing environment centered around TUI Vim and tmux for a
decade, I know the pain of trying to make the shell, the terminal, and every
command line application running in it have the same look, feel, and behavior of
being inside Vim. It is impossible.
The emergence of the web browser as a software platform is in part an answer to this disconnectedness.
In contrast, Emacs unifies and equalizes the computing experience as much as one
desires, so that it happens—and is extended—in a single, cohesive environment.
My computing environment.
People commonly
use Emacs
to process their email inbox, communicate in chat, navigate the web, write code,
and prose. At their core, these activities are one and the same: text-editing.
Being able to perform them using the same keybindings and functionality for
movement, search, text manipulation, completion, undo-redo, copy-paste, is
already a big deal, but it's not all: their integration into a single
environment unlocks pleasant and efficient workflows that would be much less
convenient elsewhere.
Creating a to-do item for something your partner just asked you (while you were
concentrated on a task) that will show up automatically on your agenda tomorrow?
Just a few keystrokes and you're back to the task. Converting some text notes
you quickly scribbled down into a beautiful PDF via LaTeX, uploading it to S3,
and referencing the URL in a reworded existing git commit message? If you're an
experienced Emacs user, chances are you can visualize effortlessly and
efficiently executing these actions without ever leaving Emacs.
This interconnectedness is part of the value of computing environments like the
one provided by Emacs. The whole becomes greater than the sum of its parts.
Programmability gives it a further dimension of value: parts can be combined as
you see fit, in arbitrary and possibly unforeseen ways.
Now, back to the displaying an image from a remote host use case. For that, I
can think of some alternatives:
scp the image file between hosts and open it locally
Mount the remote filesystem and open it locally
ssh into the remote host from a modern terminal emulator capable of
rendering images and use a program like
icat
In Emacs, you just press enter.
Opening a file
At work I contribute to a moderately-sized monorepo at 70 thousand files,
8-digit lines of code and hundreds of PRs merged every day. One day I opened a
remote buffer at that repository and ran M-x find-file.
💡️
find-file is an interactive function that shows a narrowed list
of files in the current directory, prompts the user to filter and scroll
through candidates, and for a file to open.
Emacs froze for 5 seconds before showing me the find-file prompt. Which
isn't great, because when writing software, opening files is actually something
one needs to do all the time.
Luckily, Emacs is "the extensible, customizable, self-documenting real-time
display editor", and comes with profiling capabilities: M-x profiler-start
starts a profile and M-x profiler-report displays a call tree showing how much
CPU cycles are spent in each function call after starting the profile. Starting
a profile and running M-x find-file showed that all time was being spent in a
function called ffap-guess-file-name-at-point, which was being called by
file-name-at-point-functions, an
abnormal hook
run when find-file is called.
💡️
If you're familiar with Vim you can think of Emacs hooks as Vim
autocommands, only with much better ergonomics.
I checked the documentation for ffap-guess-file-name-at-point with
M-x describe-function ffap-guess-file-name-at-point and it didn't seem to be
something essential, so I removed the hook by running M-x eval-expression,
writing the form below, and pressing RET.
This solved the immediate problem of Emacs blocking for 5 seconds every time I
ran find-file, with no noticeable drawbacks.
As I write this I attempt to reproduce the issue by re-adding
ffap-guess-file-name-at-point
to
file-name-at-point-functions.
I can't reproduce it anymore. The initial issue might have been
caused by having manually mutated the Emacs environment via ad-hoc code
evaluation (drifting from the state defined in configuration)
caused by settings or packages that aren't in my configuration anymore
fixed by settings or packages that were recently added to my configuration
fixed by some recent package upgrade
Or some combination of the above. I have no idea exactly what. Which is to say:
maintaining Emacs configurations is complicated.
I could now navigate around and open files. The next thing I tried in this
remote git repository was searching through project files. The great
projectile package provides the projectile-find-file function for that, but
I had previously given up making projectile perform well with remote buffers;
given how things are currently implemented it seems to be
impractical. So I installed
the find-file-in-project package for use on remote projects exclusively:
M-x package-install find-file-in-project.
💡️
Most Emacs commands are accessible via key combinations, with defaults that can be customized to be anything you want. I'll stick to referencing command names themselves instead of their default keybindings.
Both projectile-find-file and find-file-in-project (aliased as ffip):
show a narrowed list of all project files in the minibuffer
prompt the user to filter and scroll through candidates
open a file when RET is pressed on a candidate.
To disable projectile on remote buffers I had the following form in my
configuration.
Which causes the projectile-project-root function to not run its usual
implementation on remote buffers, but instead return nil unconditionally.
projectile-project-root is used as a way to either get the project root for a
given buffer (remote or not), or as a boolean predicate to test if the buffer is
in a project (e.g., a git repository directory). Having it return nil on
remote buffers effectively disables projectile on remote buffers.
💡️
Emacs advices are a way of modifying the behavior of existing functions without having to redefine them. They serve a similar purpose as hooks, but are more flexible.
I then wrote a function that falls back to ffip when projectile is disabled
and bound it to the keybinding I had for projectile-find-file, so that I could
press the same keybinding whenever I wanted to search for projects files, and
not have to think about whether I'm on a remote buffer or not:
(apply 'max '(12))
(defun maybe-projectile-find-file ()
"Run `projectile-find-file' if in a project buffer, `ffip' otherwise."
(interactive)
(if (projectile-project-p)
(projectile-find-file)
(ffip)))
Emacs froze for 30 seconds. After that, it showed the prompt with the
narrowed list of files in the project. 30 seconds! What was it doing during the
whole time? Let's try out the profiler again.
Start a new profile:
M-x profiler-start
Call the function to be profiled:
M-x maybe-projectile-find-file (it freezes Emacs again for 30 seconds)
And display the report:
M-x profiler-report
Which showed:
Function CPU samples %
+ ... 21027 98%
+ command-execute 361 1%
This tells us that 98% of the CPU time was spent in whatever ... is. Pressing
TAB on a line will expand it by showing its child function calls.
Expanding ... shows that Emacs spent 64% of CPU time in
ivy--insert-minibuffer and 9% of the time—roughly 3 whole seconds!—garbage
collecting. I had garbage-collection-messages set to t so I could already
tell that Emacs was GCing a lot; enabling this setting makes a message be
displayed in the echo area whenever Emacs garbage collects. I could also see the
Emacs process consuming 100% of one CPU core while it was frozen and
unresponsive to input.
The profiler package implements a sampling profiler. The elp package can be used for getting actual wall clock times.
Drilling down on #<compiled 0x131f715d2b6fa0a8> shows that cycles there (17%
of CPU time) were spent on Emacs waiting for user input, so we can ignore it for
now.
As I get deep in drilling down on ivy--insert-minibuffer, names in the
"Function" column start getting truncated because the column is too narrow. A
quick Google search (via M-x google-this emacs profiler report width) shows me
how to make it wider:
Describing those variables with M-x describe-variable shows that the default
values are 50.
From the profiler report buffer I run M-x eval-expression, paste the form
above with C-y and press RET. I also persist this form to my configuration.
Pressing c in the profiler report buffer (bound to
profiler-report-render-calltree) redraws it, now with a wider column, allowing
me to see the function names.
Here is the abbreviated expanded relevant portion of the call stack.
A couple of things to unpack here. From lines 8-11 it could deduced that ffip
maps a lambda that calls expand-file-name over all completion candidates,
which in this case are around 70 thousand file names. Running
M-x find-function ffip-project-search and narrowing to the relevant region in
the function shows
exactly that:
💡️
find-function shows the definition of a given function, in its source file.
On line 11 of the profiler report we can see that 60% of 30 seconds (18 seconds)
was spent on expand-file-name calls. By dividing 18 seconds by 70000 we get
that expand-file-name calls took 250µs on average. 250µs is how long a modern
computer takes to
read 1MB sequentially from RAM! Why
would my computer need to do that amount of work 70000 times just to display a
narrowed list of files?
Let's see if the function documentation for expand-file-name provides any
clarity.
M-x describe-function expand-file-name
expand-file-name is a function defined in C source code.
Signature
(expand-file-name NAME &optional DEFAULT-DIRECTORY)
Documentation
Convert filename NAME to absolute, and canonicalize it.
Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
(does not start with slash or tilde); both the directory name and
a directory's file name are accepted. If DEFAULT-DIRECTORY is nil or
missing, the current buffer's value of default-directory is used.
NAME should be a string that is a valid file name for the underlying
filesystem.
Ok, so it sounds like expand-file-name essentially transforms a file path into
an absolute path, based on either the current buffer's directory or optionally,
a directory passed in as an additional argument. Let's try evaluating some forms
with M-x eval-expression both on a local and a remote buffer to get a sense of
what it does.
In a local dired buffer at my local home directory:
The expand-file-name call in ffip-project-search doesn't specify a
DEFAULT-DIRECTORY (the optional second parameter to expand-file-name) so
like in the examples above it defaults to the current buffer's directory, which
in the profiled case is a remote path like in the second example above.
With a better understanding of what expand-file-namedoes, let's now try to
understand how it performs. We can benchmark it with benchmark-run in local
and remote buffers, and compare their runtimes.
M-x describe-function benchmark-run
benchmark-run is an autoloaded macro defined in benchmark.el.gz.
Signature
(benchmark-run &optional REPETITIONS &rest FORMS)
Documentation
Time execution of FORMS.
If REPETITIONS is supplied as a number, run forms that many times,
accounting for the overhead of the resulting loop. Otherwise run
FORMS once.
Return a list of the total elapsed time for execution, the number of
garbage collections that ran, and the time taken by garbage collection.
Benchmarking it in a local dired buffer at my local home directory
showed that it took 0.3 seconds to run expand-file-name 70 thousand times on a
local buffer, and 30 seconds to do so on a remote buffer: two orders of
magnitude slower. 30 seconds is more than what we observed in the profiler
report (18 seconds), and I'll attribute this discrepancy to unknowns; maybe the
ffip execution took advantage of byte-compiled code evaluation, or there's
some overhead associated with benchmark-run, or something else entirely.
Nevertheless, this experiment clearly corroborates the profiler report results.
So! Back to ffip. Looking again at the previous screenshot, it seems that the
list of displayed files doesn't even show absolute file paths. Why is
expand-file-name being called at all? Maybe calling it isn't too important...
Let's remove the expand-file-name call by
visiting the ffip-project-search function in the library file with
M-x find-function ffip-project-search
"raising" file in the lambda
re-evaluating ffip-project-search with M-x eval-defun
It's faster. This change alone reduces the time for ffip to show the candidate
list from 30 seconds to 8 seconds with no noticeable drawbacks. Which is
better, but still not even close to acceptable.
Profiling the changed function shows that now most of the time is spent in
sorting candidates with ivy-prescient-sort-function, and garbage collection.
Automatic sorting of candidates based on selection recency comes from the
excellent ivy and ivy-prescient packages, which I had installed and
configured. Disabling ivy-prescient with M-x ivy-prescient-mode and
re-running my function reduces the time further from 8 seconds to 4 seconds.
Another thing I notice is that ffip allows
fd to be used as a backend instead of GNU
find. fd claims to have better performance, so I install it on the remote host
and configure ffip to use it. I evaluate the form below like before, but I
could also have used the very handy M-x counsel-set-variable, which shows a
narrowed list of candidates of all variables in Emacs (in my setup there's
around 20 thousand) along with a snippet of their docstrings, and on selection
allows the variable value to be set. Convenient!
(setq ffip-use-rust-fd t)
Which brings my function's runtime to a little over 2 seconds—a 15x performance
improvement overall—achieved via:
Manually evaluating a modified function from an installed library file
Installing a program on the remote host and configuring ffip to use it
The last point is not really an issue, but the whole situation is not ideal.
Even putting aside all of the above points, I don't want to wait for over 2
seconds every time I search for files in this project.
Let's see if we can do better than that.
So far we've been mostly configuring and introspecting Emacs. Let's now
extend it with new functionality that satisfies our needs.
We want a function that:
Based on a remote buffer's directory, figures out its remote project root
directory
Runs fd on the remote project root directory
Presents the output from fd as a narrowed list of candidate files, with it
being possible to filter, scroll, and select a candidate from the list
Has good performance and is responsive even on large, remote projects
Let's see if there's anything in find-file-in-project that we could reuse. I
know that ffip is figuring out project roots and running shell commands
somehow. By checking out its library file with
M-x find-library find-file-in-project (which opens a buffer with the
installed find-file-in-project.el package file) I can see that the
shell-command-to-string function (included with Emacs) is being used for
running shell commands, and that there's a function named ffip-project-root
that sounds a lot like what we need.
I have a keybinding that shows the documentation for the thing under the cursor.
I use it to inspect the two functions:
ffip-project-root
ffip-project-root is an autoloaded function defined in
find-file-in-project.el.
Signature
(ffip-project-root)
Documentation
Return project root or default-directory.
shell-command-to-string
shell-command-to-string is a compiled function defined in
simple.el.gz.
Signature
(shell-command-to-string COMMAND)
Documentation
Execute shell command COMMAND and return its output as a string.
Perfect. We should be able to reuse them.
I also know that the ivy-read function provided by ivy should take care of
displaying the narrowed list of files. Looks like we won't need to write a lot
of code.
To verify that our code will work on remote buffers we'll need to evaluate forms
in the context of one. The with-current-buffer macro can be used for that.
M-x describe-function with-current-buffer
with-current-buffer is a macro defined in subr.el.gz.
Signature
(with-current-buffer BUFFER-OR-NAME &rest BODY)
Documentation
Execute the forms in BODY with BUFFER-OR-NAME temporarily current.
BUFFER-OR-NAME must be a buffer or the name of an existing buffer.
The value returned is the value of the last form in BODY. See
also with-temp-buffer.
For writing our function, instead of evaluating forms ad-hoc with
M-x eval-expression, we'll open a scratch buffer and write and evaluate
forms directly from there, which should be more convenient.
I have a clone of the Linux git repository
on my remote host. Let's assign a remote buffer for the officially funniest file
in the Linux kernel, jiffies.c—
Notice that the buffer is just a value, and can be passed around to functions.
We'll use it further ahead to emulate evaluating forms as if we had that
buffer opened, with the with-current-buffer macro.
Let's start exploring by writing to the *scratch* buffer and continuing to
evaluate forms one by one with eval-defun.
fd tells us that there are 28 C files in /home/mpereira/linux/kernel/time.
Let's see if we can get the project root, which would be /home/mpereira/linux.
Let's now play with default-directory. This is a buffer-local variable that
holds a buffer's working directory. By evaluating forms with a redefined
default-directory it's possible to emulate being in another directory, which
could even be on a remote host. The code block below is an example of that—the
second form redefines default-directory to be the project root.
Since it was declared (interactive) we can also to call it via
M-x my-project-find-file.
Going back to the large remote project and running my-project-find-file a few
times shows that it now runs in a little over a second—a 30x improvement
compared with what we started with.
This is still not good enough, so I went ahead and evolved the function we
were working on to most of the time show something on screen immediately and
redraw it asynchronously. You can check out the code at
fast-project-find-file.el.
As an aside: having the whole text editor block for over a second while I wait
for it to show something so simple is unacceptable. Through desensitization
and acquiescence, we, users of software have come to expect that it will
either not work at all, not work consistently, or exhibit poor or
unpredictable performance.
Did you notice how the function implementation came almost naturally from
exploration? The immediate feedback from evaluating forms and modifying a live
system—even though old news to Lisp programmers—is incredibly powerful. Combine
it with an "extensible, customizable, self-documenting" environment and you have
a very satisfying and productive means of creation.
In 2018 Bryan Cantrill gave a brilliant talk where he shared his recent
experiences with the Rust programming language. More profoundly, he explored a
facet of software that is oftentimes overlooked: the values of the software we
use. To paraphrase him slightly:
Values are defined as expressions of relative importance. Two things that
we're comparing could both be good attributes. The real question is, when you
have to make a choice between two of them, what do you choose? That choice
that you make, reflects your core values.
He goes ahead to contrast the core values of some programming languages with the
core values we demand from systems software, like operating system kernels, file
systems, microprocessors, and so on. It is a really good talk and you should
watch it.
It is important to think about values because they are core to the decisions
that we make.
Unlike systems software, the values demanded from text editors or IDEs vary
greatly depending on who you ask. These are much more personal tools and make
room for a diverse set of desires.
The following listing enumerates values that could be attributed to development
tools.
Value
Commentary
Approachability
Ease of getting started with for typical tasks, and contribution friendliness
Doing one thing well
Unix philosophy, fitting into an ecosystem
Editing efficiency
Fewer interactions, mnemonics, composable keystrokes, etc.
Extensibility
The degree to which behavior and appearance can be changed
Cohesive core and concerted third-party functionality
Introspectability
Capable of being understood and inspected ad-hoc
Keyboard centrism
Focus on keyboard interactions
Maintainability
The degree to which it can be modified without introducing faults
Progressiveness
A measure of eagerness to make progress and leverage modern technology
Stability
Things that worked before continue to work the same way
Text centrism
Text as a universal interface
Velocity
Short and focused release cycles, aligned personpower, leveraging the community effectively
💭
Before we go any further, I'd like to point that out if you care about any of the topics discussed ahead you will likely strongly disagree with something or the other.
That's fine! We probably just have different values.
In my view, Emacs has the following core values:
Emacs
Extensibility
Freedom
Introspectability
Keyboard centrism
Stability
Text centrism
We can feel the clasp of stability in the following—rather poetic—exchange in
the Emacs development mailing list, which also provides useful historical
perspectives.
Emacs is older than the operating systems people use today. (It is almost as
old as the first Unix, which barely resembled the Unix of later decades.) It
is much older than Linux, the kernel.
The oldest design elements were not designed for the uses we make of them
today. And since we wrote those, people have developed other areas of software
which don't fit Emacs very well. So there are good reasons to redesign some of
them.
However, people actually use Emacs, so a greatly incompatible change in Emacs
is as unthinkable as a greatly incompatible change in the New York City
subway.
We have to build new lines through the maze of underground pipes and cables.
The following exchange reifies freedom and stability while demonstrating a
disinclination to progressiveness. Which is neither good nor bad; it's just
what it is.
If Emacs was to become a "modern" app tomorrow, an editor extended in Lisp
still only has appeal for a minority of programmers, much like the Lisp
language itself. Most programmers looking for easy and modern experiences will
likely stick with Atom and Sublime.
Most of the push for a "modern look" comes from the desire for Emacs to play
more nicely with proprietary platforms. Rather, the goal of Emacs is to
support platforms like GNU/Linux. Platforms that respect your freedom, and
also do not push a corporate UI/UX vision of "modernity".
(Perhaps if we do move forward with modernization, we should think of
modernization in the context of something like GNOME rather than MacOS or
Windows. Surely Emacs could be a better citizen of GNOME.)
Given that many of the people complaining about "how Emacs looks" are not
submitting patches to fix the problem themselves, resources would be diverted
from actual functionality to "modernity".
By the time we do major code refactoring "modernizing" Emacs on the major
proprietary platforms, what is "modern" has now once again changed, and our
resources were put towards a project with a poor return on investment.
Basically, I don't see a "modernizing" project playing out well. We will spend
extensive time and energy on a moving target, and even if we succeed, our
Lisp-based vision still has limited appeal. Additionally, I don't think
"modernizing" Emacs advances the cause of free software, given that there are
other more popular casual libre tools for text editing that individuals can
use.
Core values are self-reinforcing. They attract like-minded people, who will
then defend them.
I'm an Emacs user, and reading the Emacs mailing lists serves to remind me that
my values are very different from the values held by maintainers and core
contributors. I don't value freedom or stability nearly as strongly and have
an inner affinity for progressiveness and velocity.
💭
One part of valuing progressiveness is constantly re-evaluating: is our current process or technology as good as it could be? What could be improved? How do we measure improvement? How are others solving these problems? Were there any advances in our area that we could leverage?
* * *
Now let's talk about Vim. I see Vim as intersecting with a few of Emacs' values,
but ultimately diverging radically with its narrow focus on providing really
efficient editing capabilities.
Vim
Doing one thing well
Editing efficiency
Keyboard centrism
Stability
Text centrism
One might notice that extensibility is not in the list. That's intentional.
Vim is certainly extensible to a degree, but it just does not compare to Emacs.
Vim has a "plugin system", while Emacs is the system. Your code becomes part
of it the moment it's evaluated. Since I'm sticking to yes/no indicators for
values I'm giving it a no.
Stability emanates from communications with the primary maintainer.
Vim development is slow, it's quite stable and still there are plenty of bugs
to fix. Adding a new feature always means new bugs, thus hardly any new
features are going to be added now. I did add a few for Vim 7.3, and that did
introduce quite a few new problems. Even though several people said the patch
worked fine.
At the end of 2013, a few folks were trying to get new concurrency primitives
merged into Vim. This would empower plugin authors to create entirely new types
of functionality and by extension, make Vim better.
This is what one of them had to say about the process:
The author of Neovim (Thiago de Arruda) tried to add support for
multi-threaded plugins to Vim and has been stymied.
I'm not sure how to get a patch merged into Vim. Bram Moolenar is the only
person with commit access, and he's not a fan of most changes beyond bug
fixes. My co-founder and I tried to add setTimeout & setInterval to vimscript.
Even six weeks of full-time effort and bending over backwards wasn't enough.
Eventually we were just ignored.
I've contributed to a lot of open source projects, and the Vim community has
been the most difficult to work with. I've been writing C for almost two
decades, and the Vim codebase is the worst C I've ever seen. The project is
definitely showing its age, and I'd love for something new to replace it.
While they understood that some of their values were ultimately incompatible
with the values of the Vim maintainers—who prioritized stability—they still
tried to push for a change, because they treasured the idea of Vim, embodied
by some of its values.
It didn't happen, so a Vim fork came to life: Neovim.
The vision was grand, and is summarized in a statement of its values:
Neovim is a Vim-based text editor engineered for extensibility and
usability, to encourage new applications and contributions.
improving testing, tooling, and CI to simplify maintenance, make aggressive
refactorings possible, and greatly reduce contributor friction
decoupling the core from the UI, making it possible to embed the Vim core into
browsers or IDEs (or any computer program really), also making way for more
powerful and diverse GUIs
embedding a Lua runtime and providing concurrency primitives to open the doors
for smoother, more efficient, and powerful plugins
extensive refactoring: bringing C code to modern standards (C99, leveraging
new compiler features), replacing platform-specific IO code with libuv,
removing support for legacy systems and compilers, including automatic
formatting, and fixing static analysis warnings and errors
creating a scriptable terminal emulator
And they delivered it.
In a very short amount of time they were able to, and I don't use this word
lightly, revolutionize Vim. The impact can be seen in Vim development, which
picked up considerably as Neovim gained ground, with features and processes
ending up being reimplemented in Vim.
💡️
And they aren't stopping there. Current plans include:
translating all Vimscript to Lua under the hood, increasing execution performance due to leveraging LuaJIT, a very, very fast runtime
shipping a built-in LSP client
Neovim builds upon Vim, and the way I see it, holds the following core values:
Neovim
Approachability
Editing efficiency
Extensibility
Keyboard centrism
Progressiveness
Text centrism
Velocity
As I see it, it also currently has a better story than Emacs on:
user interface: all UI-related code for every single platform was removed
from the core, replaced by a consistent
API (both TUIs and GUIs use it) that
made a diversity of decoupled
display implementations possible (UIs
are literally plugins!)
out of the box experience: better defaults, built-in LSP client
💡️
This article provides interesting perspectives on mailing-list-driven-development (and conveniently aligns with my own thinking).
These items are the outcome of massive change that came about through consistent
hard work from a few individuals who shared a vision and a set of values.
Crucially, it included aggressively improving the human side of software:
raising money to support development, lowering contribution friction, unblocking
contributors, reconciling and combining efforts, documenting processes. In other
words, the type of invaluable work non-software engineers do in technology
companies. To our detriment, in open source these tasks are often neglected.
Code is the easy part of building software.
It's hard to contest that Neovim's achievement happened because of its
approachable development process focused on maintainability and velocity,
while in contrast, it could be argued that current progress in Emacs happens
despite its development process.
For example, because Emacs highly values freedom, contributing to Emacs core
(or to packages in the official repository) requires
assigning copyright
to the FSF. To incorporate packages into the main repository, everyone who
committed to the project needs to have gone through that procedure. Even in the
case of a very willing, actual core Emacs maintainer, of an uncontroversially
valuable package used by virtually everyone, this process can take
years.
💡️
It also makes it impossible for some to contribute to Emacs. Check out this lively discussion about Emacs copyright assignment on Reddit for more context.
It is also not hard to find
criticism coming from
folks who have already and continue to give so much to the community and
ecosystem.
It all comes down to core values.
* * *
Let's now address the 800-pound gorilla in the room: VSCode.
It provides a powerful, refined, cohesive out of the box experience with great
performance.
It has immense leverage by building on top of Electron, NodeJS, and Chromium,
projects that receive contributions in the millions of person-hours of work,
from both the open source community and heavily invested corporations.
Here's how I see its values.
VSCode
Approachability
Integration
Maintainability
Progressiveness
Velocity
We can now put it all together in this very uncontroversial table.
Value
Emacs
Vim
Neovim
VSCode
Approachability
✓
✓
Doing one thing well
✓
Editing efficiency
✓
✓
Extensibility
✓
✓
Freedom
✓
Integration
✓
Introspectability
✓
Keyboard centrism
✓
✓
✓
Maintainability
✓
Progressiveness
✓
✓
Stability
✓
✓
Text centrism
✓
✓
✓
Velocity
✓
✓
Irrespective of values, VSCode is looking more and more as an acceptable Emacs
replacement.
It is somewhat extensible and very configurable
It can be mostly driven from a keyboard
It has a great extension language, TypeScript (which is in my opinion superior
to Emacs Lisp in terms of maintainability for non-trivial projects)
It also shines in areas where Emacs doesn't: if you're a programmer working on
typical contemporary projects, mostly just wanting to get stuff done, things
usually... just work. You install VSCode, open a source code file, get asked
to install the extension for that particular language, and that's it. You get
smart completion, static analysis, linting, advanced debugging, refactoring
tools, deep integration with git, and on top of that, great performance and a
cohesive user experience.
💡️
Ironically, LSP (originally developed by Microsoft for VSCode) is one of the main things bringing not only progressiveness and approachability but also integration to Emacs.
This type of experience is the selling point of
Doom and
Spacemacs, two initiatives driven by
relentless maintainers. These projects bring approachability and integration
to Emacs, and are in my view, along with
LSP,
Magit,
and Org, the biggest reasons drawing people to Emacs
nowadays. It is however clear from looking at their issue trackers just how
difficult it is to provide this cohesive experience by combining parts from the
ecosystem.
💡️
Since Emacs is so malleable, it is very easy for packages to interfere with one another, depend on functionality from other packages that get deprecated, changed in incompatible ways, or removed. There's currently no way for a package to depend on a specific version of another package, or for multiple versions of a single package to be loaded at the same time, for example.
With Neovim also shaping up as a worthwhile up-and-comer, this is probably the
first time Emacs has actual competition in its own turf.
In
"Emacs is my "favourite Emacs package""
Protesilaos Stavrou talks about the importance of Emacs, the platform. While
Emacs packages can be valuable in isolation, combined, they amplify the platform
that made them possible. The whole becomes greater than the sum of its parts.
It wouldn't be a stretch to say that Org represents 10% of my cognitive
function. Magit really is "Git at the speed of thought", and I have yet to see a
more integrated and rich interactive shell than Eshell.
And yet, even being a very enthusiastic Emacs user, I have a hard time
recommending it to folks who mostly just want to get stuff done. Some will argue
that those who aren't willing to build their computing environment from scratch
shouldn't be using a "power tool" like Emacs anyway. I don't see a fundamental
reason for that to be the case, and believe that not having young folks trying
out, using, and contributing to Emacs, is not a good thing for Emacs.
This existential threat seems to be acknowledged by maintainers.
One of the gravest problems I see for the future of Emacs development is that
we slowly but steadily lose old-timers who know a lot about the Emacs
internals and have lots of experience hacking them, whereas the (welcome)
newcomers mostly prefer working on application-level code in Lisp. If this
tendency continues, we will soon lose the ability to make deep infrastructure
changes, i.e. will be unable to add new features that need non-trivial changes
on the C level.
For better or worse, Emacs overfits to the needs and priorities of its
maintainers, and contributors who overcome its barriers to entry. Being a
decentralized, volunteer-based project, people will commonly scratch their own
itches or work on
whatever they find interesting. Which is only fair: they could be doing
literally anything else, and yet they choose to sacrifice their time and do
their best to advance Emacs according to their values. They owe no one anything
and deserve gratitude.
* * *
In a 2015 keynote, while laying out an argument for why the Go programming
language is open source at all, Russ Cox portrayed an active and intentional
effort to lower barriers to entry and deliberately improve the human side of Go,
so that as many people as possible used and contributed to it.
The core values of Go are incidentally made apparent through the talk:
Approachability
Developer productivity
Large-scale development
Performance
Simplicity
Go was created to make Google's developers more productive and give the company
a competitive advantage by being able to build products faster and maintain them
more easily. Why share it with the world?
Russ argues that the business justification for it is that it is the only way
that Go can succeed.
A language needs large, broad communities.
A language needs lots of people writing lots of software, so that when you
need a particular tool or library, there's a good chance it has already been
written, by someone who knows the topic better than you, and who spent more
time than you have to make it great.
A language needs lots of people reporting bugs, so that problems are
identified and fixed quickly. Because of the much larger user base, the Go
compilers are much more robust and spec-compliant than the Plan 9 C compilers
they're loosely based on ever were.
A language needs lots of people using it for lots of different purposes, so
that the language doesn't overfit to one use case and end up useless when the
technology landscape changes.
A language needs lots of people who want to learn it, so that there is a
market for people to write books or teach courses, or run conferences like
this one.
None of this could have happened if Go had stayed within Google. Go would have
suffocated inside Google, or inside any single company or closed environment.
Fundamentally, Go must be open, and Go needs you. Go can't succeed without all
of you, without all the people using Go for all different kinds of projects
all over the world.
The parallel to Emacs isn't direct, but it's clear.
Emacs evolved greatly since its inception in 1976 as a collection of macros for
the programmable TECO text editor, which is itself from 1962. Take a look at
this example TECO program, from Wikipedia:
0uz
<j 0aua l
<0aub
qa-qb"g xa k -l ga -1uz '
qbua
l .-z;>
qz;>
It was a different world then. Updating the display text in real-time as users
typed into the keyboard was a recent innovation.
Since then, Emacs Lisp was created, Emacs forks came and went, a graphical UI
was added, lexical scope was implemented, rudimentary networking and concurrency
primitives were introduced, and more.
It can be argued that stability and incremental evolution are the reasons why
Emacs survived and is still thriving. Stability though is necessarily
antithetical to progressiveness. The very thing that likely made it succeed is
what slows it down.
It doesn't, however, affect progressiveness as much as freedom. Because of
freedom, when faced with a question of using technology that is
Non-free, but objectively better
Free, but objectively worse
the latter
will always be picked. Given that most technological progress happens
through the mechanisms of capitalism,
"free" alternatives commonly lag behind to a large
degree.
Freedom has a price.
It can be seen clearly and succinctly in this exchange:
[...] I think freedom is more important than technical progress. Proprietary
software offers plenty of technical "progress", but since I won't surrender my
freedom to use it, as far as I'm concerned it is no progress at all.
If I had valued technical advances over freedom in 1984, instead of developing
GNU Emacs and GCC and GDB I would have gone to work for AT&T and improved its
nonfree software. What a big head start I could have got!
Agree with him or not, RMS has a point. The ability to inspect and change
software running on our computing machines gives us control.
Apple for example seems to be growing increasingly antagonistic to the privacy
of its users (and
bullish to its developers).
As I write this, it's been discovered that newer versions of macOS include
anti-malware functionality that transmits tracking information almost every time
any program is run. This information is sent unencrypted via a third-party
CDN, so not only this could be seen as a privacy violation but also a dangerous
data breach: anyone listening on the network can
roughly know which applications
you use, how often you use them, when do you use them, and from where.
There are measures that can still be taken to ameliorate this situation and
others, but it's ultimately outside of our control. macOS is a proprietary
operating system and can easily prevent users from taking these steps in the
future.
I choose to pay the price of compromising my freedom by tolerating invasions of
my privacy so that I can have a computer that mostly just works and allows me
to be productive towards achieving my life goals. We have to pick our battles,
and the hills we die on depend strongly on our values.
💭
Becoming aware of these facts is still not enough to make me switch back to GNU/Linux for my personal computing needs. At least for now...
We are increasingly finding ourselves in a world where we have to choose between
extremes: do you want a computing machine that respects your privacy, or a
modern, powerful one?
* * *
Back to Emacs.
Maintainers and core contributors likely use it in very different ways than the
majority of users and casual contributors and therefore have very different
priorities. For example, until it got stolen in 2012,
RMS used a 9-inch netbook
because "it could run with free software at the BIOS level" (these days he uses
an 11-year-old T400s). The recent
Emacs User Survey 2020 might help identify
prevailing usage patterns, and hopefully have an impact on the direction of
Emacs.
Emacs doesn't need a Neoemacs as much as Vim needed Neovim. Unlike Vim, Emacs
always had a rich ecosystem of active contributors, and maintainers who—to some
degree—listen to user feedback. There is still tension, rooted in ideology and
values, which throughout Emacs history materialized as forks: Lucid/XEmacs,
Guile Emacs, Aquamacs, Mac port, Remacs.
Forking is incredibly difficult to pull off. A successful one requires not only
an initial momentum and enthusiasm, but also unrelenting, sustained hard work
from a group of individuals, not to mention buy-in from a critical mass of
users. As someone said to me, you have to be a "special kind of crazy" to start
an Emacs fork.
I hope that Emacs doesn't find itself becoming "perfectly suited for a world
that no longer exists". Still, I understand and appreciate the difficulty of the
situation. People have diverging values and are
highly fallible.
Everything requires so much effort. So is our condition.
Ultimately, building software is a complex and deeply human activity. Everything
is contextual and there are rarely easy answers. Most meaningful progress
happens through consensus, compromise, luck, and lots of hard work.
In the end, a lot can be understood through the lens of values.
With corporations like Microsoft, Oracle, and Google truly reinventing
themselves to adapt to an open source world, and typical open source projects
moving towards—oftentimes centralized—governance models, the
Cathedral-Bazaar
dichotomy feels increasingly less relevant.
While being an entertaining piece of history with useful takeaways, its most
important achievement was arguably helping create a sense of identity for hacker
culture via the revolutionary Open Source movement, and promoting the value of
the Internet for software development.
In the Cathedral-Bazaar continuum, contemporary projects like Kubernetes,
Chromium, and VSCode are fusion reactors.
They have the backing of heavily invested companies with virtually infinite
capital, who are able to staff highly competent teams that not only work
full-time on these projects, but also have enough personpower to maximally
leverage the benefits brought by a gigantic user base.
Like with fusion power, they seem to be able to leverage a high amount of energy
to generate even more.
Sometimes, their user base includes other organizations with endless resources: by means of its success, VSCode is getting sizable contributions from Facebook, for example.
In contrast, the vast majority of open source projects depend almost exclusively
on decentralized volunteer efforts from people sacrificing time out of their
busy schedules and lives to move things forward.
And yet, projects following this style of development can end up becoming
backbones of modern computing:
The OpenSSL project has been around since 1998. Since the project is open
source, it is an informal group comprised primarily of about a dozen members
throughout the world, most of whom have day jobs, and some of whom work on a
volunteer basis. Being open source, the OpenSSL project's code has always been
public facing. Any person could download it and modify it or implement it in
their own software.
[...]
The fascinating, mind-boggling fact here is that you have this critical piece
of network infrastructure that really runs a large part of the internet, and
there's basically one guy working on it full time.
Heartbleed is a symptom of an ever-existing problem: corporations profit
massively by leveraging typically under-resourced open source projects while not
giving back proportionally, or (most commonly) at all; either with money or
people.
After Heartbleed the Core Infrastructure Initiative was created to support software essential to the "functioning of the Internet and other major information systems".
Less critical software like Emacs also follows this decentralized development
style, and similarly, lacks resources.
So if Emacs wants to compete with these tools then it has to have seamless,
context aware code completion and refactoring support, and GNU tools has to
provide Emacs the necessary information to implement these features.
I agree. But to have that, the only way is to have motivated volunteers step
forward and work on these features. Otherwise we will never have them.
Right now, no one is working on that, though everyone is talking. [T]he same
as with weather.
One way to incentivize contributions is by funding developers. Some (most?) open
source contributors would gladly take income to fund their work.
Long-term my big dream has always been to accumulate enough backing to be able
to work full-time on open-source projects, but whether I'll achieve this dream
or not is entirely up to you.
While others feel that accepting funding would degrade their
intrinsic motivation
to contribute.
I can't speak for all FLOSS developers, but I can speak for myself: I don't
want monetary rewarding from users. Mainly for the reason, that I don't want
to change the relationship with my users. Currently it is mostly a team
attitude, we're working together to solve the problem. And there is also no
legal obligation for me to work on something I don't like.
If I accepted contributions I think many users would get a "but I paid for
that, so do what I want" attitude. I definitely don't want that. I do FLOSS in
my free time to do something that matters, and for my personal fulfillment,
not for money.
It will also get harder to do the right thing (in contrary to doing what the
users want) since the users can stop the payments.
Which is fair: with compensation comes responsibility, timelines, expectations,
and things can get complicated. From the point of view of the project, there
doesn't seem to be a conflict: capture funding to enable those who need (or
want) it, while still empowering those who don't, to contribute on their own
terms.
Not everyone is in a position to spend unpaid time on open source, especially
consistently. "Free" time
isn't free—it
costs life. By funding work, a project might get to see contributions from
talented folks passionate about it who wouldn't be able to volunteer their time.
And sometimes, the reason why your PR isn't getting immediate attention is that the maintainer is busy literally fighting a revolution.
For Emacs specifically, one problem is that there's no clear way of funding
"Emacs". Sending money to the FSF doesn't guarantee that it will fund Emacs
development. Even if there was a way of "funding Emacs", the Emacs
community—like many things in life—seems to be roughly divided in two sides:
those who prioritize
freedom, and those who
prioritize progress. So one would potentially want to fund one side or the
other, depending on their values.
In the excellent "Working in Public:
The Making and Maintenance of Open Source Software", Nadia Eghbal calls
attention to a shift in how individuals support open source. Similarly to
platforms like Twitch, more and more people are funding creators directly
instead of projects, as a way of "incentivizing the ongoing creation of creative
work" from developers who produce things that are in their interests.
Maintainers of popular Emacs packages, for example, have their own
separate streams of patronage,
which receive varying levels of support depending on their popularity and the
value added by what they create.
For efforts that involve multiple people, like maintaining and evolving Emacs
itself, services like Open Collective could be of
great assistance by helping on three fronts:
providing a legal banking entity
recurrently collecting funds from individuals and companies
distributing funds to contributors
Funds can be transparently collected, and dispersed for specific contract work,
infrastructure costs, and even developer salaries. Take for example the
Babel project, which draws in enough
recurrent income to finance multiple contractors and a full-time developer
earning a San Francisco salary. An Emacs Open Collective could not only be an
answer to "how can I fund Emacs development?" but also a way to financially
support developers working on it.
The Clojure community seems to be doing an excellent job at not only funding efforts that are making the whole Clojure ecosystem better but also at surveying and responding accordingly to user feedback.
GitHub Sponsors is another great example of
developer empowerment. With it, not only does GitHub equip people to:
be more productive by providing great code hosting, bug tracking, wiki, code
reviewing and merging, project management, continuous integration,
documentation, artifact hosting, etc.
have a broader impact by giving projects more visibility and standardized
workflows that are familiar to others already on the platform
It also makes it possible for its
40 million users to frictionlessly fund work on
open source, and for a large number of maintainers to be conveniently
compensated for their labor. I just started sponsoring someone with literally
two clicks!
The power of platforms can't be understated.
Whether you like GitHub or not, it's undeniable that it has, and continues to
revolutionize Open Source, simply by providing a significantly better and
unified experience for all aspects of building software.
When there are people making over 100k/year on GitHub Sponsors, you better have a great reason to not try to take advantage of it.
In the case of Emacs, the reason is freedom.
I wouldn't mind if Emacs development moved to GitHub, but
I don't think it's ever going to happen.
Maybe for good reason: GitHub is backed by a for-profit corporation and is far
from perfect, both in
moral
and
technical
terms. It might be a great tool today, but being a proprietary platform, its
users are at their complete mercy.
I should point out that from my perspective GitHub has been for the most part a force for good.
It would be great if main development at least moved from a mailing-list-driven
process to a modern forge style of contribution. It
seems that it might, but whether or not it
will is still unclear.
In the same way that corporations extract value out of open source, open source
projects should as much as possible leverage "energy" generated by corporations.
In this new open source world, companies have their workforce contributing
millions of person-hours to projects that benefit everyone. LLVM equips people
to build programming languages. LSP gives people potent software development
capabilities. Rails empowers people to build powerful web applications.
More than 3,000 people have committed man-decades, maybe even man-centuries,
of work for free. Buying all that effort at market rates would have been
hundreds of millions of dollars. Who would have been able to afford funding
that?
That's a monumental achievement of humanity! Thousands, collaborating for a
decade, to produce an astoundingly accomplished framework and ecosystem
available to anyone at the cost of zero. Take a second to ponder the magnitude
of that success. Not just for Rails, of course, but for many other, and
larger, open source projects out there with an even longer lineage and
success.
In many cases, the
ideology ingrained in Emacs prevents it from leveraging value generated by
efforts not totally compatible with the goals of the Free Software movement.
Still, there are many non-conflicting opportunities for improvement.
Maybe Emacs doesn't need to be a fusion reactor. I only hope it continues to
generate energy for many years to come.
It just needs volunteers to keep the fire going.
From catching up to getting ahead
I started using
Emacs
almost exactly four years ago, after almost a decade of
Vim. I made the switch cold turkey. I
vividly remember being extremely frustrated by unbearable slowness while
editing a Clojure file at work. With no sane way of debugging it, just moving
the cursor up and down would result in so much lag that I had to step away from
the computer to breathe for a while. When I came back I quit Vim (I knew how at
that point), opened Emacs, and started building my configuration.
More recently, I've been very put off by the performance and stability (or lack
thereof) of building large scale software via Tramp. This has been sufficient to
have me looking out again. On a whim, I installed VSCode for the first time and
tried its "remote development" capabilities and holy smokes are they good.
Getting up and running was trivial and the performance was great. Saving files
was snappy and LSP worked out of the box. What a different experience from my
carefully-put-together, half-working, slow Emacs setup.
💭
My common denominator for rage-quitting software seems to be consistent: bad performance.
There has recently been more discussion than usual regarding "modernizing"
Emacs, by making keybindings more consistent with other applications and using
more attractive color schemes and visuals, with the end goal of attracting more
users and by extension more contributors.
In my view improving these aspects of user experience wouldn't hurt. The way I
see it, though, is that for Emacs to attract more users it needs to be
objectively better than the alternatives. And the way to do it is for Emacs to
become even more like Emacs.
💭
I see Emacs as being fundamentally two things: a programmable runtime, and a beacon for free software. I'm talking more about the former.
It needs to be a more robust, more efficient, and more integrated platform
with a more powerful extension language, to empower its users to build their
own environment.
Getting LSP integrated pervasively in Emacs in a way that it reliably just
works and performs well out of the box, would go a long way towards making
Emacs more attractive not just to new users, but to existing ones too. Imagine
an experience similar to VSCode's:
Open Emacs for the first time
Open a source code file
Emacs asks if you want it to configure itself for the programming language
of that source file
Saying "yes" automatically sets up Emacs to have a modern programming
environment for that programming language with smart code completion,
navigation, and refactoring, rich hover information, highlighting, automatic
formatting, snippets, etc. Maybe even open a side window with a buffer with
a short "getting started" tutorial showing the available keybindings.
Given enough users, opinionated community-built Emacs "distributions" like
Spacemacs, Doom, and Prelude will do the job of making it easier for newcomers
to get started with typical contemporary tasks: building software with popular
programming languages, writing documents, managing machines, etc.
Building and maintaining these "distributions" also becomes much easier given a
more robust, more efficient, and more integrated platform with a more powerful
extension language.
Having a
wizard
showing up in new Emacs installations might be a great low-hanging fruit way of
making Emacs more accessible. Assuming buy-in from core maintainers, the wizard
could even directly reference popular Emacs "distributions" like the ones
mentioned above, so that new users can kickstart their lives in Emacs.
The way to attract contributors can also be stated simply: directly improve
the contribution process.
💭
Easier said than done.
Many have created their Emacs
wishlists. This is mine:
Improved single-core efficiency
Improved display efficiency and rendering engine
Leveraging preemptive parallelism
Emacs Lisp improvements
Enhanced stability
Dealing with non-text
Improved contribution and development process
Let's get into it.
1. Improved single-core efficiency
There are two dimensions to this:
garbage collection efficiency
code execution efficiency
For the past one and a half years,
Andrea Corallo, a compiler engineer, has been
working on adding native compilation
capabilities to the Emacs Lisp interpreter. His work is available in a branch in
the official Emacs repository. Folks have been trying it out, and according to
the reports I'm hearing, the results are staggeringly positive. I am very
excited about Andrea's work, which seems to bring enough improvement to the
"code execution speed" side of the equation to make it a non-issue for now.
Andrea's work will also allow for more of Emacs to be implemented in Emacs Lisp
itself (instead of C), which is what most contributors are used to. This is a
great win for maintainability and extensibility: incrementally having more and
more of Emacs be implemented in the language with which it's extended.
The garbage collector is still in much need of improvement. Many resort to hacks
to ameliorate frequent and sometimes long pauses that seem to be unavoidable
while working on large git repositories, fast-scrolling font-locked Eshell
buffers, displaying dynamically updating child frames, navigating big Org files,
and many other tasks.
💡️
Also, try this out:(setq garbage-collection-messages t)
2. Improved display efficiency and rendering engine
The display implementation in Emacs core is... less than ideal.
GNU Emacs is an old-school C program emulating a 1980s Symbolics Lisp Machine
emulating an old-fashioned Motif-style Xt toolkit emulating a 1970s text
terminal emulating a 1960s teletype. Compiling Emacs is a challenge. Adding
modern rendering features to the redisplay engine is a miracle.
It would be great if Emacs did like Neovim and decoupled the editor runtime from
the display engine. This would make it possible for the community to build
powerful GUIs without having to change
Emacs core,
possibly
using technologynot fully sanctioned
by core maintainers.
Take a look at the screenshots of these Neovim GUIs:
They're powerful, look great, perform well, and more importantly, are based on
industry standard, cross-platform graphics APIs (Vulkan and WebGL respectively)
that get lots of personpower contributions from companies and individuals alike.
The Onivim and
Xi text editors could also be sources
of inspiration:
Separating the core runtime from the user interface
Ropes for faster
incremental changes and parallelization of text operations
Emacs does not support parallel code execution via multi-core processing. Code
execution happening on any buffer will freeze the whole program, preventing not
only user interaction but other cooperative threads of execution from making
progress as well.
Adding parallelism to Emacs in a way that automatically makes existing code run
in parallel is about as close to impossible as it can get. What would be more
feasible is including new primitives for parallel execution that new code could
leverage, to build more powerful extensions to Emacs.
Emacs-ng is a recent effort that
implements just that: an additive layer over Emacs that brings not only
parallelism, but also asynchronous I/O capabilities via an embedded
Deno runtime, and GPU-based rendering via
WebRender. I am super excited about the
very fast progress from the folks working on emacs-ng, and I think the project
holds great promise for the future of Emacs itself.
There also seems to be advances in the area of immutable data structures that
could be leveraged by the Emacs core, as seen in
"Persistence for the Masses: RRB-Vectors in a Systems Language".
Persistent data structures would make building thread-safe parallel code much
easier.
4. Enhanced stability
It is very easy to either freeze Emacs or cause it to run very slowly. Multiple
times a day I have to hit C-g incessantly to bring it back from being frozen.
When that fails, I am sometimes able to get it back with pkill -SIGUSR2 Emacs.
At least once per week I have to pkill -9 Emacs because it turned completely
unresponsive. I suspect doing more work outside of the main thread might help
with this?
There are many hacks to ameliorate issues caused by long lines, but they're
still fundamentally there. Advancements in the "display efficiency and rendering
engine" effort would help with this too.
I recently tried a package that displays pretty icons on completion prompts, and
noticed that it made scrolling through candidates really slow. Profiling showed
that the package was creating thousands of
timers,
which were somehow causing the issue. There are lots of cases like this, where
folks attempt to create something nice, but inevitably have to resort to
hacks to either
achieve acceptable performance, or to be able to implement the thing at all.
Having a more robust/efficient/integrated core with a more powerful extension
language would help here.
Impressive efforts from folks like Lars Ingebrigtsen who routinely comes in and
obliterates 10% of all reported Emacs bugs
also have a sizable impact. We users should follow the lead and do a better job
not only creating good bug reports but also dipping in our toes and helping out:
fixing bugs, writing tests, and documentation.
Yuan Fu recently wrote a
nice guide for
contributing to Emacs.
5. Emacs Lisp improvements
Emacs Lisp is a much better language than Vimscript. Unfortunately, that's not
saying much. It's not a particularly good Lisp and has lots of room for
improvement.
For example, if you want to use a map, you have three choices: you can use
alists, plists or hash maps. There are no namespaces in Emacs Lisp, so for
each of the three data types you get a bunch of functions with weird names.
For alists get is assoc and set is add-to-list, for hash maps get is
gethash and set is puthash, for plists get is plist-get and set is
plist-put. For each of those types it is easy to find basic use cases that
are not covered by the standard library, plus it is easy to run into
performance pitfalls, so you end up rewriting everything several times to get
something working. The experience is the same across the board, when working
with files, working with strings, running external processes etc. There are
3rd party libraries for all those things now because using the builtins is so
painful.
Emacs Lisp APIs evolved incrementally while maintaining backwards compatibility
over a long period of time. This is good: code written more than a decade ago
still runs. These increments came about via decentralized volunteer efforts, and
it shows: there are many inconsistencies and conflicts between and within
libraries, which feel like having evolved without an overarching design.
Programmers used to languages that did go through
careful, deliberate design brought some
of it to Emacs Lisp:
It would be great to have more and more of these influencing and being
incorporated to the Emacs Lisp standard library and made to be very performant.
map.el and seq.el seem to already be in thanks to Nicolas Petton!
Assuming a multi-core future for Emacs, it will also be critical to have good
ergonomics for writing concurrent code. It should be easy to do the right thing
(writing thread-safe code), and hard to do the wrong thing. I believe Clojure
can also be a
source of inspiration.
How easy it is to just say these things! Easy to do the right thing, hard to do the wrong thing!
Other than that, a great module system, possibly one that allows different
versions of libraries to coexist, would also be a great addition. Andrea Corallo
seems to be trying out some
new ideas in this space
(discussion).
6. Dealing with non-text
It is currently possible to browse the web in Emacs in an embedded
fully-featured WebKit widget. We need to go further—I want to have the same
experience of the likes of Nyxt and
vimperator, integrated to Emacs:
switch between tabs with fuzzy completion (ivy, helm, etc.)
navigate via link hinting
Isearch web pages
easily copy text content from web pages, paste it elsewhere
create macros to repeat actions on web pages
I can kinda do some of these things right now with the existing WebKit widget
along with some clever hacks. There are
other more
adventurous hacks
which work around Emacs to create a full graphical interface. It would be
great if this type of functionality was deeply integrated to Emacs. It's a
difficult thing to do because of the existing display engine implementation and
Emacs Lisp limitations.
I believe doing like Neovim (and others) and separating the
core from
display would help here. But, it would
likely bring its
own problems.
There are unfortunately no silver bullets.
Less importantly but still desirable: email. Even though I write most of my
email messages in Emacs, I read them mostly outside of it. I prefer to exchange
plain text email, but sometimes I receive HTML email. When I do, I'd prefer to
visualize it as the author intended. This is currently technically possible, but
suffers from the same challenges as web browsing.
7. Improved contribution and development process
Contributing to Emacs core and packages in the official repository requires
assigning copyright to the FSF. Employed software developers need to get
paperwork signed by their employers' legal departments, a process that takes
many days. Copyright assignment is likely not going away—can it be made more
convenient?
It would be great to move to a forge style of contribution. It is honestly
incredible to me how people keep track of patches flying around in email
threads. Unless something like sourcehut or Patchwork is being used there's no
automated CI making sure individual patches and overall contributions are in a
good state. Hopefully the Emacs
GitLab instance starts being more actively
used and becomes the official way to contribute.
Copyright assignment and mailing-list driven development are definitely
off-putting to folks who just want to contribute, and aren't looking forward to
having to sign paperwork or learn a special way to contribute to every project
they work with. The GitHub generation of open source developers are used to a
standardized, powerful and convenient platform—anything other than that just
feels not worth it.
Emacs will likely always have a niche of users, but it could grow to not have
developers. Having large parts of the core implementation be in C makes it not
very approachable to anyone outside the handful of contributors who do feel
confident to change it.
It probably makes sense to continuously look for functionality implemented in
the C core that could be replaced with focused libraries, like Neovim did by
replacing almost all of their hacky, platform-specific code with
libuv.
Also, would it make sense to start an Emacs
Open Collective to fund work on Emacs?
Last "small" thing
Improve Tramp performance to match the experience of using terminal Emacs via
SSH, or VSCode's
Remote Development.
* * *
Talking is easy. Accomplishing any of these would require lots of work. It may
not seem like it but text editors are a
hardproblem. And
people, an even harder one.
I wonder if Emacs will stick around long enough and grow the necessary
functionality for us to someday run M-x neuralink-mode and evaluate Lisp in
the brain?
The Why of technology
I think one of the things that really separates us from the high primates is
that we're tool builders. I read a study that measured the efficiency of
locomotion for various species on the planet. The condor used the least energy
to move a kilometer. Humans came in with a rather unimpressive showing about a
third of the way down the list. It was not too proud a showing for the crown
of creation. So, that didn't look so good.
But then, somebody at Scientific American had the insight to test the
efficiency of locomotion for a man on a bicycle. And, a man on a bicycle, a
human on a bicycle, blew the condor away, completely off the top of the
charts.
And that's what a computer is to me. What a computer is to me is it's the most
remarkable tool that we've ever come up with.
No one knows
when or how we, the human species, started talking to each other. It is likely a
natural progression from gesturing, but we can only speculate about it.
Language allowed us to break out of our brains and reveal the inner workings of
our consciousness to others.
Language is the vessel that carried us from the stone age through the
agricultural revolution, the development of written language, the scientific and
industrial revolutions, and now, the digital age.
Writing allowed us to offload memories to the physical world—outside of our
brains. Through our collective and external memories, each generation has a head
start on the previous one. Little by little, standing on the shoulders of taller
and taller giants, we accumulate knowledge about ourselves and everything around
us.
We've been for long using tools to help us think: notebooks help us calculate
formulas, reason geometrically and preserve our ideas. With computers, our
thinking is now occurring outside of our brains.
Computers are extensions of our minds in that they allow us to store, process,
and retrieve information from them. With the advent of the internet we now have
immediate access to not only almost all of the information ever produced by
humankind but also to reproducible thinking encoded into these machines:
algorithms.
Our brain is still a much more impressive device than any of today's computers. Computers learn mostly by finding patterns in massive quantities of examples given by us. Teaching a young kid about cars—how to recognize one, what they are, what their purpose is, and how they're related to other things—requires little supervision. Noam Chomsky talks about it in this interview.
Each of these processes—storing, processing and retrieving information—have
concrete effects on the physical world: if I'm in Munich, saying "show route to
Hamburg" to my phone will immediately show me the distance, ETAs and paths for
different types of transport to reach my destination. Not only do I now suddenly
know how to navigate across the country to reach another city, I'm also able to
follow through the exact path via GPS—a sixth sense giving me perfect
geolocation!
These things that we created—computers, and the internet—are literally
rewiring our brains,
right now, shaping how we think, and engage in social relationships, changing
not only our individual selves but the societies we live in.
They started as mechanical machines that filled entire laboratories, turned into
beige boxes in our homes and places of work, and are now sleek slabs of plastic,
metal and glass in everyone's pockets. Step by step they get closer to our
bodies, their interfaces more intuitive and natural.
The way we communicate with them is changing: before, we could only interact
with them by speaking their language. We have now taught them ours. The torch of
progress blazes on: it's a matter of time until they're connected directly
with our brains—which is equally terrifying and awe-inspiring.
Brain-computer interfaces present a monumental scientific and engineering
challenge, and brain-to-brain, a whole other category of difficulty.
First, we have no idea how information is encoded in the brain. That needs to be
understood. Second, even assuming we're able to take a perfect snapshot of a
piece of information in someone's brain—for example, how a particular movie
scene makes them feel—we still need to be able to encode it in a way that
includes the full context of their subjective experiences. Maybe the scene
evokes unique memories of their childhood or is somehow entangled with the smell
of a particular cinema's leather seats. Third, we need to figure out how to
safely write this perfect snapshot into someone else's brain in a way that can
be perceived identically.
Which is to say, it's a difficult problem. But a worthwhile one: imagine having
the capability to suddenly become aware of answers for questions you just
thought about. To expertly control
truly integrated prosthetics giving you
superhuman abilities. To give movement to the paralized, sound to the deaf, and
sight to the blind.
What would be the impacts on society if we were able to communicate an order of
magnitude more effectively? What if everyone was equipped with the same
undisputed basic knowledge of history and science?
There are internal thoughts that we can attempt to describe with a thousand
words, but ultimately fail to capture in a way that's precise, much less
comprehensible by someone else. Words and sentences are an incomplete
representation of our internal thoughts. In the same way that 3D objects cast 2D
shadows (and 4D, 3D)
communicating through language doesn't carry all of our cultural and
developmental context—transmitting all of that along with every phrase would be
impractical. Language is in this sense, lossily compressed thought.
Inert strings of words of ink and paper take a life of their own inside our
heads. It's why the exact same information can be interpreted completely
differently by different people.
Before language, fire and cooking technology allowed us to reallocate energy
usage from the digestive system to the brain by outsourcing digestion to outside
of our bodies, making macronutrients more efficiently absorbable. Almost all of
a cooked meal is metabolized by the body, whereas raw foods yield less than half
of their nutrients.
Cooking is an extension of our digestive system, and enabled us to develop
large, calorie-hungry brains. It also gave us time to think: our primate cousins
spend half of their days chewing raw food to consume enough calories to stay
alive.
Brains can be seen as survival machines, locked inside dark skulls, constantly
building a model of the outside world by predicting and learning through senses
and memory. The biological human brain evolved to have the necessary
sophistication to not only expertly navigate and understand the brute physical
reality but also to construct social reality. Democracy, religion, money: all
made up by us, for us.
We remember the past so that we can predict the future, and by doing so, we
thrive.
We create technology, which functions as a non-biological extra layer to our
brains and bodies, augmenting, complementing, and sometimes replacing our
natural capabilities.
The wheel… is an extension of the foot.
The book… is an extension of the eye…
Clothing, an extension of the skin…
Electric circuitry, an extension of the central nervous system.
To be human is to have the ability to change the world around us. The shift from
hunting and gathering to farming allowed us to spend less energy to acquire food
while giving us a predictable calorie supply.
The resulting food surplus made it possible for populations to settle down and
grow quickly while supporting people not being directly involved in the
production of food—before agriculture that was everyone's job. For one, it
allowed some to specialize and focus on developing better farming tools and more
resistant crops, starting a vicious cycle of improvement and consumption that
continues until today.
The transition from active foraging to a more sedentary lifestyle resulted in
worse health for the general population. The average farmer worked harder than
the average forager and got a worse diet in return. Our teeth, bones and joints
became more fragile, and we became afflicted by novel diseases coming from newly
domesticated animals, carriers of pathogens that incubated in our new densely
populated cities.
Owning land suddenly became really important. Agriculture and the concept of
private property reinforced each other and grew together, allowing us to create
value and secure the fruits of our labor. It also created the circumstances for
slavery to arise, and wars to be waged.
The groups of people growing the first crops could not have anticipated all of
the collateral effects of their breakthrough. They just wanted more food.
If the past has taught us anything is that we have to be mindful of the
consequences of our progress. In an increasingly connected world, change is
often nonlinear
and unpredictable. Cars didn't just replace horses—they forever changed the
entire outlook of every city. Did Tim Berners-Lee anticipate his invention
adding to forces pulling whole countries apart?
Our progress will continue to bring us previously unimaginable challenges.
Against an unknowable future, it doesn't hurt to keep improving our capabilities
to adapt and, more difficultly, to cooperate—especially at scale.
Computers are getting pretty good at driving cars—even in the most difficult
situations—and can already instantly diagnose some diseases better than human
doctors. Technology has a way to
reveal
the potential of our environment, and ourselves. We have to be careful not to
look at what surrounds us as mere raw materials to be consumed for the purposes
we conceive—sooner or later we'll start calling humans resources too...
It serves us well to leverage technology to give us time. Time to create and
enjoy art, follow the trail of our curiosities and passions, be fully present
with loved ones, or even just appreciate the freedom to idle and ponder about
the inconsequential—the stuff that seems to make us, us.
We are born with incomplete brains that get imbued with language and the
accumulated collective knowledge of our previous generation. Knowledge,
roughly
defined as a justified, true belief, doesn't fit the bill of much of the waves
of man-made information hitting the shores of our eyes and ears these days.
Acquiring it requires many things, and passively consuming content curated by
profit-maximizing algorithms is not one of them.
We attempt to transfer our gathered knowledge to machines and we specify the
rules for their learning, and by doing that we're inherently encoding our own
biases and limitations in algorithms that will be making life-altering choices.
Should your out-of-control self-driving car
automatically swerve to avoid
running over kids on the street, and by doing so put your own life at
considerable risk? Should you be able to opt-out of this behavior with a
checkbox?
We need to be careful about what we teach machines, and prevent them from making
the same mistakes we do, because they will do them orders of magnitude more
efficiently and at scale. Human judgment is both fallible and (still)
indispensable, especially when the stakes are higher.
Our learning machines already exhibit emergent behaviors that go beyond human understanding and could be interpreted as creativity, like AlphaGo's 37th move on the second match against Lee Sedol in 2016.
To be human is to have the ability to change oneself. Through open source and
hardware hacking, people with type 1 diabetes—who need to continuously measure
and manipulate their glucose and insulin levels to stay alive—took it upon
themselves to build an
artificial pancreas
and hook it up to their own bodies. The technology they created not only removed
an enormous cognitive burden from their lives but also decreased the likelihood
of physical complications and increased their lifespans.
What wouldn't you give to free up a large part of your brain processing
power and at the same time considerably improve all of your health
indicators?
Empowered by knowledge and technology, they didn't have to wait for the world
around them to change—they went ahead and changed it themselves. And in the
process, they changed their lives.
Arunachalam Muruganantham, who grew up in poverty and dropped out of school at
14 to support his single mother, also didn't wait for the world around him to
change. Going against conservative rural India—who treats sex education as
taboo—he provided underpriviledged women with
affordable sanitary pads by
creating a set of pad-producing machines. Poor menstrual hygiene cause women to
miss school, risk infections, and die from cervical cancer. The industry around
his invention provides women with income, gives them dignity, and saves their
lives.
Technology and the tools we create drastically accelerate our progress. Matt
Taylor compellingly puts it in perspective in
"Humanity 2.0". In it he presents
the chart below, which seems to show life's steady and even progress from
unicellular organisms to us, human beings, building
artificial suns, taking pictures of
black holes,
and unlocking the
mysteries of life itself.
The scale of this chart could be initially misleading: the visual distance
between the birth of life and the first eukaryotic cells—2 billion years—is
represented identically as the distance between the industrial revolution and
the personal computer—200 years.
The chart below tells the same story on a scale more easily digestible by us.
The mostly horizontal line depicts the slow process of biological evolution,
which eventually—out of only randomness and constraints—brought our neomammalian
brains into existence, kick-starting the journey of fire and language towards
modern civilization. It's been a long ride, and in the relative time scale of
biology our evolution through technology is happening fast and only seems to be
accelerating.
Each technological advance builds upon the last, creating a positive feedback
loop of progress.
We are inevitably shaped by what we create, and fundamentally driven by our
deeply human essence: the anticipation of discovery, the satisfaction of
attainment, and the joy in relationships we cultivate along the way.
The technology we create will survive us, and its impact will be unevenly
felt—the fruits of progress aren't unconditionally good.
We've come a long way
towards improving our lives, and we can still go so much further. There are so
many problems to solve.
In these unprecedented times of tremendous individual potential, it's good to
keep our values in check and constantly revisit the question: are we building
the right things?
History is a metaphorical pathway, and just like physical ones, it's built
purposefully by us, based on the topography and constraints of the environment.
Unlike physical ones, it sometimes takes us by surprise.
The future is still not determined, and "the best way to predict it is to invent
it". Informed by our knowledge and empowered by our technology, it is up to us
to lay the bricks.
* * *
Now, back to work. What was I doing again? Oh yeah, let's open that file.
Thanks to Evan Lezar, Fabrício Nascimento, Helder Ribeiro, John Wiegley and
Protesilaos Stavrou for reading and giving suggestions for drafts of parts of
this.
Special thanks to Bozhidar Batsov, Hunter McClelland and Thorsten Ball for
insightful, attentive and precise feedback.
As 2020 is almost running out, I have noticed that I didn't manage to post anything this year. So
here's a haphazard attempt to fix that.
Org-roam
I'd like to share a wonderful Emacs package I've started using this year:
org-roam. Even though in my Org notes I see that I
encountered this package in February 2020, I've actually started using it on December 10, trying to
figure out what Zettelkasten are and how to deal with
them.
Overall, 22 days isn't a whole lot of experience, and I'm looking forward to see how useful
Zettelkasten and org-roam will be to me in a year. But I already have written down 341 notes,
which amounts to around 15 notes per day.
In my previous/current system, plain-org-wiki, I
organize my knowledge into 209 tag categories. For instance, one of them, Linux, has 290
entries. Another, VPN, has 15 entries. I'll show an example of how org-roam improved my workflow
by showing a note I've added:
#+title: Use a custom DNS when resolving a specific host on Linux
On Linux, I have a DNS server that can resolve certain VPN-only websites. But the problem is
that it's only available if I'm connected to my work VPN. So I can't configure it in a common way,
since the common way would simply use that DNS always, whether or not it's up. The =dnsmasq=
package solves all these issues.
The code: local, and permalink
Above is one zettel - a note named "Use a custom DNS when resolving a specific host on Linux" that's
liked to two other notes: "Linux", and "DNS". Both of those notes are basically empty files that are
used for tagging purposes. I use M-xorg-roam-buffer-activate to see what other notes
are linked to them.
Suppose, instead of using org-roam, I wanted to add the above piece of knowledge to my old system
of tagged data. Does this information belong in Linux.org or in VPN.org? Moreover, do I really
want to add one more entry to Linux.org? It's getting really crowded with 290 entries already
there. So with the old system, it's the anxiety on where to put the note combined with the pressure
of keeping it short. Both things are solved by using Zettelkasten!
All of the functionality that I use is in the hydra below:
(defhydrahydra-org-roam(:exitt:idle0.8)"Launcher for `org-roam'."("i"org-roam-insert"insert")("f"ora-org-roam-find-file"find-file")("v"org-roam-buffer-activate"backlinks")("t"ora-roam-todo"todo"))
The only difference between ora-org-roam-find-file and org-roam-find-file is that my variant
supports ivy actions.
And ora-roam-todo is a small wrapper around this code, which gives me an overview of loose ends in
my notes that I'd like to follow up on:
What the above does technically: runs counsel-rg on my org-roam-directory looking for * TODO,
while sorting on the file modification date, instead of the default sorting by path. Here, C-c
C-o is the binding of ivy-occur, and M-> is end-of-buffer. And the whole code
overall produces an ivy-occur buffer without having to go into minibuffer and press C-c
C-o manually. The unread-command-events trick is a nice way to automate this.
Outro
I'd like to thank the creators of org-roam. Well done!
Happy New Year! I wish everyone good health, justice, equality, and happiness!
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
Emacs, relative to most software is old and has seen many
iterations. Recently there was a survey that was
conducted that helps us understand the
current state of the world for Emacs.
Let’s talk about how we can take what we know from the past and what
we know now to help develop Emacs to a brighter future. This
conversation can go in any number of directions and we will see where
the conversation runs its course.
On December 7th, 2020, we had a talk by Josh
Holbrook
Org mode, the task management and document markup system for Emacs,
includes a tool called Babel which may be used for literate
programming. In this talk I will explain literate programming,
discuss how Org mode and Babel enable it, and go over an example
using the slide deck itself. I will also cover some real-world
experiences writing literate programs in Emacs and the pros and cons
of doing so.
Josh has made his
slides
available, as well as their
source
I'm a little past the one year mark of using Emacs as my default text editor/IDE/super organizer, which means I have developed opinions about what I want out of a program (mainly one that will do ALL THE THINGS).
Unfortunately, until now when I've been working on C# projects, I've had to use either Visual Studio or VS Code, finding some emacs keybindings as a poor add and getting frustrated when said keybindings don't match up with my emacs config.
Well today, that problem has finally been SOLVED!
Here are the steps I followed:
First up, I installed an omnisharp server for emacs based on omnisharp-rosalyn. For the projects I'm currently working on, I'm using the .NET 3.1 SDK, but the omnisharp server should still work with multiple SDKs.
The final bit I needed was to install the nuget packages for my project, so that I'd be able to get all the code completion and syntax checkers for my project. Hypar's nuget package is still in beta, so it threw errors when I tried to install it using dotnet-mode, but once I provided the correct version number (hypar -v=0.0.1-beta6), it went through.
And now I can get code completion, build, and debug all from Emacs!
On November 2nd, 2020, we had a series of lightning talks by members of the community.
Raymond Puzio — emacs hypernotebooks
Raymond presented his notebook software that uses Emacs to bring
together multiple programs to make a single set of computations. It’s
able to pass data back and forth between these programs, generate
results, and even output publication-quality documents. Raymond
referenced a previous EmacsNYC talk Evan Misshula gave about
reproducible
research. Raymond will
be publishing his code soon.
Adrien Brochard – emacs user survey 2020
Adrien spoke about his user survey, aimed at finding out how the
community uses Emacs. You can find more information and fill it out at
https://emacssurvey.org/.
Qiantan Hong – crdt.el, a collaborative environment
Qiantan spoke about crdt.el, a project which enables collaboration
through shared editing sessions. Check it out
here. This project uses
algorithms from the Conflict-free Replicated Data Types family. A talk
about this can be viewed
here
Qiantan Hong – reflexive-music, an experimental music environment with Emacs as frontend
Qiantan presented reflexive-music, an environment to create music
through code. He ended his presentation with a live concert. He was
inspired by Ivan Wyshnegradsky, a Russian composer known for his
microtonal compositions. Contact Qiantan to encourage him to publish
his code!
Zachary Kanfer – Composing Electronic Music in Emacs
Zachary presented his solution to boredom during quarantine: his
software to compose looping music in Emacs. He had to dive into WAVE
files to create it. Find it at https://hg.sr.ht/~zck/zmusic.
In addition to our lightning talks, there are a few announcements from
members of the community.
The 2020 EmacsConf is happening on November 28 and 29. More
information at
https://emacsconf.org/2020. Raymond and
Zachary’s talks will be presented there.
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
Josh Holbrook github is a Staff Data Engineer at DoubleVerify and will be speaking about literate programming using org mode.
Org mode, the task management and document markup system for Emacs, includes a tool called Babel which may be used for literate programming. In this talk I will explain literate programming, discuss how Org mode and Babel enable it, and go over an example using the slide deck itself. I will also cover some real-world experiences writing literate programs in Emacs and the pros and cons of doing so.
After a bit of a false start last month we’re going to try again.
This month we are doing lightning talks!
We look forward to any talk you want to give that is Emacs or Emacs
adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 5 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
We had some success with our focused discussions and we’re going to try it again.
This time we’re going to step outside of Emacs and delve into something adjacent, software privacy. We’ll talk about best practices, what to look for, how to push back, and how to spread the word to others.
We’ll be watching a TED talk from Finn Lützow-Holm Myrstad(Norwegian Consumer Counsel): https://www.youtube.com/watch?v=4E_1AB1rsSw
Also please join us on freenode on the #emacsnyc channel to ask questions and keep the conversation going.
Like Emacs itself, GNU Hyperbole is an integrated, extensible,
self-documenting, and programmable hypertextual editing environment
delivered as a single ELPA package for quick installation and
evaluation. But where to start with such a large package?
This talk will provide a detailed, interactive overview of GNU
Hyperbole’s major capabilities and how they can speed knowledge
work, including:
Implicit, Explicit and Global Buttons for interlinking your
textual information regardless of type or mode;
Org Mode Integration that reduces the complexity of dealing with
Org constructs and lets you leverage Hyperbole in Org documents;
The Koutliner for rapid outlining with multi-level autonumbering
(like legal numbering), per outline heading/cell permanent
hyperlink anchors, and dynamic views that can be triggered by
links themselves;
HyRolo for fast contact or any hierarchical record management
including Org files or normal Emacs outlines;
HyControl for fast control over your Emacs windows and frames:
interactively increase or decrease your your face sizes, adjust
window sizes and layouts; replicate frame sizes and attributes
precisely; show what you want where you want it.
Whatever you like about Emacs you’ll likely find similar in Hyperbole.
Hyperbole grows with you as your knowledge and work complexity
increases. An hour invested in Hyperbole has the potential to save
you hundreds of hours in your future knowledge work. Come find out
about the magic and why its not all hyperbole.
Due to an unforeseen pandemic, our group has moved to be online
only. This is a story all too familiar to so many previously in-person
gatherings.
This is not a new move. We’ve been doing remote meetups since March
and had some success, but know that this could be better. We know that
we don’t need to keep things so local and we can support a community
for the rest of the world. This became even more apparent when we had
someone from India waking up at 6am to join us. So cool!
So with that we would like to make this attempt to expand to a larger
more inclusive community. This is our call for participants, whether
you wish to speak or participate in another way we would like to have
you.
I would encourage you to reach out directly and join us at our next
meetup. Your help and involvement can help us grow and make a more
vibrant and exciting community.
It is important that as our community does grow that we maintain a
safe and inclusive community that welcomes people from all
backgrounds. For those joining, please review our code of
conduct
A while back, setting up a runit service for an Emacs daemon. The idea behind that post was that you'd have a system-level service for your user Emacs session. But what if you want a "user service", like what systemd-using folks have? Read on to find out how to replicate this with runit!
We're excited to have you join us for EmacsNYC a group of dedicated
lambda enthusiasts that come together once a month to share our mutual
joy of a piece of software that's over 40 years old.
Whether you are first time user, long time contributor, software
developer, writer, or just curious what this is all about, you will
find an open and welcome community that is eager for you to be a
part.
To create an environment that is welcoming,
harrassment-free, and enjoyable to everyone, we have a
code-of-conduct
that we following for every get together.
A talk by the author, Bob Weiner
Like Emacs itself, GNU Hyperbole is an integrated, extensible,
self-documenting, and programmable hypertextual editing environment
delivered as a single ELPA package for quick installation and
evaluation. But where to start with such a large package?
This talk will provide a detailed, interactive overview of GNU
Hyperbole’s major capabilities and how they can speed knowledge
work, including:
Implicit, Explicit and Global Buttons for interlinking your
textual information regardless of type or mode;
Org Mode Integration that reduces the complexity of dealing with
Org constructs and lets you leverage Hyperbole in Org documents;
The Koutliner for rapid outlining with multi-level autonumbering
(like legal numbering), per outline heading/cell permanent
hyperlink anchors, and dynamic views that can be triggered by
links themselves;
HyRolo for fast contact or any hierarchical record management
including Org files or normal Emacs outlines;
HyControl for fast control over your Emacs windows and frames:
interactively increase or decrease your your face sizes, adjust
window sizes and layouts; replicate frame sizes and attributes
precisely; show what you want where you want it.
Whatever you like about Emacs you’ll likely find similar in Hyperbole.
Hyperbole grows with you as your knowledge and work complexity
increases. An hour invested in Hyperbole has the potential to save
you hundreds of hours in your future knowledge work. Come find out
about the magic and why its not all hyperbole.
We look forward to any talk you want to give that is Emacs or Emacs
adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 5 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
We’ll be meeting on https://meet.jit.si/EmacsNYC, a Free Software
video chat platform.
As a new experiment we’re going to try a focused discussion and maybe
try again next month.
For this one we’d like to talk about Software Freedom in Practice,
we’ll be using some content assistance coming from the EmacsConf 2019
and having a more focused discussion afterwards on what people have
done to accomplish their own software freedom and how they have sought
that out.
Please feel free to check out the video here by Greg Farough: GNU
Emacs as Software Freedom in Practice:
Tomorrow will be the third event we’ve had since the global pandemic
started, it also marks the third event that has been entirely online.
I would be lying if I said this has been a smooth transition, but that
isn’t to say that we haven’t had some great things to come from
this. It turns out when you’re online people can join from anywhere in
the world! Who knew!?
During this transition we’ve have had to think about how we can
maintain the most free(freedom) software and tooling we can along the
way to continue in the spirit of what our group represents.
The Free Software Foundation has a great resource for communicating
during these times and something we’ve pulled a lot of inspiration
from. We currently run all of our events using Jitsi and starting next
event will encourage everyone to join us on #emacsnyc freenode channel.
During our last event, we had some success leading a discussion on how
people have been coping with working/not working from home. Talking
about how they’ve setup their workspace, do they do specific things to
make sure they are getting some sunshine/exercise/socialization, and
other considerations. It turned into some great discussions and even
led to a bit of debate.
This next month we intend to try a similar discussion based event. We
also lean into the benefits that we gain from being pushed online. The
value of getting onto IRC and having platform that can connect to
anyone means there is no reason our group cannot extend beyond New York.
Our hope is that even when(if?) we are able to have meetups in person
we can maintain this moment and encourage people to join us and
develop a hybrid group that allows for a base in New York, but welcomes
people around the globe to participate.
Each of my Emacs packages has a Makefile to byte-compile all source
files, run the tests, build a package file, and, in some cases, run the
package in an interactive, temporary, isolated Emacs instance. These
portable Makefiles have a similar structure and follow the same
conventions. It would require more thought and feedback before I’d try
to make it a standard, but these are conventions I’d like to see in
other package Makefiles.
You should make a habit of compiling your Emacs Lisp files even if you
don’t think you need the performance. The byte-compiler, while
dumb, does static analysis and may spot bugs and other
issues early.
First things first: Every portable Makefile starts with a special
target, .POSIX, to request standard behavior. This is followed by
macro definitions. When compiling a C program, the CC macro is the
name of the compiler. Analogously, when compiling Emacs packages the
EMACS macro is the name of the Emacs program.
.POSIX:EMACS= emacs
Users can now override the macro to specify alternate Emacs binaries. I
use this all the time to test my packages under different versions of
Emacs.
$ make clean
$ make EMACS=emacs-24.3 check
$ make clean
$ make EMACS=emacs-25.1 check
Note: It’s common to use ?= assignment here, but that is both
non-standard and unnecessary. If you want to override macro definitions
from the environment, use the -e option:
$ export EMACS=emacs-24.3
$ make -e
The first non-special target in the Makefile is the default target. For
Emacs packages, this target should byte-compile all the source files,
including tests. List the byte-compiled file names as the target
dependencies:
compile:foo.elc foo-test.elc
Now for the tedious part: Define the dependencies between your different
source files. It would be nice to automate this part somehow, but
fortunately most packages just aren’t that complicated. You do not need
to list trivial dependencies — i.e. mapping each .el file to its .elc
file — since make will figure that out on its own.
Since foo-test.elc relies on foo.elc — it’s testing this file after
all — the relationship must be indicated to make. For single file
packages (one package file, one test file), this is all that’s needed:
foo-test.elc:foo.elc
I call my testing targets “check” and this target must depend on the
byte-compiled files containing tests. It will transiently depend on the
other package source files because of the previous section.
The -Q option runs Emacs with “minimum customizations.” The -L .
option puts the current directory in the load path so that (require
'foo) will work. Finally it loads the file containing the tests and
instructs ERT to run all defined tests.
A good build can clean up after itself:
clean:rm-ffoo.elcfoo-test.elc
Finally we need one more thing to tie it all together: an inference rule
to teach make how to compile .elc files from .el files.
This is similar to the “check” target, but compiles a source file
instead of running tests.
For simple, single source file packages, this is all you need!
Complex packages
My most complex package is Elfeed which has 10 source files and 4 test
files. It also includes a target to build a package file, which I would
upload to Marmalade when it was still functioning. I did a few extra
things to keep this tidy.
First, I define the package version in the Makefile:
VERSION= 1.2.3
It would be nice to grab this information from a reliable place (Git
tag, source file, etc.), but I never found a reliable and satisfactory
way to do this. Simple wins.
To avoid repeating myself, I list the source files in a macro as well:
These will still need to have all their interdependencies individually
defined for make. For example, if C depends on both A and B, but neither
A nor B depend on each other, this is all you’d need:
foo-c.elc:foo-a.elc foo-b.elc
Done correctly you can perform parallel builds with the non-standard but
common -j make option. This is pretty nice since Emacs can’t do
parallel builds itself.
I use the file list macros in the “compile” and “check” targets:
The “package” target copies everything under a directory and tars it up.
The directory is removed first, if it exists, so that any potenntial
leftover garbage from doesn’t get included.
In Elfeed, the target to test in an interactive, temporary Emacs
instance is called “virtual”. In Skewer it’s called “run”. The name of
the target and the specific rules will depend on the package, should you
even want this target at all. It’s handy to have the option test without
my own configuration contaminating Emacs, and vice versa. When people
report issues, I can also direct them to reproduce their issue in the
clean environment.
Here’s what a simple “run” target might look like:
Make is not really designed to run interactive programs like this, but
it works in practice.
Dependencies
What about packages with dependencies? I’ve used Cask in the
past but was never satisfied, especially when integrating it into a
Makefile. So, again, I’ve opted for the dumb-but-reliable option:
request that dependencies are cloned in adjacent directories matching
the dependency’s package name. For example, the EmacSQL Makefile
header:
# Clone the dependencies of this package in sibling directories:
# $ git clone https://github.com/cbbrowne/pg.el ../pg
I also define a new “linker flags” macro, LDFLAGS. Like with EMACS,
this lets users override it if needed:
LDFLAGS=-L ../pg
Everywhere I use -L . I also include $(LDFLAGS). For example, in the
inference rule:
If the dependencies follow these conventions, then these can also be
compiled in a recursive way with little effort:
$ make -C ../pg
I’m not completely satisfied with this solution, particularly since it’s
an odd burden on anyone using the Makefile, but it’s worked well enough
for my needs. This is when I wish Emacs had distributed package
management.
It's been nearly 3 months since I've "never graduated" from the Recurse Center, which has given me some time to reflect on the experience.
I have a complicated history with programming. I never took a programming class in high school, and when I got to college, I was completely lost in the Intro to Programming with Java course required for my civil engineering degree. It was one of the largest classes I took. Half the class had already taken AP Computer Science and were using this class as a "gimmie". I did okay after having my sister spend several late nights explaining to me how arrays worked using flower pots and marbles, but the class just really didn't "click". In my other engineering courses, we used Matlab, which was dissimilar enough from Java that I never really got the hang of object oriented programming, and never had to revisit what I'd done in that single CS course.
All that changed when I started working as a structural engineer, and realized that most of the initial tasks I would be doing consisted of pulling data from a model into Excel, reformatting it to a specific shape, and then pasting it in another spreadsheet. I balked hard at the prospect of filling my days with that and taught myself VBA. When I got assigned to check shop drawings (looking at drawings and checking for compliance all day), I revisited a text-based interface FEA program used for the structural analysis. It wasn't really its own programming language, but seeing the input files written and revised in Excel line by line using concatenation pushed me to explore Python, and I started going to PyLadies meetups in NYC, where someone very kindly taught me about homebrew and package management and got me set up for my first incursion into Python.
When I considered PhD programs, I wanted one that would let me further develop my programming skills, so I chose a project on computational design of flexible structures, with an explicit focus on numerical simulation and design. There are many things I wish I had known prior to this endeavor, but my advisor was going through a period of major restructuring with his research group, and the experience was not what I signed on for. I hadn't been prepared for how cutthroat academia could be even in the same department and in the same group (having your supervising post-doc write and publish a paper about your topic without telling you beforehand is ... something else).
The problems I wanted to solve were often dismissed as "trivial" or subject to armchair explanation with little discussion, followup or resolution. Since the project was computationally heavy, I grew to loathe programming as a result of hearing on a regular basis: "Well X could code that up in a few hours, I don't know why you're having problems with it", which only eroded my enthusiasm and confidence. The research environment was both fratty and intense; any unhappiness with your project was solely your fault, since you were so fortunate and lucky to be part of this exclusive group.
I didn't leave as soon as I should have, and by the time I did, I had very little faith in my ability as an engineer and a programmer. I took the first job I was offered with a company that I had previously interned which had a nascent computational design group. However, I was hired as a structural engineer, not a software developer. This was a jump from the frying pan into the fire, leaving one toxic workplace for another toxic mega-project. I was sucked into fixing Grasshopper (visual programming) scripts and VBA headaches. In hindsight, this was almost a blessing, because working with bad programming made me want to work on good programming again. To find an outlet from 60 hour weeks, I started going back to the PyLadies meetups. Thanks to their monthly study groups, I started to enjoy programming again and built a few small Python programs with an Excel interface to use at work.
I first heard about the Recurse Center in December 2018, thanks to a tweet a friend forwarded, and pretty much immediately dismissed it. I felt very underqualified after looking through some of the people on Twitter with @recursecenter in their profile. Still, one of the things that struck me the most after looking through the website and application were the RC Social Rules.
No well-actually's
No feigned surprise
No backseat driving
No subtle-isms
Unusual yes, but I've found that unspoken and implied social norms don't necessarily lead to internal group harmony, as people have very different ideas of acceptable language and behavior. If my time as a researcher had included these rules, I wondered how much more I would have learned (and if I would have stayed).
Moving into the spring of 2019, my programming endeavors were not successful at the office. Despite writing documentation to help pass internal QA/QC, I was explicitly told by my PM that programming was not my job and should be left to the small computational group that had no practicing engineers and very rarely interfaced with the structures team. In a fit of pique, I started putting together my RC application, figuring that if I'd never have time to learn and improve on the job, maybe taking a step back and really focusing on programming was what I needed. My husband and I were already planning on moving to the west coast, so I figured that this was the right time to apply, because the worst thing that could happen is that I wouldn't get admitted.
I got my acceptance to RC exactly two weeks from the start of the Fall 1 Batch, and gave my notice at the office the next day. Most people assumed I was going to a "bootcamp", and I just got tired of correcting them and gave up, since people seem to have a hard time grokking the concept of self study.
When I got to RC, I was one of the (many!) people on the first day to raise their hands to say they didn't feel like they were supposed to be there. There were people that had been programming for 10 or 20 years at major tech companies, multiple startups with huge resumes. But the amazing thing about RC is that there is no intimidation factor. Anyone can (and will!) ask questions; everyone is there to learn. The Recurse Center has created a unique environment for curiosity to flourish and brings together people that genuinely enjoy learning AND teaching. Its environment fosters collaboration through structured and unstructured activities you can opt in on. While there are regular coffee chats and pair programming facilitated by bots, people host impromptu workshops on whatever they want, and form study groups for whatever people are interested on at the time (while I was there, some of the active study groups included category theory, machine learning, GLSL, and Haskell).
Here's a list of just a few of the things I worked on at RC:
Learning React
Learning D3
Starting functional programming in Haskell (I didn't know Haskell existed as a programming language prior to RC!)
Learning and using Cython
Doing a deep dive with Python intricacies
Building web applications with Dash and Plotly
Acting as a rubber duck for other's Javascript problems
Contact juggling!
Giving a technical talk on parsing a terrible conference website
Giving a localhost talk about structural connections and why they're important and why designing them is a PITA
Giving nontechnical talks on footbridge design, kittens, and knitting
Mastering emacs and the joys of org-mode (an ongoing endeavor)
One of the most valuable lessons I learned in the first half of my batch (and then applied in the second half) was to ask more people what they were working on. It meant I didn't accomplish as much on my own projects, but I got a much greater breadth of knowledge that I can now use to connect the dots moving forward. RC also helped me overcome lingering issues I had with giving presentations, as well as giving me the confidence to say that I am a software developer.
My experience is not the experience other people have, because RC is whatever you make of it, it just gives you the time and space to pick a path. RC also gave me the opportunity to meet people from all kinds of backgrounds and programming experiences, which I think was even more valuable than networking with the computational people that are already in the AEC industry. Because software is such a niche part of the industry, the main players are well established and pretty inaccessible. I've just not been at the right conferences or in the right circles to make those kinds of connections, especially after leaving the computational research field.
My Recurse Center experience was everything I'd hoped to get out of my research experience and more. Having the time to sit down and write software was a huge boost for my resume and for my job interviews, as I could finally develop ideas I'd had bouncing around my head for the past year. After Recurse Center, I was able to find a job with one of the few firms in the built environment that has an extensive open source code base. Even better, I didn't have to choose between structural engineering and programming, and for now at least I'll be able to pursue them in parallel. Though my time at RC is over (for now!), the motto of "Never graduate" is one I'll take with me going forward. I can't thank the faculty and fellow Recursers enough for making my Recurse experience so great and memorable.
Suppose you don’t like the names car and cdr, the traditional
identifiers for two halves of a lisp cons cell. This is
misguided. A cons is really just a 2-tuple, and the halves
don’t have any particular meaning on their own, even as “head” and
“tail.” However, maybe this is really important to you so you want to
do it anyway. What’s the best way to go about it?
defalias
Emacs Lisp has a built-in function just for this, defalias, which
is the obvious choice.
(defalias'car-alias#'car)
The car built-in function is so fundamental to the language that it
gets its own byte-code opcode. When you call car in your code,
the byte-compiler doesn’t generate a function call, but instead uses a
single instruction. For example, here’s an add function that sums
the car of its two arguments. I’ve followed the definition with its
disassembly (Emacs 26.3, lexical scope):
There are zero function calls because of the dedicated car opcode, and
it has the optimal six byte-code instructions.
The problem with defalias is that the definition is permitted change
— or be advised — and that robs the byte-compiler of
optimization opportunities. It’s a constraint. When the
byte-code compiler sees car-alias, it must emit a function call:
This has two function calls and eight byte-code instructions. Those
function calls are significantly more expensive than a car
instruction, which will show in the benchmark later.
defsubst
An alternative is defsubst, an inlined function definition, which
will inline an actual car. The semantics for defsubst are, like
macros, explicit that re-definitions may not affect previous uses, so
the constraint is gone. Unfortunately the byte-code compiler is
pretty dumb, and does a poor job inlining car-subst.
There are zero function calls and ten byte-code instructions. The
car opcode is in use, but there are five unnecessary instructions.
This is still faster than making the function calls, though. If the
byte-code compiler was just a little smarter and could compile this to
the ideal case, then this would be the end of the discussion.
cl-first
The built-in cl-lib package has a cl-first alias for car. This was
written by someone with intimate knowledge of Emacs Lisp, so how how
well did they do?
They used defalias, but they also manually told the byte-compiler to
inline the definition like defsubst. In fact, defsubst expands to an
expression that sets byte-compile-inline-expand, but, as seen above,
the inline function overhead gets inlined and doesn’t get eliminated.
(The car of the list is the running time.) Since add and
add-cl-first have the same byte-codes, we shouldn’t, and didn’t, see
a significant difference. The simple use of defalias doubles the
running time, and using defsubst is about 15% slower.
I regularly make throwaway “projects” and do a surprising amount of
programming in /tmp. For Emacs Lisp, the equivalent is the
*scratch* buffer. These are places where I can make a mess, and the
mess usually gets cleaned up before it becomes a problem. A lot of my
established projects (ex.) start out in volatile storage and
only graduate to more permanent storage once the concept has proven
itself.
Throughout my whole career, this sort of throwaway experimentation has
been an important part of my personal growth, and I try to encourage it
in others. Even if the idea I’m trying doesn’t pan out, I usually
learn something new, and occasionally it translates into an article here.
I also enjoy small programming challenges. One of the most abused
tools in my mental toolbox is the Monte Carlo method, and I readily
apply it to solve toy problems. Even beyond this, random number
generators are frequently a useful tool (1, 2), so I
find myself reaching for one all the time.
Nearly every programming language comes with a pseudo-random number
generation function or library. Unfortunately the language’s standard
PRNG is usually a poor choice (C, C++, C#, Go).
It’s probably mediocre quality, slower than it needs to be
(also), lacks reliable semantics or behavior between
implementations, or is missing some other property I want. So I’ve
long been a fan of BYOPRNG: Bring Your Own Pseudo-random Number
Generator. Just embed a generator with the desired properties directly
into the program. The best non-cryptographic PRNGs today are
tiny and exceptionally friendly to embedding. Though, depending on what
you’re doing, you might need to be creative about seeding.
Crafting a PRNG
On occasion I don’t have an established, embeddable PRNG in reach, and
I have yet to commit xoshiro256** to memory. Or maybe I want to use
a totally unique PRNG for a particular project. In these cases I make
one up. With just a bit of know-how it’s not too difficult.
Probably the easiest decent PRNG to code from scratch is the venerable
Linear Congruential Generator (LCG). It’s a simple recurrence
relation:
x[1] = (x[0] * A + C) % M
That’s trivial to remember once you know the details. You only need to
choose appropriate values for A, C, and M. Done correctly, it
will be a full-period generator — a generator that visits a
permutation of each of the numbers between 0 and M - 1. The seed —
the value of x[0] — is chooses a starting position in this (looping)
permutation.
M has a natural, obvious choice: a power of two matching the range of
operands, such as 2^32 or 2^64. With this the modulo operation is free
as a natural side effect of the computer architecture.
Choosing C also isn’t difficult. It must be co-prime with M, and
since M is a power of two, any odd number is valid. Even 1. In
theory choosing a small value like 1 is faster since the compiler
won’t need to embed a large integer in the code, but this difference
doesn’t show up in any micro-benchmarks I tried. If you want a cool,
unique generator, then choose a large random integer. More on that
below.
The tricky value is A, and getting it right is the linchpin of the
whole LCG. It must be coprime with M (i.e. not even), and, for a
full-period generator, A-1 must be divisible by four. For better
results, A-1 should not be divisible by 8. A good choice is a prime
number that satisfies these properties.
If your operands are 64-bit integers, or larger, how are you going to
generate a prime number?
Primes from Emacs Calc
Emacs Calc can solve this problem. I’ve noted before how
featureful it is. It has arbitrary precision, random number
generation, and primality testing. It’s everything we need to choose
A. (In fact, this is nearly identical to the process I used to
implement RSA.) For this example I’m going to generate a 64-bit
LCG for the C programming language, but it’s easy to use whatever
width you like and mostly whatever language you like. If you wanted a
minimal standard 128-bit LCG, this will still work.
Start by opening up Calc with M-x calc, then:
Push 2 on the stack
Push 64 on the stack
Press ^, computing 2^64 and pushing it on the stack
Press k r to generate a random number in this range
Press d r 16 to switch to hexadecimal display
Press k n to find the next prime following the random value
Repeat step 6 until you get a number that ends with 5 or D
Press k p a few times to avoid false positives.
What’s left on the stack is your A! If you want a random value for
C, you can follow a similar process. Heck, make it prime, too!
The reason for using hexadecimal (step 5) and looking for 5 or D
(step 7) is that such numbers satisfy both of the important properties
for A-1.
Calc doesn’t try to factor your random integer. Instead it uses the
Miller–Rabin primality test, a probabilistic test that, itself,
requires random numbers. It has false positives but no false negatives.
The false positives can be mitigated by repeating the test multiple
times, hence step 8.
Trying this all out right now, I got this implementation (in C):
However, we can still do a little better. Outputting the entire state
doesn’t have great results, so instead it’s better to create a
truncated LCG and only return some portion of the most significant
bits.
This won’t quite pass BigCrush in 64-bit form, but the results
are pretty reasonable for most purposes.
But we can still do better without needing to remember much more than
this.
Appending permutation
A Permuted Congruential Generator (PCG) is really just a
truncated LCG with a permutation applied to its output. Like LCGs
themselves, there are arbitrarily many variations. The “official”
implementation has a data-dependent shift, for which I can
never remember the details. Fortunately a couple of simple, easy to
remember transformations is sufficient. Basically anything I used
while prospecting for hash functions. I love xorshifts, so
lets add one of those:
This is a big improvement, but it still fails one BigCrush test. As
they say, when xorshift isn’t enough, use xorshift-multiply! Below I
generated a 32-bit prime for the multiply, but any odd integer is a
valid permutation.
This passes BigCrush, and I can reliably build a new one entirely from
scratch using Calc any time I need it.
Bonus: Adapting to other languages
Sometimes it’s not so straightforward to adapt this technique to other
languages. For example, JavaScript has limited support for 32-bit
integer operations (enough for a poor 32-bit LCG) and no 64-bit
integer operations. Though BigInt is now a thing, and should
make a great 96- or 128-bit LCG easy to build.
Java doesn’t have unsigned integers, so how could you build the above
PCG in Java? Easy! First, remember is that Java has two’s complement
semantics, including wrap around, and that two’s complement doesn’t
care about unsigned or signed for multiplication (or addition, or
subtraction). The result is identical. Second, the oft-forgotten >>>
operator does an unsigned right shift. With these two tips:
Ivy is a completion method that's
similar to Ido, but with emphasis on simplicity and customizability.
Overview
The current release constitutes of 183 commits and 3 months of
progress since 0.12.0. Many issues ranging from
#2153 to
#2278 were fixed.
The number of people who contributed code as grown to form 148 to
160. Thanks,
everyone!
Details on changes
Changelog.org has been a part of the repository since 0.6.0, you
can get the details of the current and past changes:
Many improvements are incremental and don't require any extra code to
enable. I'll go over a few selected features that require a bit of
information to make a good use of them.
New bindings
counsel-find-file
~~ to move to the local home directory from remote or /sudo::.
/ RET ~ achieves the same thing, but is longer.
M-o R calls find-file-read-only.
counsel-git-grep
C-x C-d to switch the current directory.
swiper-isearch
M-o w to copy the current line and go back to where you started.
New features
counsel-package
The idea of counsel-package is to install and remove packages with a single binding:
(global-set-key(kbd"C-c P")'counsel-package)
But if the package contents weren't up to date, a separate M-xpackage-refresh-contents
had to be triggered, which was an annoying overhead. Now counsel-package will look at the time
stamps of the appropriate archive-contents files, and call package-refresh-contents if the
timestamp is outdated by more than 4 hours.
counsel-M-x
Some commands are intended to be called only via their key binding. Make them disappear
from counsel-M-x like this:
(put'counsel-find-symbol'no-counsel-M-xt)
counsel-rg
The default setting of ivy-case-fold-search-default is 'auto, which means:
the input "emacs" matches "emacs", "EMACS", and "Emacs"
the input "Emacs" matches only "Emacs"
This now also applies to counsel-rg: Ivy will pass the -i flag to ripgrep appropriately, based
on ivy-case-fold-search-default. You should remove the -S flag from counsel-rg-base-command if
you customized it.
ivy-update-candidates
This is a new API for asynchronous calls. To use it, pass to ivy-read: :dynamic-collection t,
and a collection function that takes a user input string, starts some asynchronous process based on
that input, and returns 0. The 0 return result tells Ivy that no candidates were returned;
instead, ivy-update-candidates is used in the async callback.
See counsel-google for a reference implementation.
ivy-use-virtual-buffers
You can now choose between: recent files, or bookmarks, or both, or none. Don't forget that
counsel-set-variable makes it very easy to set customization options.
New Commands
I have put these separately so they don't get lost in the crowd. Be sure to try them out.
counsel-buffer-or-recentf - list buffers visiting files (highlighted) then the recentf file
list.
counsel-fonts - show a list of all supported font families for a particular frame.
counsel-google - asynchronously query the Google predictive search API.
counsel-major - switch major-mode.
counsel-slime-repl-history - browse SLIME REPL history.
Outro
Again, thanks to all the contributors. Happy hacking!
PS. Thanks to everyone who supports me on Liberapay and
Patreon!
I am now also on Github sponsors, which is an
interesting new effort by Github. The cool thing is that's more integrated with Github, there are
less transaction fees, and Github matches every donation for up to $5000 for a whole year.
Please consider joining there, since every $1 per month that you donate is doubled by Github.
Ivy is a completion method that's
similar to Ido, but with emphasis on simplicity and customizability.
Overview
The current release constitutes of 398 commits and 6 months of
progress since 0.11.0. Many issues ranging from
#1904 to
#2151 were fixed.
The number of people who contributed code as grown to
148. Thanks,
everyone!
Details on changes
Changelog.org has been a part of the repository since 0.6.0, you
can get the details of the current and past changes:
Many improvements are incremental and don't require any extra code to
enable. I'll go over a few selected features that require a bit of
information to make a good use of them.
New bindings
counsel-descbinds
M-o x execute action.
counsel-file-jump
M-o d dired.
counsel-find-file
M-o c copy file.
` bookmarks: efficiently jump between recent directories.
$ directories stored in environment variables.
C-DEL go up directory. Customize: counsel-up-directory-level.
RET open file. Customize: counsel-find-file-extern-extensions.
// when on remote, cd to remote root.
/ C-j select local root.
~ when on remote, cd to remote home.
/ C-j ~ cd to local home from remote.
counsel-git-log
M-o v open the current commit in magit.
counsel-rg
C-x C-d change the current directory for grep.
ivy-avy
C-v to scroll down.
M-v to scroll up.
ivy-readC-o
m mark and move down.
u unmark and move down.
DEL move up and unmark.
t toggle marks.
d perform the action on all marked elements.
ivy-switch-buffer
C-k kill buffer.
M-o x open buffer file externally.
ivy-reverse-i-search
C-k remove item from the history.
New Commands extensions
These commands are new variants and adaptations of existing commands.
Thing at point variants:
swiper-all-thing-at-point.
swiper-isearch-thing-at-point.
swiper-thing-at-point.
Search variants that go backwards:
swiper-backward.
counsel-grep-or-swiper-backward.
swiper-isearch-backward.
A variant of ivy-switch-buffer with live preview:
counsel-switch-buffer.
counsel-switch-buffer-other-window.
And finally:
counsel-dired - like counsel-find-file, but open dired.
swiper-isearch-toggle - toggle between swiper and isearch.
New Commands
I have put these separately so they don't get lost in the crowd. Be sure to try them out.
counsel-compile - completion for compile.
counsel-register - completion for registers.
counsel-minor - completion for minor modes.
swiper-isearch - a faster swiper that's not line-based.
Outro
Again, thanks to all the contributors. Happy hacking!
I'm sure many are aware of the C-r functionality in bash (a whole lot of Emacs
bindings are valid in bash and do the same thing). I also like the quirky Emacs-style prompt, that
uses a backquote to open and a straight quote to close a your quoted input:
So when you want to cd to somewhere where you were before you do C-rcd. And then the
C-r "roulette" begins: you keep pressing C-r over and over, in the hopes to find
what you were looking for.
Getting better history completion with Ivy
Ivy improves the "roulette" situation in two ways:
You get an overview of the matching candidates and their count,
You can quickly narrow down the candidates with fuzzy matching.
Here's the basic setup to enable C-r completion using ivy:
The first key binding is also part of counsel-mode, while the second needs to be set up separately,
after shell-mode was loaded.
And here's how counsel-shell-history looks like:
The cool thing is that ivy-reverse-i-search applies to any Ivy command, not just for shell
command completion. I find it especially useful for:
counsel-find-file
eval-expression
Recent improvement: delete history items
While searching with regexes is great, it's not so great when the old stuff that we won't ever need
gets in the way. So now you can delete it with C-k. Since C-k also has to
serve as ivy-kill-line, the history deleting behavior will only come into effect when the point is
at the end of the line (so that ivy-kill-line does not make sense anyway).
This way, your typos don't stay around to haunt you.
Outro
I hope you'll find ivy-reverse-i-search a useful boost to your completion needs. Happy hacking!
When Ivy just started out as a completion framework, the
functionality was supposed to be simple: select one string from a list of strings. The UI is simple
enough:
Show the list of strings that match the entered text,
Use C-n and C-p to navigate them,
Use C-m or C-j to submit.
Emacs has three key bindings that mean "Enter": RET, C-m, and
C-j. But in terminal mode, emacs -nw, RET and C-m are the same
binding: Emacs can't distinguish them. So we have at most two bindings. Fortunately, the world of
completion is simple at the moment, and we only need one binding.
File name completion
Enter file name completion. When you're completing file names, you are selecting not one string, but
many strings in succession while moving from one directory to the next. So we need at least two key
bindings:
Use C-m (ivy-done) to select the current candidate and exit completion.
Use C-j (ivy-alt-done) to change the current directory to the current candidate without exiting completion.
What to do when C-j is used on a file and not on a directory? Might as well open the
file: same action as C-m. OK, we had two key bindings, and we have used them. Hopefully
nothing else comes up.
Enter creating a new file, i.e. selecting something that's not on the list of strings.
Suppose I call find-file, enter "do", and the only match is a directory named "doc":
Pressing C-m will end completion with the "doc" directory selected.
Pressing C-j will continue completion inside the "doc" directory.
So creating a file named "do" is the third action. Our two "Enter" keybindings are already taken by
the first two different useful actions. So we need a third key binding. The one I chose is
C-M-j (ivy-immediate-done). It means: I don't care that the current input is not on
the list of candidate strings, submit it anyway.
Directory creation
Enter directory creation: dired-create-directory and make-directory. These built-in Emacs
commands request file name completion, but what they tell Ivy is no different from what find-file
tells: "I want to select a file". However, for these commands, the C-M-j action is the
one that makes most sense. Here it would be nice for Ivy to take the back seat and just act like an
interactive ls, since the user will enter a completely new string that's not on the list of
candidates.
For a long time, you still had to use C-M-j with those commands, to much frustration of
new but also existing users, including myself. But a few days ago, I looked at the prompt that
dired-create-directory uses: "Create directory: ". That prompt is passed to Ivy. Using the prompt
to detect the intention of the command is a bit of a hack, but I think in this case it's
justifiable. So now Ivy will recognize that the intention commands that request file name completion
and pass the "Create directory: " prompt is to create a directory, and all key bindings will do just
that: C-m, C-j, and C-M-j will behave the same in this case.
An alternative key binding scheme
Note that C-m and C-j behave differently only for directories. But thanks to
the fact that "." is always the first candidate, C-m for directories is equivalent to
C-j C-j. So we can get away with just using ivy-alt-done, and bind C-m to
ivy-immediate-done. Or swap the two meanings:
The price to pay here is the extra context switch when we simply want to select a directory. We
could the bind ivy-done to C-M-j and avoid the context switch, but then we're back to
three bindings once more. Still, I thought that swapping the bindings is an interesting idea worth
sharing.
Canceling dired-dwim-target
Setting dired-dwim-target is a nice productivity boost. It allows to use Emacs in a similar way to
a two-pane file explorer, like mc(1). But it was really annoying when I was in dir-1 with the
intention to copy a file to a different name in dir-1 (e.g. create a backup, or copy a template),
but the current directory was set to dir-2 because of a random dired window I had open. In that
case, I had to call delete-other-windows, perform the copy, and then restore the window
configuration with winner-undo.
I did the above many times over many years, until I finally dug into the code of dired.el to see
how dired-dwim-target worked. Turns out it was storing dir-1 in the minibuffer-defaults
variable. So now Ivy will use that variable when I press M-n. All in all, it was a five
minute fix. But rather than regret that I didn't do it years ago, I'm glad I did it now. It only
remains to build some muscle memory to press M-n in that situation.
I'm guesstimating that dired-dwim-target works to my advantage 90% of the time when I press
C in dired. For the other 10% of the times, I can now press M-n.
Outro
I hope you find the new functionality useful. I'm always open to new ideas and pull
requests. Happy hacking!
When designing or, in some cases, implementing a programming language
with built-in support for Unicode strings, an important decision must be
made about how to represent or encode those strings in memory. Not all
representations are equal, and there are trade-offs between different
choices.
One issue to consider is that strings typically feature random access
indexing of code points with a time complexity resembling constant
time (O(1)). However, not all string representations actually
support this well. Strings using variable length encoding, such as
UTF-8 or UTF-16, have O(n) time complexity indexing, ignoring
special cases (discussed below). The most obvious choice to achieve
O(1) time complexity — an array of 32-bit values, as in UCS-4 —
makes very inefficient use of memory, especially with typical strings.
Despite this, UTF-8 is still chosen in a number of programming
languages, or at least in their implementations. In this article I’ll
discuss three examples — Emacs Lisp, Julia, and Go — and how each takes a
slightly different approach.
Emacs Lisp
Emacs Lisp has two different types of strings that generally can be used
interchangeably: unibyte and multibyte. In fact, the difference
between them is so subtle that I bet that most people writing Emacs Lisp
don’t even realize there are two kinds of strings.
Emacs Lisp uses UTF-8 internally to encode all “multibyte” strings and
buffers. To fully support arbitrary sequences of bytes in the files
being edited, Emacs uses its own extension of Unicode to
precisely and unambiguously represent raw bytes intermixed with text.
Any arbitrary sequence of bytes can be decoded into Emacs’ internal
representation, then losslessly re-encoded back into the exact same
sequence of bytes.
Unibyte strings and buffers are really just byte-strings. In practice,
they’re essentially ISO/IEC 8859-1, a.k.a. Latin-1. It’s a Unicode
string where all code points are below 256. Emacs prefers the smallest
and simplest string representation when possible, similar to CPython
3.3+.
(multibyte-string-p"hello");; => nil(multibyte-string-p"π ≈ 3.14");; => t
Emacs Lisp strings are mutable, and therein lies the kicker: As soon as
you insert a code point above 255, Emacs quietly converts the string to
multibyte.
(defvarfish"fish")(multibyte-string-pfish);; => nil(setf(areffish2)?ŝ(areffish3)?o)fish;; => "fiŝo"(multibyte-string-pfish);; => t
Constant time indexing into unibyte strings is straightforward, and
Emacs does the obvious thing when indexing into unibyte strings. It
helps that most strings in Emacs are probably unibyte, even when the
user isn’t working in English.
Most buffers are multibyte, even if those buffers are generally just
ASCII. Since Emacs uses gap buffers it generally doesn’t matter:
Nearly all accesses are tightly clustered around the point, so O(n)
indexing doesn’t often matter.
That leaves multibyte strings. Consider these idioms for iterating
across a string in Emacs Lisp:
The latter expands into essentially the same as the former: An
incrementing index that uses aref to index to that code point. So is
iterating over a multibyte string — a common operation — an O(n^2)
operation?
The good news is that, at least in this case, no! It’s essentially just
as efficient as iterating over a unibyte string. Before going over why,
consider this little puzzle. Here’s a little string comparison function
that compares two strings a code point at a time, returning their first
difference:
With using two, zeroed unibyte strings it takes 13ms. How about changing
the last code point in one of them to 256, converting it to a multibyte
string:
That took 2.3 seconds: about 2000x longer to run! Iterating over two
multibyte strings concurrently seems to have broken an optimization.
Can you reason about what’s happened?
To avoid the O(n) cost on this common indexing operating, Emacs keeps
a “bookmark” for the last indexing location into a multibyte string.
If the next access is nearby, it can starting looking from this
bookmark, forwards or backwards. Like a gap buffer, this gives a big
advantage to clustered accesses, including iteration.
However, this string bookmark is global, one per Emacs instance, not
once per string. In the last benchmark, the two multibyte strings are
constantly fighting over a single string bookmark, and indexing in
comparison function is reduced to O(n^2) time complexity.
So, Emacs pretends it has constant time access into its UTF-8 text
data, but it’s only faking it with some simple optimizations. This
usually works out just fine.
Julia
Another approach is to not pretend at all, and to make this limitation
of UTF-8 explicit in the interface. Julia took this approach, and it
was one of my complaints about the language. I don’t think
this is necessarily a bad choice, but I do still think it’s
inappropriate considering Julia’s target audience (i.e. Matlab users).
Julia strings are explicitly byte strings containing valid UTF-8 data.
All indexing occurs on bytes, which is trivially constant time, and
always decodes the multibyte code point starting at that byte. But
it is an error to index to a byte that doesn’t begin a code point.
That error is also trivially checked in constant time.
s="π"s[1]# => 'π's[2]# ERROR: UnicodeError: invalid character index# in getindex at ./strings/basic.jl:37
Slices are still over bytes, but they “round up” to the end of the
current code point:
s[1:1]# => "π"
Iterating over a string requires helper functions which keep an internal
“bookmark” so that each access is constant time:
foriineachindex(string)c=string[i]# ...end
So Julia doesn’t pretend, it makes the problem explicit.
Go
Go is very similar to Julia, but takes an even more explicit view of
strings. All strings are byte strings and there are no restrictions on
their contents. Conventionally strings contain UTF-8 encoded text, but
this is not strictly required. There’s a unicode/utf8 package for
working with strings containing UTF-8 data.
Beyond convention, the range clause also assumes the string contains
UTF-8 data, and it’s not an error if it does not. Bytes not containing
valid UTF-8 data appear as a REPLACEMENT CHARACTER (U+FFFD).
A further case of the language favoring UTF-8 is that casting a string
to []rune decodes strings into code points, like UCS-4, again using
REPLACEMENT CHARACTER:
So, like Julia, there’s no pretending, and the programmer explicitly
must consider the problem.
Preferences
All-in-all I probably prefer how Julia and Go are explicit with
UTF-8’s limitations, rather than Emacs Lisp’s attempt to cover it up
with an internal optimization. Since the abstraction is leaky, it may
as well be made explicit.
Software tests are great! I’m fortunate enough to have only worked with
code-bases with reasonable-to-excellent test coverage, and I wouldn’t want to
work in a world without tests. In fact, a thoroughly tested system is nothing
short of liberating.
That said, tests are not free. I’m not talking about CI time, that is a cost but
it’s usually reducible. Nor am I referring to the effort it takes to write the
test, that’s a very real cost, but people are usually very mindful of that (it’s
easy to take it into account the very real effort you’re having right now).
The cost that people tend to underestimate is the time wasted with false
failures.
Let’s get the basics right. Tests are designed to fail. A test that never fails
under any circumstance is a useless test. But there are good failures and bad
failures. Which gets me to the entire point of this post.
Write tests with real failures
Real failures are good, false failures are bad.
You want a test to fail when you break functionality, not when you harmlessly
change code.
Let’s start with a quick Ruby example. Consider a model with an attribute called
value. It wouldn’t be surprising to see a test like this for such a model.
it{is_expected.torespond_to(:value)}
If you don’t know Rspec, this is roughly testing that you can call .value on
the model.
But when will this test ever fail?
Under any reasonable condition, this will only fail if you rename the column in
the database. Nobody will ever do that by accident!
What’s worse, this failure will never carry any useful information. It doesn’t
tell the developer that this change is unexpectedly breaking something. All it
ever does is give us yet another piece of code to fix while in the middle of an
already long refactoring.
And how do we fix it? By editing the test to use the new name.
A false failure is one that you fix by editing the test, while a real failure
is one you fix by editing the code.
False failures are unavoidable. Every test is exposed to them, and every time they
happen the developer wastes some amount of time identifying and fixing the
failure. That is a negative cost that every test carries and not everyone takes
into account.
For most cases, we happily pay this cost because not having a test is way worse
than having to fix it. Because one real failure preventing a bug from going live
outweighs several false failures, adding up to a positive net effect.
But some tests (such as the example above) virtually never have real failures.
Without any positive upside to them, they are strictly negative tests.
And how do we avoid negative tests?
Test function, not code
Let’s expand on our previous example.
Rails provides a helpful one-liner to validate the presence of a mandatory
attribute.
validates:value,presence: true
That’s fine and good. The problem is when you see a similar one-liner testing
that validation.
it{is_expected.tovalidate_presence_of(:value)}
Pause on that for a moment. We’ve written a spec to test a single line of code.
The only way that can fail is if we rename the attribute or remove the
validation. Again, nobody is ever going to do that by accident. We’re not really
testing that a specific functionality works as it should, we’re just testing that
a particular line of code is written in a particular way.
That is a code test, not a function test, and code tests are negative tests.
A function test is one that verifies non-trivial functionality, functionality that
could be accidentally broken by a number of reasons.
Testing the interface of a service, for instance, is basically always good. As
there’s usually at least a few branching code paths inside it where one could
inadvertently break a branch while editing another or while adding functionality.
Unit tests for simple functions and methods, in my opinion, are not
no-brainers. People like to go nuts with them, because they’re easy to write and
quick to run (so “why not?”), but a lot of them fall under the category of
negative tests.
Unit tests are good when testing some reasonably complicated algorithm, as
someone could actually break an edge case while trying to optimize the
implementation. And even then, you shouldn’t just write a couple of mindless tests,
as they will probably be negative. You should put some effort into figuring out
the edge cases and testing them specifically.
Think before you test
Hopefully, you started thinking well before you wrote that first line of code,
so there’s no reason to stop now just because you changed from the app/ to the
specs/ directory.
Thinking and being mindful of what you’re testing will not only help you avoid
negative tests, but will go a long way to making your positive tests more
effective at catching the bugs they’re supposed to catch.
This release consists of 45 commits done over the course of the last 2 years. With this version, I
have introduced a
Changelog.org,
similar to what ivy and
avy have.
Highlights
Display hints using posframe
A new defcustom hydra-hint-display-type was introduced that can be either lv (the default),
message, or posframe.
Posframe is a package that leverages a new feature in Emacs
26.1: the ability to display child frames. The advantage of using child frames is that you can
easily position them anywhere within your frame area. For example, the default setting is to put it
in the center of the current window, which is closer to where your eyes are focused than the
minibuffer. Child frames don't interfere with the content of the buffers which they
overlap. Finally, you can select a different font for the child frame.
Less boilerplate in defhydra
You no longer have to add :hint nil, and you can skip the docstring as well:
The new macro defhydra+ takes the same arguments as defhydra, so it's quite easy to split up or
join your hydras.
The use case of defhydra+ is when you have many packages that want to add heads to an existing
hydra. Some of them may be optional or loaded lazily.
You can now have a base defhydra, and then use defhydra+ to add heads to it when a new package
is loaded. Example:
(defhydrahydra-toggle()("q"nil"quit":column"Exit")("w"whitespace-mode(format"whitespace-mode: %S"whitespace-mode):column"Toggles"))(use-packageorg:config(defhydra+hydra-toggle()("l"org-toggle-link-display(format"org link display: %S"org-descriptive-links))))
Outro
Big thanks to all contributors, and I hope you enjoy the new release. Happy hacking!
PS. Thanks to everyone who supports me on Liberapay and
Patreon!
This release consists of 109 commits done over the course of the last 3 years by me and many
contributors. Similarly to the 0.4.0
release, the release notes are in
Changelog.org. I recommend
reading them inside Emacs.
Highlights
A lot of new code is just straight upgrades, you don't need to do
anything extra to use them. Below, I'll describe the other part of the
new code, which is new commands and custom vars.
New API functions
New functions have been added as drop-in replacements of double-dash (private) Avy functions that
were used in other packages and configs. Please replace the references to the obsolete functions.
avy-jump is a drop-in replacement of avy--generic-jump,
avy-process is a drop-in replacement of avy--process.
New dispatch actions
The concept of dispatch actions was introduced in 0.4.0.
Suppose you have bound:
(global-set-key(kbd"M-t")'avy-goto-word-1)
and a word that starts with a "w" and is select-able with "a". Here's what you can do now:
M-t w a to jump there
M-t w x a - avy-action-kill-move: kill the word and move there,
M-t w X a - avy-action-kill-stay: kill the word without moving the point,
M-t w i a - avy-action-ispell: use ispell/flyspell to correct the word,
M-t w y a - avy-action-yank: yank the word at point,
M-t w t a - avy-action-teleport: kill the word and yank it at point,
M-t w z a - avy-action-zap-to-char: kill from point up to selected point.
You can customize avy-dispatch-alist to modify these actions, and also ensure that there's no
overlap with your avy-keys, if you customized them.
New avy-style setting: 'words
You can now customize:
(setqavy-style'words)
And you'll see overlays like "by", "if", "is", "it", "my" for 2-letter sequences, and "can", "car",
"cog" for 3-letter sequences. You might find them easier to type than "hla", "lls" and "jhl". But
you will have to adjust your avy-dispatch-alist, e.g. to use only upper case characters.
avy-linum-mode
This is feature is a mix of linum-mode and ace-window-display-mode. You'll see the overlays when
you enable this mode, so that there's less context switch when you call avy-goto-line.
Restarting an avy search
Suppose you jumped to a word that starts with "a". Now you want to jump to a different word that
also starts with "a". You can use avy-resume for this.
Additionally, you can use avy-next and avy-prev to cycle between the last avy
candidates. Here's an example hydra to facilitate it:
I'm constantly amazed by other people's Org workflows. Now that the weekly tips are a
thing, I see more and more cool Org configs, and I'm inspired to get more organized myself.
My own Org usage is simplistic in some areas, and quite advanced in others. While I wrote a lot of
code to manipulate Org files ( worf,
org-download, orca,
org-fu, counsel), the
amount of Org files and TODO items that I have isn't huge:
Still, that's enough to get out-of-date files: just today I dug up a file with 20 outstanding TODO
items that should have been canceled last November!
How to close 20 TODOs using a timestamp in the past
When I cancel an item, pressing tc (mnemonic for TODO-Cancel), Org mode inserts a time
stamp with the current time. However, for this file, I wanted to use October 31st 2018 instead of
the current time. Org mode already has options like org-use-last-clock-out-time-as-effective-time,
org-use-effective-time, and org-extend-today-until that manipulate the current time for
timestamps, but they didn't fit my use case.
So I've advised org-current-effective-time:
(defvar-localworf--current-effective-timenil)(defunworf--current-effective-time(orig-fn)(orworf--current-effective-time(funcallorig-fn)))(advice-add'org-current-effective-time:around#'worf--current-effective-time)(defunworf-change-time()"Set `current-time' in the current buffer for `org-todo'.Use `keyboard-quit' to unset it."(interactive)(setqworf--current-effective-time(condition-casenil(org-read-datet'totime)(quitnil))))
A few things of note here:
worf--current-effective-time is buffer-local, so that it modifies time only for the current
buffer
I re-use the awesome org-read-date for a nice visual feedback when inputting the new time
Instead of having a separate function to undo the current-time override, I capture the quit
signal that C-g sends.
Outro
The above code is already part of worf and is bound to
cT. I even added it to the manual. I hope you find it useful. Happy organizing!
Since its introduction in 2015, swiper, while nice most of the time, had two problems:
Slow startup for large buffers.
Candidates were lines, so if you had two or more matches on the same line, the first one was selected.
Over time, workarounds were added to address these problems.
Problem 1: slow startup
Almost right away, calling font-lock-ensure was limited to only small enough buffers.
In 2016, counsel-grep-or-swiper was introduced. It
uses an external process (grep) to search through large files.
In 2017, I found ripgrep, which does a better
job than grep for searching one file:
(setqcounsel-grep-base-command"rg -i -M 120 --no-heading --line-number --color never %s %s")
The advantage here is that the search can be performed on very large files. The trade-off is that we
have to type in at least 3 characters before we send it to the external process. Otherwise, when the
process returns a lot of results, Emacs will lag while receiving all that output.
Problem 2: candidates are lines
In 2015, swiper-avy was added, which could also be
used as a workaround for many candidates on a single line. Press C-' to visually select
any candidate on screen using avy.
Enter swiper-isearch
Finally, less than a week ago, I wrote swiper-isearch to fix #1931.
Differences from the previous commands:
Every candidate is a point position and not a line. The UX of going from one candidate to the next
is finally isearch-like, I enjoy it a lot.
Unlike swiper, no line numbers are added to the candidates. This allows it to be as fast as
anzu.
Unlike counsel-grep, no external process is used. So you get feedback even after inputting a
single char.
I like it a lot so far, enough to make it my default search:
(global-set-key(kbd"C-s")'swiper-isearch)
Outro
Try out swiper-isearch, see if it can replace swiper for you; counsel-grep-or-swiper still has
its place, I think. Happy hacking!
PS. Thanks to everyone who supports me on Liberapay and
Patreon!
For a couple years now, I use M-xshell as my main shell. Recently, I have fixed one of
the minor annoyances that go along with using shell in Emacs. At least since Ubuntu 18.04, the
terminal "progress bar" feature, displayed below is non-optional:
It uses terminal escape codes to display the progress bar, and shell-mode can't handle them well,
so they clobber a lot of the output.
Initial work around
Previously, I was using this work around, since apt-get doesn't display the progress bar:
# sudo apt upgrade
sudo apt-get upgrade
Progress bar in the mode line
But typing 4 extra chars is hard. And apt-get will likely get these progress bars at some point as
well. So I spent around an hour of my weekend hacking an Elisp solution. Here is the code:
(advice-add'ansi-color-apply-on-region:before'ora-ansi-color-apply-on-region)(defunora-ansi-color-apply-on-region(beginend)"Fix progress bars for e.g. apt(8).Display progress in the mode line instead."(let((end-marker(copy-markerend))mb)(save-excursion(goto-char(copy-markerbegin))(while(re-search-forward"\0337"end-markert)(setqmb(match-beginning0))(when(re-search-forward"\0338"end-markert)(ora-apt-progress-message(substring-no-properties(delete-and-extract-regionmb(point))2-2)))))))(defunora-apt-progress-message(progress)(setqmode-line-process(if(string-match"Progress: \\[ *\\([0-9]+\\)%\\]"progress)(list(concat":%s "(match-string1progress)"%%%% "))'(":%s")))(force-mode-line-update))
The solution will detect e.g. "\0337...Progress: [ 25%]...\0338", remove it from the shell buffer
and display "25%" in the mode line instead.
Use the Echo Area instead of the mode line
The above is a good enough solution specifically for apt(8), but not for the generic case. Let's
try to emulate how e.g. gnome-terminal handles these escape sequences. It takes sequences like
"\0337.*\0338" and displays them in the bottom of the window. Kind of like the Emacs Echo
Area. That's easy enough to do:
Above, we use ansi-color-apply to get rid of any extra terminal escape codes. I decided to stay
with the Echo Area version instead of the mode line version. Here's how it looks like:
You can find all of the above code in my config.
Happy hacking!
In fact, both Python and JavaScript async functions are essentially just
fancy generator functions with some specialized syntax and semantics.
That is, they’re stackless coroutines. Both languages already had
generators, so their generator-like async functions are a natural
extension that — unlike stackful coroutines — do not require
significant, new runtime plumbing.
Emacs officially got generators in 25.1 (September 2016),
though, unlike Python and JavaScript, it didn’t require any additional
support from the compiler or runtime. It’s implemented entirely using
Lisp macros. In other words, it’s just another library, not a core
language feature. In theory, the generator library could be easily
backported to the first Emacs release to properly support lexical
closures, Emacs 24.1 (June 2012).
For the same reason, stackless async/await coroutines can also be
implemented as a library. So that’s what I did, letting Emacs’ generator
library do most of the heavy lifting. The package is called aio:
It’s modeled more closely on JavaScript’s async functions than Python’s
asyncio, with the core representation being promises rather than a
coroutine objects. I just have an easier time reasoning about promises
than coroutines.
I’m definitely not the first person to realize this was
possible, and was beaten to the punch by two years. Wanting to
avoid fragmentation, I set aside all formality in my first
iteration on the idea, not even bothering with namespacing my
identifiers. It was to be only an educational exercise. However, I got
quite attached to my little toy. Once I got my head wrapped around the
problem, everything just sort of clicked into place so nicely.
In this article I will show step-by-step one way to build async/await
on top of generators, laying out one concept at a time and then
building upon each. But first, some examples to illustrate the desired
final result.
aio example
Ignoring all its problems for a moment, suppose you want to use
url-retrieve to fetch some content from a URL and return it. To keep
this simple, I’m going to omit error handling. Also assume that
lexical-binding is t for all examples. Besides, lexical scope
required by the generator library, and therefore also required by aio.
The most naive approach is to fetch the content synchronously:
The result is returned directly, and errors are communicated by an error
signal (e.g. Emacs’ version of exceptions). This is convenient, but the
function will block the main thread, locking up Emacs until the result
has arrived. This is obviously very undesirable, so, in practice,
everyone nearly always uses the asynchronous version:
The main thread no longer blocks, but it’s a whole lot less
convenient. The result isn’t returned to the caller, and instead the
caller supplies a callback function. The result, whether success or
failure, will be delivered via callback, so the caller must split
itself into two pieces: the part before the callback and the callback
itself. Errors cannot be delivered using a error signal because of the
inverted flow control.
The situation gets worse if, say, you need to fetch results from two
different URLs. You either fetch results one at a time (inefficient),
or you manage two different callbacks that could be invoked in any
order, and therefore have to coordinate.
Wouldn’t it be nice for the function to work like the first example,
but be asynchronous like the second example? Enter async/await:
A function defined with aio-defun is just like defun except that
it can use aio-await to pause and wait on any other function defined
with aio-defun — or, more specifically, any function that returns a
promise. Borrowing Python parlance: Returning a promise makes a
function awaitable. If there’s an error, it’s delivered as a error
signal from aio-url-retrieve, just like the first example. When
called, this function returns immediately with a promise object that
represents a future result. The caller might look like this:
How wonderfully clean that looks! And, yes, it even works with
interactive like that. I can M-x display-fortune and a fortune is
printed in the minibuffer as soon as the result arrives from the
server. In the meantime Emacs doesn’t block and I can continue my
work.
You can’t do anything you couldn’t already do before. It’s just a
nicer way to organize the same callbacks: implicit rather than
explicit.
Promises, simplified
The core object at play is the promise. Promises are already a
rather simple concept, but aio promises have been distilled to their
essence, as they’re only needed for this singular purpose. More on
this later.
As I said, a promise represents a future result. In practical terms, a
promise is just an object to which one can subscribe with a callback.
When the result is ready, the callbacks are invoked. Another way to
put it is that promises reify the concept of callbacks. A
callback is no longer just the idea of extra argument on a function.
It’s a first-class thing that itself can be passed around as a
value.
Promises have two slots: the final promise result and a list of
subscribers. A nil result means the result hasn’t been computed
yet. It’s so simple I’m not even bothering with cl-struct.
(defunaio-promise()"Create a new promise object."(record'aio-promisenil()))(defsubstaio-promise-p(object)(and(eq'aio-promise(type-ofobject))(=3(lengthobject))))(defsubstaio-result(promise)(arefpromise1))
If the result isn’t ready yet, add the callback to the list of
subscribers. If the result is ready call the callback in the next
event loop turn using run-at-time. This is important because it
keeps all the asynchronous components isolated from one another. They
won’t see each others’ frames on the call stack, nor frames from
aio. This is so important that the Promises/A+ specification
is explicit about it.
The other half of the equation is resolving a promise, which is done
with aio-resolve. Unlike other promises, aio promises don’t care
whether the promise is being fulfilled (success) or rejected
(error). Instead a promise is resolved using a value function — or,
usually, a value closure. Subscribers receive this value function
and extract the value by invoking it with no arguments.
Why? This lets the promise’s resolver decide the semantics of the
result. Instead of returning a value, this function can instead signal
an error, propagating an error signal that terminated an async function.
Because of this, the promise doesn’t need to know how it’s being
resolved.
When a promise is resolved, subscribers are each scheduled in their own
event loop turns in the same order that they subscribed. If a promise
has already been resolved, nothing happens. (Thought: Perhaps this
should be an error in order to catch API misuse?)
Before we get into pausing functions, lets deal with the slightly
simpler matter of delivering their return values using a promise. What
we need is a way to evaluate a “body” and capture its result in a
promise. If the body exits due to a signal, we want to capture that as
well.
The body result is captured in a closure and delivered to the promise.
If there’s an error signal, it’s “rethrown” into subscribers by the
promise’s value function.
This is where Emacs Lisp has a serious weak spot. There’s not really a
concept of rethrowing a signal. Unlike a language with explicit
exception objects that can capture a snapshot of the backtrace, the
original backtrace is completely lost where the signal is caught.
There’s no way to “reattach” it to the signal when it’s rethrown. This
is unfortunate because it would greatly help debugging if you got to see
the full backtrace on the other side of the promise.
Async functions
So we have promises and we want to pause a function on a promise.
Generators have iter-yield for pausing an iterator’s execution. To
tackle this problem:
Yield the promise to pause the iterator.
Subscribe a callback on the promise that continues the generator
(iter-next) with the promise’s result as the yield result.
All the hard work is done in either side of the yield, so aio-await is
just a simple wrapper around iter-yield:
Remember, that funcall is here to extract the promise value from the
value function. If it signals an error, this propagates directly into
the iterator just as if it had been a direct call — minus an accurate
backtrace.
So aio-lambda / aio-defun needs to wrap the body in a generator
(iter-lamba), invoke it to produce a generator, then drive the
generator using callbacks. Here’s a simplified, unhygienic definition of
aio-lambda:
The body is evaluated inside aio-with-promise with the result
delivered to the promise returned directly by the async function.
Before returning, the iterator is handed to aio--step, which drives
the iterator forward until it delivers its first promise. When the
iterator yields a promise, aio--step attaches a callback back to
itself on the promise as described above. Immediately driving the
iterator up to the first yielded promise “primes” it, which is
important for getting the ball rolling on any asynchronous operations.
If the iterator ever yields something other than a promise, it’s
delivered right back into the iterator.
When the iterator is done, nothing more needs to happen since the
iterator resolves its own return value promise.
The definition of aio-defun just uses aio-lambda with defalias.
There’s nothing to it.
That’s everything you need! Everything else in the package is merely
useful, awaitable functions like aio-sleep and aio-timeout.
Composing promises
Unfortunately url-retrieve doesn’t support timeouts. We can work
around this by composing two promises: a url-retrieve promise and
aio-timeout promise. First define a promise-returning function,
aio-select that takes a list of promises and returns (as another
promise) the first promise to resolve:
Cool! Note: This will not actually cancel the URL request, just move
the async function forward earlier and prevent it from getting the
result.
Threads
Despite aio being entirely about managing concurrent, asynchronous
operations, it has nothing at all to do with threads — as in Emacs 26’s
support for kernel threads. All async functions and promise callbacks
are expected to run only on the main thread. That’s not to say an
async function can’t await on a result from another thread. It just must
be done very carefully.
Processes
The package also includes two functions for realizing promises on
processes, whether they be subprocesses or network sockets.
aio-process-filter
aio-process-sentinel
For example, this function loops over each chunk of output (typically
4kB) from the process, as delivered to a filter function:
Exercise for the reader: Write an awaitable function that returns a line
at at time rather than a chunk at a time. You can build it on top of
aio-process-filter.
I considered wrapping functions like start-process so that their aio
versions would return a promise representing some kind of result from
the process. However there are so many different ways to create and
configure processes that I would have ended up duplicating all the
process functions. Focusing on the filter and sentinel, and letting the
caller create and configure the process is much cleaner.
Unfortunately Emacs has no asynchronous API for writing output to a
process. Both process-send-string and process-send-region will block
if the pipe or socket is full. There is no callback, so you cannot await
on writing output. Maybe there’s a way to do it with a dedicated thread?
Another issue is that the process-send-* functions are
preemptible, made necessary because they block. The
aio-process-* functions leave a gap (i.e. between filter awaits)
where no filter or sentinel function is attached. It’s a consequence
of promises being single-fire. The gap is harmless so long as the
async function doesn’t await something else or get preempted. This
needs some more thought.
Update: These process functions no longer exist and have been
replaced by a small framework for building chains of promises. See
aio-make-callback.
Testing aio
The test suite for aio is a bit unusual. Emacs’ built-in test suite,
ERT, doesn’t support asynchronous tests. Furthermore, tests are
generally run in batch mode, where Emacs invokes a single function and
then exits rather than pump an event loop. Batch mode can only handle
asynchronous process I/O, not the async functions of aio. So it’s
not possible to run the tests in batch mode.
Instead I hacked together a really crude callback-based test suite. It
runs in non-batch mode and writes the test results into a buffer
(run with make check). Not ideal, but it works.
One of the tests is a sleep sort (with reasonable tolerances). It’s a
pretty neat demonstration of what you can do with aio:
I’m quite happy with how this all came together. Once I had the
concepts straight — particularly resolving to value functions —
everything made sense and all the parts fit together well, and mostly
by accident. That feels good.
Earlier this year I did an expansion to my entries on MPD, sort of a state of my setup piece, and I've decided to do the same thing for my Emacs and Python setups as well. If you write Python code and use Emacs, or are interested in either, read on!
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
This month we are doing lightning talks!
We look forward to any talk you want to give that is Emacs or Emacs adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 10 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
Suyash Bire: Spin Your Own Spacemacs-lite
I will talk about my setup that uses evil-mode, general.el, which-key,
and use-package to emulate spacemacs-like behavior (in most cases). My
main focus will be on how general.el enables modularization. I will
show how I use this setup to code in python, write LaTeX documents,
and conduct reproducible research using org-mode
Avito Demand Prediction Challenge asks Kagglers to predict the
"demand" likelihood of an advertisement. If an listed 2nd-hand Iphone
6 is selling for £20,000, then the "demand" is likely to be very low.
This is the my first competition to build model using tabular data,
text, and also images.
I teamed up with Rashmi, Abhimanyu, Yiang, Samrat and we finished at
22 among 1917 teams. So far, I have four silver medals and my rank is
542 among 83,588 Kaggler.
This is an interesting competition for me. I was about to quit this
competition and Kaggle because of other commitments in life/work. Just
one day before team merge deadline, Rashmi asked me to join, at that
time, my position is 880-th, about 50%, and Rashmi's team is about
82-th. So I decided to join and finish this competition which I
already spent about many hours.
Final Ensemble Models
As part of this team, I worked on final ensemble models. Immediately
after join, i completed 5 tasks:
make sure everyone uses the same agreed cross validation schema.
This is essential for building ensemble model.
provide model_zoo.md document to keep track of all level 1
models, their train/valid/lb scores, feature used, and file path
to their oof/test prediction.
write merge_oof.py to combine all oof/test predictions together.
write R scripts for glmnet ensemble.
write python scripts for LightGBM ensemble.
Once new model is built, other team member update the model_zoo.md
and upload the data to a private github repo. Then I update the
merge_oof.py to include new models' result, and run glmnet and
LightGBM ensemble. We had this ensemble workflow automated so it
takes little effort to see the ensemble model's performance.
I spent some times analysing the coefficients/weights of L1 model
and tried to exclude models with negative and lower weights. To my
surprise it doesn't help at all. The final submission is a glmnet
ensemble with 41 models (lgb + xgb + NN).
Also, LightGBM ensemble has much better cv score but the LB score
is worse. I suspect it is because there are leakage in L1 models
and glmnet is more robust to leakage since it's linear model.
Unfortunately, there's no enough time to identify which models have
leakage.
Collaboration
This is my 2nd time work in a team, although there's a lot space
for improvement collaborating when compared with a professional
data scientist team but as night/weekend project, we have done a
really good job as a team.
The setup for collaboration:
Slack for discussion. we have channel for general,
final_ensemble, random for cat photos etc.
we also used Slack for sharing features which i personal don't
like.
Private github repo for sharing code and oof/test predictions.
Monday.com for managing tasks. it gives a nice overview of what
everyone's up to.
we tried very hard to get a gold, but other teams work even more
harder. At one point we were at 17, and finished at 22.
Some Kagglers to Avoid
Finally, when we waited 1 hour for the final deadline, we had a
lovely discussion about our past disqualification experience. We
were all shocked when we were at different team in Toxic
competition but team up with the same person. We shared their
person's multiple Kaggle accounts and added to our personal
block-list.
So, you want to run an Emacs daemon as a runit service - and you also want to connect to it in your desktop session. Thanks to the new --fg-daemon option in Emacs 26.x you now can! I'm going to describe how to set up the service, as well as sudo rules for managing it without requiring a password each time. Read on for the exciting details!
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
This month we are doing lightning talks!
We look forward to any talk you want to give that is Emacs or Emacs adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 10 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
Correction: Generators were actually introduced in Emacs 25.1
(Sept. 2016), not Emacs 26.1. Doh!
Update: ThreadSanitizer (TSan) quickly shows that Emacs’
threading implementation has many data races, making it completely
untrustworthy. Until this is fixed, nobody should use Emacs
threads for any purpose, and threads should disabled at compile time.
Generators
Generators are one of those cool language features that provide a lot of
power at a small implementation cost. They’re like a constrained form of
coroutines, but, unlike coroutines, they’re typically built entirely on
top of first-class functions (e.g. closures). This means no additional
run-time support is needed in order to add generators to a language.
The only complications are the changes to the compiler. Generators are
not compiled the same way as normal functions despite looking so
similar.
What’s perhaps coolest of all about lisp-family generators, including
Emacs Lisp, is that the compiler component can be implemented
entirely with macros. The compiler need not be modified at all,
making generators no more than a library, and not actually part of the
language. That’s exactly how they’ve been implemented in Emacs Lisp
(emacs-lisp/generator.el).
So what’s a generator? It’s a function that returns an iterator
object. When an iterator object is invoked (e.g. iter-next) it
evaluates the body of the generator. Each iterator is independent.
What makes them unusual (and useful) is that the evaluation is
paused in the middle of the body to return a value, saving all the
internal state in the iterator. Normally pausing in the middle of
functions isn’t possible, which is what requires the special compiler
support.
Emacs Lisp generators appear to be most closely modeled after Python
generators, though it also shares some similarities to
JavaScript generators. What makes it most like Python is the use
of signals for flow control — something I’m not personally enthused
about. When a Python generator
completes, it throws a StopItertion exception. In Emacs Lisp, it’s
an iter-end-of-sequence signal. A signal is out-of-band and avoids
the issue relying on some special in-band value to communicate the end
of iteration.
In contrast, JavaScript’s solution is to return a “rich” object wrapping
the actual yield value. This object has a done field that communicates
whether iteration has completed. This avoids the use of exceptions for
flow control, but the caller has to unpack the rich object.
Fortunately the flow control issue isn’t normally exposed to Emacs Lisp
code. Most of the time you’ll use the iter-do macro or (my preference)
the new cl-loop keyword iter-by.
To illustrate how a generator works, here’s a really simple iterator
that iterates over a list:
The iterator object itself is opaque and you shouldn’t rely on any
part of its structure. That being said, I’m a firm believer that we
should understand how things work underneath the hood so that we can
make the most effective use of at them. No program should rely on the
particulars of the iterator object internals for correctness, but a
well-written program should employ them in a way that best exploits
their expected implementation.
Currently iterator objects are closures, and iter-next invokes the
closure with its own internal protocol. It asks the closure to return
the next value (:next operation), and iter-close asks it to clean
itself up (:close operation).
Since they’re just closures, another really cool thing about Emacs
Lisp generators is that iterator objects are generally readable.
That is, you can serialize them out with print and bring them back to
life with read, even in another instance of Emacs. They exist
independently of the original generator function. This will not work if
one of the values captured in the iterator object is not readable (e.g.
buffers).
How does pausing work? Well, one of other exciting new features of
Emacs 26 is the introduction of a jump table opcode, switch. I’d
lamented in the past that large cond and cl-case expressions could
be a lot more efficient if Emacs’ byte code supported jump tables. It
turns an O(n) sequence of comparisons into an O(1) lookup and jump.
It’s essentially the perfect foundation for a generator since it can
be used to jump straight back to the position where evaluation
was paused.
Buuut, generators do not currently use jump tables. The generator
library predates the new switch opcode, and, being independent of it,
its author, Daniel Colascione, went with the best option at the time.
Chunks of code between yields are packaged as individual closures. These
closures are linked together a bit like nodes in a graph, creating a
sort of state machine. To get the next value, the iterator object
invokes the closure representing the next state.
I’ve manually macro expanded the walk generator above into a form
that roughly resembles the expansion of iter-defun:
This omits the protocol I mentioned, and it doesn’t have yield results
(values passed to the iterator). The actual expansion is a whole lot
messier and less optimal than this, but hopefully my hand-rolled
generator is illustrative enough. Without the protocol, this iterator is
stepped using funcall rather than iter-next.
The state variable keeps track of where in the body of the generator
this iterator is currently “paused.” Continuing the iterator is
therefore just a matter of invoking the closure that represents this
state. Each state closure may update state to point to a new part of
the generator body. The terminal state is obviously state-2. Notice
how state transitions occur around branches.
I had said generators can be implemented as a library in Emacs Lisp.
Unfortunately theres a hole in this: unwind-protect. It’s not valid to
yield inside an unwind-protect form. Unlike, say, a throw-catch,
there’s no mechanism to trap an unwinding stack so that it can be
restarted later. The state closure needs to return and fall through the
unwind-protect.
A jump table version of the generator might look like the following.
I’ve used cl-labels since it allows for recursion.
When byte compiled on Emacs 26, that cl-case is turned into a jump
table. This “switch” form is closer to how generators are implemented in
other languages.
Iterator objects can share state between themselves if they
close over a common environment (or, of course, use the same global
variables).
For years there has been a very crude way to “pause” a function and
allow other functions to run: accept-process-output. It only works in
the context of processes, but five years ago this was sufficient for me
to build primitives on top of it. Unlike this old process
function, generators do not block threads, including the user interface,
which is really important.
Threads
Emacs 26 also bring us threads, which have been attached in a very
bolted on fashion. It’s not much more than a subset of pthreads: shared
memory threads, recursive mutexes, and condition variables. The
interfaces look just like they do in pthreads, and there hasn’t been
much done to integrate more naturally into the Emacs Lisp ecosystem.
This is also only the first step in bringing threading to Emacs Lisp.
Right now there’s effectively a global interpreter lock (GIL), and
threads only run one at a time cooperatively. Like with generators, the
Python influence is obvious. In theory, sometime in the future this
interpreter lock will be removed, making way for actual concurrency.
This is, again, where I think it’s useful to contrast with JavaScript,
which was also initially designed to be single-threaded. Low-level
threading primitives weren’t exposed — though mostly because
JavaScript typically runs sandboxed and there’s no safe way to expose
those primitives. Instead it got a web worker API that exposes
concurrency at a much higher level, along with an efficient interface
for thread coordination.
For Emacs Lisp, I’d prefer something safer, more like the JavaScript
approach. Low-level pthreads are now a great way to wreck Emacs with
deadlocks (with no C-g escape). Playing around with the new
threading API for just a few days, I’ve already had to restart Emacs a
bunch of times. Bugs in Emacs Lisp are normally a lot more forgiving.
One important detail that has been designed well is that dynamic
bindings are thread-local. This is really essential for correct
behavior. This is also an easy way to create thread-local storage
(TLS): dynamically bind variables in the thread’s entrance function.
However, cl-letf “bindings” are not thread-local, which makes
this otherwise incredibly useful macro quite dangerous in the
presence of threads. This is one way that the new threading API feels
bolted on.
Building generators on threads
In my stack clashing article I showed a few different ways to
add coroutine support to C. One method spawned per-coroutine threads,
and coordinated using semaphores. With the new threads API in Emacs,
it’s possible to do exactly the same thing.
Since generators are just a limited form of coroutines, this means
threads offer another, very different way to implement them. The
threads API doesn’t provide semaphores, but condition variables can fill
in for them. To “pause” in the middle of the generator, just wait on a
condition variable.
So, naturally, I just had to see if I could make it work. I call it a
“thread iterator” or “thriter.” The API is very similar to iter:
This is merely a proof of concept so don’t actually use this library
for anything. These thread-based generators are about 5x slower than
iter generators, and they’re a lot more heavy-weight, needing an
entire thread per iterator object. This makes thriter-close all the
more important. On the other hand, these generators have no problem
yielding inside unwind-protect.
Originally this article was going to dive into the details of how
these thread-iterators worked, but thriter turned out to be quite a
bit more complicated than I anticipated, especially as I worked
towards feature matching iter.
The gist of it is that each side of a next/yield transaction gets its
own condition variable, but share a common mutex. Values are passed
between the threads using slots on the iterator object. The side that
isn’t currently running waits on a condition variable until the other
side frees it, after which the releaser waits on its own condition
variable for the result. This is similar to asynchronous requests in
Emacs dynamic modules.
Rather than use signals to indicate completion, I modeled it after
JavaScript generators. Iterators return a cons cell. The car indicates
continuation and the cdr holds the yield result. To terminate an
iterator early (thriter-close or garbage collection), thread-signal
is used to essentially “cancel” the thread and knock it off the
condition variable.
Since threads aren’t (and shouldn’t be) garbage collected, failing to
run a thread-iterator to completion would normally cause a memory leak,
as the thread sits there forever waiting on a “next” that will never
come. To deal with this, there’s a finalizer is attached to the
iterator object in such a way that it’s not visible to the thread. A
lost iterator is eventually cleaned up by the garbage collector, but, as
usual with finalizers, this is only a last resort.
The future of threads
This thread-iterator project was my initial, little experiment with
Emacs Lisp threads, similar to why I connected a joystick to Emacs
using a dynamic module. While I don’t expect the current thread
API to go away, it’s not really suitable for general use in its raw
form. Bugs in Emacs Lisp programs should virtually never bring down
Emacs and require a restart. Outside of threads, the few situations
that break this rule are very easy to avoid (and very obvious that
something dangerous is happening). Dynamic modules are dangerous by
necessity, but concurrency doesn’t have to be.
There really needs to be a safe, high-level API with clean thread
isolation. Perhaps this higher-level API will eventually build on top of
the low-level threading API.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
Rocky Bernstein will talk about what bytecode is, its value, and its
limitations. Focusing primarily on the Emacs bytecode runtime, he’ll
compare it to other implementations of elisp bytecode, such as Emacs in
Rust
Rocky Bernstein (github) is a long time emacs user, prolific
developer, and is heavily involved in the free software community. He
wrote and maintains realgud and literally wrote the book on Emacs
Lisp Bytecode
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
This month we are doing lightning talks!
We look forward to any talk you want to give that is Emacs or Emacs adjacent.
We do want to hear everything you have to say, but we will be limiting
each talk to 10 minutes and we will be strict about this. If you have
more to say please consider talking to us about doing a longer talk
next month.
I like Git. A lot. After years of use it has
really grown on me. It's (mostly) fast, (often) reliable, and (always)
distributed. For me, all properties are important, but being able to
do git init to start a new project in seconds is the best feature.
When it comes to working with Git day-to-day, a nice GUI can really
make a difference. In Emacs world, of course it's
Magit. Outside of Emacs (brr),
git-cola looks to be the most promising
one. If you're aware of something better, please share - I'm keeping a
list of suggestions for my non-Emacs using colleagues.
Ivy integration for Git
The main two commands in ivy
that I use for Git are:
counsel-git: select a file tracked by Git
counsel-rg: grep for a line in all files tracked by Git, using
ripgrep as the backend.
There are many alternatives to counsel-rg that use a different
backend: counsel-git-grep, counsel-ag, counsel-ack,
counsel-pt. But counsel-rg is the fastest, especially when I have
to deal with Git repositories that are 2Gb in size (short explanation:
it's a Perforce repo with a bunch of binaries, because why not; and
I'm using git-p4 to interact with
it).
Using .ignore with ripgrep
Adding an .ignore file to the root of your project can really speed
up your searches. In my sample project, I went from 10k files to less
than 500 files.
Example content:
/TAGS
*.min.js*
/Build/Output/
/ThirdParty/
As you can see, both file patterns and directories are supported. One
other nifty thing that I discovered only recently is that you can use
ripgrep as the backed for counsel-git in addition to
counsel-rg. Which means the same .ignore file is used for both
commands. Here's the setting:
(setqcounsel-git-cmd"rg --files")
And here's my setting for counsel-rg:
(setqcounsel-rg-base-command"rg -i -M 120 --no-heading --line-number --color never %s .")
The main difference in comparison to the default
counsel-rg-base-command is -M 120 which means: truncate all lines
that are longer than 120 characters. This is really helpful when Emacs
is accepting input from ripgrep: a megabyte long line of minified JS
is not only useless since you can't see it whole, but it will also
likely hang Emacs for a while.
Outro
I hope you found these bits of info useful. Happy hacking!
This week I made a mistake that ultimately enlightened me about the
nature of function objects in Emacs Lisp. There are three kinds of
function objects, but they each behave very differently when evaluated
as objects.
But before we get to that, let’s talk about one of Emacs’
embarrassing, old missteps: eval-after-load.
Taming an old dragon
One of the long-standing issues with Emacs is that loading Emacs Lisp
files (.el and .elc) is a slow process, even when those files have
been byte compiled. There are a number of dirty hacks in place to deal
with this issue, and the biggest and nastiest of them all is the
dumper, also known as unexec.
The Emacs you routinely use throughout the day is actually a previous
instance of Emacs that’s been resurrected from the dead. Your undead
Emacs was probably created months, if not years, earlier, back when it
was originally compiled. The first stage of compiling Emacs is to
compile a minimal C core called temacs. The second stage is loading
a bunch of Emacs Lisp files, then dumping a memory image in an
unportable, platform-dependent way. On Linux, this actually requires
special hooks in glibc. The Emacs you know and love is this
dumped image loaded back into memory, continuing from where it left
off just after it was compiled. Regardless of your own feelings on the
matter, you have to admit this is a very lispy thing to do.
There are two notable costs to Emacs’ dumper:
The dumped image contains hard-coded memory addresses. This means
Emacs can’t be a Position Independent Executable (PIE). It can’t
take advantage of a security feature called Address Space Layout
Randomization (ASLR), which would increase the difficulty of
exploiting some classes of bugs. This might be
important to you if Emacs processes untrusted data, such as when it’s
used as a mail client, a web server or generally
parses data downloaded across the network.
It’s not possible to cross-compile Emacs since it can only be dumped
by running temacs on its target platform. As an experiment I’ve
attempted to dump the Windows version of Emacs on Linux using
Wine, but was unsuccessful.
The good news is that there’s a portable dumper in the works
that makes this a lot less nasty. If you’re adventurous, you can
already disable dumping and run temacs directly by setting
CANNOT_DUMP=yes at compile time. Be warned, though, that a
non-dumped Emacs takes several seconds, or worse, to initialize
before it even begins loading your own configuration. It’s also
somewhat buggy since it seems nobody ever runs it this way
productively.
The other major way Emacs users have worked around slow loading is
aggressive use of lazy loading, generally via autoloads. The major
package interactive entry points are defined ahead of time as stub
functions. These stubs, when invoked, load the full package, which
overrides the stub definition, then finally the stub re-invokes the
new definition with the same arguments.
To further assist with lazy loading, an evaluated defvar form will
not override an existing global variable binding. This means you can,
to a certain extent, configure a package before it’s loaded. The
package will not clobber any existing configuration when it loads.
This also explains the bizarre interfaces for the various hook
functions, like add-hook and run-hooks. These accept symbols — the
names of the variables — rather than values of those variables as
would normally be the case. The add-to-list function does the same
thing. It’s all intended to cooperate with lazy loading, where the
variable may not have been defined yet.
eval-after-load
Sometimes this isn’t enough and you need some some configuration to
take place after the package has been loaded, but without forcing it
to load early. That is, you need to tell Emacs “evaluate this code
after this particular package loads.” That’s where eval-after-load
comes into play, except for its fatal flaw: it takes the word “eval”
completely literally.
The first argument to eval-after-load is the name of a package. Fair
enough. The second argument is a form that will be passed to eval
after that package is loaded. Now hold on a minute. The general rule
of thumb is that if you’re calling eval, you’re probably doing
something seriously wrong, and this function is no exception. This is
completely the wrong mechanism for the task.
The second argument should have been a function — either a (sharp
quoted) symbol or a function object. And then instead of eval it
would be something more sensible, like funcall. Perhaps this
improved version would be named call-after-load or run-after-load.
This was all corrected in Emacs 25. If the second argument to
eval-after-load is a function — the result of applying functionp is
non-nil — then it uses funcall. There’s also a new macro,
with-eval-after-load, to package it all up nicely.
Though in both of these examples the compiler will likely warn about
httpd-mime-types not being defined. That’s a problem for another
day.
A workaround
But what if you need to use Emacs 24, as was the situation that
sparked this article? What can we do with the bad version of
eval-after-load? We could situate a lambda such that it’s evaluated,
but then smuggle the resulting function object into the form passed to
eval-after-load, all using a backquote.
;;; Note: this is subtly broken(eval-after-load'simple-httpd`(funcall,(lambda()(push'("c"."text/plain")httpd-mime-types)))
When everything is compiled, the backquoted form evalutes to this:
Where the second value (#[...]) is a byte-code object.
However, as the comment notes, this is subtly broken. A cleaner and
correct way to solve all this is with a named function. The damage
caused by eval-after-load will have been (mostly) minimized.
But, let’s go back to the anonymous function solution. What was broken
about it? It all has to do with evaluating function objects.
Evaluating function objects
So what happens when we evaluate an expression like the one above with
eval? Here’s what it looks like again.
(funcall#[...])
First, eval notices it’s been given a non-empty list, so it’s probably
a function call. The first argument is the name of the function to be
called (funcall) and the remaining elements are its arguments. But
each of these elements must be evaluated first, and the result of that
evaluation becomes the arguments.
Any value that isn’t a list or a symbol is self-evaluating. That is,
it evaluates to its own value:
(eval10);; => 10
If the value is a symbol, it’s treated as a variable. If the value is a
list, it goes through the function call process I’m describing (or one
of a number of other special cases, such as macro expansion, lambda
expressions, and special forms).
So, conceptually eval recurses on the function object #[...]. A
function object is not a list or a symbol, so it’s self-evaluating. No
problem.
;; Byte-code objects are self-evaluating(let((x(byte-compile(lambda()))))(eqx(evalx)));; => t
What if this code wasn’t compiled? Rather than a byte-code object,
we’d have some other kind of function object for the interpreter.
Let’s examine the dynamic scope (shudder) case. Here, a lambda
appears to evaluate to itself, but appearances can be deceiving:
(eval(lambda());; => (lambda ())
However, this is not self-evaluation. Lambda expressions are not
self-evaluating. It’s merely coincidence that the result of
evaluating a lambda expression looks like the original expression.
This is just how the Emacs Lisp interpreter is currently implemented
and, strictly speaking, it’s an implementation detail that just so
happens to be mostly compatible with byte-code objects being
self-evaluating. It would be a mistake to rely on this.
Instead, dynamic scope lambda expression evaluation is
idempotent. Applying eval to the result will return
an equal, but not identical (eq), expression. In contrast, a
self-evaluating value is also idempotent under evaluation, but with
eq results.
;; Not self-evaluating:(let((x'(lambda())))(eqx(evalx)));; => nil;; Evaluation is idempotent:(let((x'(lambda())))(equalx(evalx)));; => t(let((x'(lambda())))(equalx(eval(evalx))));; => t
So, with dynamic scope, the subtly broken backquote example will still
work, but only by sheer luck. Under lexical scope, the situation isn’t
so lucky:
These interpreted lambda functions are neither self-evaluating nor
idempotent. Passing t as the second argument to eval tells it to
use lexical scope, as shown below:
;; Not self-evaluating:(let((x'(lambda())))(eqx(evalxt)));; => nil;; Not idempotent:(let((x'(lambda())))(equalx(evalxt)));; => nil(let((x'(lambda())))(equalx(eval(evalxt)t)));; error: (void-function closure)
I can imagine an implementation of Emacs Lisp where dynamic
scope lambda expressions are in the same boat, where they’re not even
idempotent. For example:
Most Emacs Lisp would work just fine under this change, and only code
that makes some kind of logical mistake — where there’s nested
evaluation of lambda expressions — would break. This essentially
already happened when lots of code was quietly switched over to
lexical scope after Emacs 24. Lambda idempotency was lost and
well-written code didn’t notice.
There’s a temptation here for Emacs to define a closure function or
special form that would allow interpreter closure objects to be either
self-evaluating or idempotent. This would be a mistake. It would only
serve as a hack that covers up logical mistakes that lead to nested
evaluation. Much better to catch those problems early.
Solving the problem with one character
So how do we fix the subtly broken example? With a strategically
placed quote right before the comma.
The quote prevents eval from evaluating the function object, which
would be either needless or harmful. There’s also an argument to be
made that this is a perfect situation for a sharp-quote (#'), which
exists to quote functions.
So your Emacs package has grown beyond a dozen or so lines of code, and
the data it manages is now structured and heterogeneous. Informal plain
old lists, the bread and butter of any lisp, are not longer cutting it.
You really need to cleanly abstract this structure, both for your own
organizational sake any for anyone reading your code.
With informal lists as structures, you might regularly ask questions
like, “Was the ‘name’ slot stored in the third list element, or was
it the fourth element?” A plist or alist helps with this problem, but
those are better suited for informal, externally-supplied data, not
for internal structures with fixed slots. Occasionally someone
suggests using hash tables as structures, but Emacs Lisp’s hash tables
are much too heavy for this. Hash tables are more appropriate when
keys themselves are data.
Defining a data structure from scratch
Imagine a refrigerator package that manages a collection of food in a
refrigerator. A food item could be structured as a plain old list,
with slots at specific positions.
Note the use of (nth 2 item) at the end, used to get the item’s
weight. That magic number 2 is easy to mess up. Even worse, if lots of
code accesses “weight” this way, then future extensions will be
inhibited. Defining some accessor functions solves this problem.
The defsubst defines an inline function, so there’s effectively no
additional run-time costs for these accessors compared to a bare
nth. Since these only cover getting slots, we should also define
some setters using the built-in gv (generalized variable) package.
This makes each slot setf-able. Generalized variables are great for
simplifying APIs, since otherwise there would need to be an equal
number of setter functions (fridge-item-set-name, etc.). With
generalized variables, both are at the same entrypoint:
(setf(fridge-item-nameitem)"Eggs")
There are still two more significant improvements.
As far as Emacs Lisp is concerned, this isn’t a real type. The
type-ness of it is just a fiction created by the conventions of the
package. It would be easy to make the mistake of passing an
arbitrary list to these fridge-item functions, and the mistake
wouldn’t be caught so long as that list has at least three items.
An common solution is to add a type tag: a symbol at the
beginning of the structure that identifies it.
It’s still a linked list, and nth has to walk the list (i.e.
O(n)) to retrieve items. It would be much more efficient to use a
vector, turning this into an efficient O(1) operation.
Addressing both of these at once:
(defunfridge-item-create(nameexpiryweight)(vector'fridge-itemnameexpiryweight))(defsubstfridge-item-p(object)(and(vectorpobject)(=(lengthobject)4)(eq'fridge-item(arefobject0))))(defsubstfridge-item-name(item)(unless(fridge-item-pitem)(signal'wrong-type-argument(list'fridge-itemitem)))(arefitem1))(defsubstfridge-item-name--set(itemvalue)(unless(fridge-item-pitem)(signal'wrong-type-argument(list'fridge-itemitem)))(setf(arefitem1)value))(gv-define-setterfridge-item-name(valueitem)`(fridge-item-name--set,item,value));; And so on for expiry and weight...
As long as fridge-mean-weight uses the fridge-item-weight
accessor, it continues to work unmodified across all these changes.
But, whew, that’s quite a lot of boilerplate to write and maintain
for each data structure in our package! Boilerplate code generation is
a perfect candidate for a macro definition. Luckily for us, Emacs
already defines a macro to generate all this code: cl-defstruct.
In Emacs 25 and earlier, this innocent looking definition expands into
essentially all the above code. The code it generates is expressed in
the most optimal form for its version of Emacs, and it
exploits many of the available optimizations by using function
declarations such as side-effect-free and error-free. It’s
configurable, too, allowing for the exclusion of a type tag (:named)
— discarding all the type checks — or using a list rather than a
vector as the underlying structure (:type). As a crude form of
structural inheritance, it even allows for directly embedding other
structures (:include).
Two pitfalls
There a couple pitfalls, though. First, for historical reasons, the
macro will define two namespace-unfriendly functions: make-NAME and
copy-NAME. I always override these, preferring the -create
convention for the constructor, and tossing the copier since it’s
either useless or, worse, semantically wrong.
If the constructor needs to be more sophisticated than just setting
slots, it’s common to define a “private” constructor (double dash in
the name) and wrap it with a “public” constructor that has some
behavior.
The other pitfall is related to printing. In Emacs 25 and earlier,
types defined by cl-defstruct are still only types by convention.
They’re really just vectors as far as Emacs Lisp is concerned. One
benefit from this is that printing and reading these
structures is “free” because vectors are printable. It’s trivial to
serialize cl-defstruct structures out to a file. This is exactly
how the Elfeed database works.
The pitfall is that once a structure has been serialized, there’s no
more changing the cl-defstruct definition. It’s now a file format
definition, so the slots are locked in place. Forever.
Emacs 26 throws a wrench in all this, though it’s worth it in the long
run. There’s a new primitive type in Emacs 26 with its own reader
syntax: records. This is similar to hash tables becoming first class
in the reader in Emacs 23.2. In Emacs 26, cl-defstruct uses
records instead of vectors.
So far slots are still accessed using aref, and all the type
checking still happens in Emacs Lisp. The only practical change is the
record function is used in place of the vector function when
allocating a structure. But it does pave the way for more interesting
things in the future.
The major short-term downside is that this breaks printed compatibility
across the Emacs 25/26 boundary. The cl-old-struct-compat-mode
function can be used for some degree of backwards, but not forwards,
compatibility. Emacs 26 can read and use some structures printed by
Emacs 25 and earlier, but the reverse will never be true. This issue
initially tripped up Emacs’ built-in packages, and when Emacs 26
is released we’ll see more of these issues arise in external packages.
Dynamic dispatch
Prior to Emacs 25, the major built-in package for dynamic dispatch —
functions that specialize on the run-time type of their arguments — was
EIEIO, though it only supported single dispatch (specializing on a
single argument). EIEIO brought much of the Common Lisp Object System
(CLOS) to Emacs Lisp, including classes and methods.
Emacs 25 introduced a more sophisticated dynamic dispatch package
called cl-generic. It focuses only on dynamic dispatch and supports
multiple dispatch, completely replacing the dynamic dispatch portion
of EIEIO. Since cl-defstruct does inheritance and cl-generic does
dynamic dispatch, there’s not really much left for EIEIO — besides bad
ideas like multiple inheritance and method combination.
Without either of these packages, the most direct way to build single
dispatch on top of cl-defstruct would be to shove a function in one
of the slots. Then the “method” is just a wrapper that call this
function.
Since cl-generic is aware of the types created by cl-defstruct,
functions can specialize on them as if they were native types. It’s a
lot simpler to let cl-generic do all the hard work. The people reading
your code will appreciate it, too:
The majority of the time a simple cl-defstruct will fulfill your
needs, keeping in mind the gotcha with the constructor and copier
names. Its use should feel almost as natural as defining functions.
Update: This article was featured on BSD Now 233 (starting
at 21:38).
For some time Elfeed was experiencing a strange, spurious
failure. Every so often users were seeing an error (spoiler
warning) when updating feeds: “error in process sentinel: Search
failed.” If you use Elfeed, you might have even seen this yourself.
From the surface it appeared that curl, tasked with the
responsibility for downloading feed data, was producing
incomplete output despite reporting a successful run. Since the run
was successful, Elfeed assumed certain data was in curl’s output
buffer, but, since it wasn’t, it failed hard.
Unfortunately this issue was not reproducible. Manually running curl
outside of Emacs never revealed any issues. Asking Elfeed to retry
fetching the feeds would work fine. The issue would only randomly rear
its head when Elfeed was fetching many feeds in parallel, under
stress. By the time the error was discovered, the curl process had
exited and vital debugging information was lost. Considering that
this was likely to be a bug in Emacs itself, there really wasn’t a
reliable way to capture the necessary debugging information from
within Emacs Lisp. And, indeed, this later proved to be the case.
A quick-and-dirty work around is to use condition-case to catch and
swallow the error. When the bizarre issue shows up, rather than fail
badly in front of the user, Elfeed could attempt to swallow the error
— assuming it can be reliably detected — and treat the fetch as simply
a failure. That didn’t sit comfortably with me. Elfeed had done its
due diligence checking for errors already. Someone was lying to
Elfeed, and I intended to catch them with their pants on fire.
Someday.
I’d just need to witness the bug on one of my own machines. Elfeed is
part of my daily routine, so surely I’d have to experience this issue
myself someday. My plan was, should that day come, to run a modified
Elfeed, instrumented to capture extra data. I would have also routinely
run Emacs under GDB so that I could inspect the failure more deeply.
Over the holidays I re-discovered Bryan Cantrill, a systems
software engineer who worked for Sun between 1996 and 2010, and is most
well known for DTrace. My first exposure to him was in a BSD
Now interview in 2015. I had re-watched that interview and decided
there was a lot more I had to learn from him. He’s become a personal
hero to me. So I scoured the internet for more of his writing and
talks. Besides what I’ve already linked in this article, here
are a couple more great presentations:
Some interesting operating system technology came out of Sun during
its final 15 or so years — most notably DTrace and ZFS — and Bryan
speaks about it passionately. Almost as a matter of luck, most of it
survived the Oracle acquisition thanks to Sun releasing it as open
source in just the nick of time. Otherwise it would have been lost
forever. The scattered ex-Sun employees, still passionate about their
prior work at Sun, along with some of their old customers have since
picked up the pieces and kept going as a community under the name
illumos. It’s like an open source flotilla.
Naturally I wanted to get my hands on this stuff to try it out for
myself. Is it really as good as they say? Normally I stick to Linux,
but it (generally) doesn’t have these Sun technologies. The main
reason is license incompatibility. Sun released its code under the
CDDL, which is incompatible with the GPL. Ubuntu doesinfamously include ZFS, but other distributions are
unwilling to take that risk. Porting DTrace is a serious undertaking
since it’s got its fingers throughout the kernel, which also makes the
licensing issues even more complicated.
Linux has a reputation for Not Invented Here (NIH) syndrome, and these
licensing issues certainly contribute to that. Rather than adopt ZFS
and DTrace, they’ve been reinvented from scratch: btrfs instead of
ZFS, and a slew of partial options instead of DTrace.
Normally I’m most interested in system call tracing, and my go to is
strace, though it certainly has its limitations — including
this situation of debugging curl under Emacs. Another famous example
of NIH is Linux’s epoll(2), which is a brokenversion of BSD kqueue(2).
So, if I want to try these for myself, I’ll need to install a
different operating system. I’ve dabbled with OmniOS, an OS
built on illumos, in virtual machines, using it as an alien
environment to test some of my software (e.g. enchive).
OmniOS has a philosophy called Keep Your Software To Yourself
(KYSTY), which is really just code for “we don’t do packaging.”
Honestly, you can’t blame them since they’re a tiny community.
The best solution to this is probably pkgsrc, which is
essentially a universal packaging system. Otherwise you’re on your
own.
There’s also openindiana, which is a more friendly
desktop-oriented illumos distribution. Still, the short of it is that
you’re very much on your own when things don’t work. The situation is
like running Linux a couple decades ago, when it was still difficult
to do.
If you’re interested in trying DTrace, the easiest option these days is
probably FreeBSD. It’s got a big, active community, thorough
documentation, and a huge selection of packages. Its license (the BSD
license, duh) is compatible with the CDDL, so both ZFS and DTrace have
been ported to FreeBSD.
What is DTrace?
I’ve done all this talking but haven’t yet described what DTrace
really is. I won’t pretend to write my own tutorial, but I’ll
provide enough information to follow along. DTrace is a tracing
framework for debugging production systems in real time, both for
the kernel and for applications. The “production systems” part means
it’s stable and safe — using DTrace won’t put your system at risk of
crashing or damaging data. The “real time” part means it has little
impact on performance. You can use DTrace on live, active systems with
little impact. Both of these core design principles are vital for
troubleshooting those really tricky bugs that only show up in
production.
There are DTrace probes scattered all throughout the system: on
system calls, scheduler events, networking events, process events,
signals, virtual memory events, etc. Using a specialized language
called D (unrelated to the general purpose programming language D),
you can dynamically add behavior at these instrumentation points.
Generally the behavior is to capture information, but it can also
manipulate the event being traced.
Each probe is fully identified by a 4-tuple delimited by colons:
provider, module, function, and probe name. An empty element denotes a
sort of wildcard. For example, syscall::open:entry is a probe at the
beginning (i.e. “entry”) of open(2). syscall:::entry matches all
system call entry probes.
Unlike strace on Linux which monitors a specific process, DTrace
applies to the entire system when active. To run curl under strace
from Emacs, I’d have to modify Emacs’ behavior to do so. With DTrace I
can instrument every curl process without making a single change to
Emacs, and with negligible impact to Emacs. That’s a big deal.
So, when it comes to this Elfeed issue, FreeBSD is much better poised
for debugging the problem. All I have to do is catch it in the act.
However, it’s been months since that bug report and I’m not really
making this connection yet. I’m just hoping I eventually find an
interesting problem where I can apply DTrace.
FreeBSD on a Raspberry Pi 2
So I’ve settled in FreeBSD as the playground for these technologies, I
just have to decide where. I could always run it in a virtual machine,
but it’s always more interesting to try things out on real hardware.
FreeBSD supports the Raspberry Pi 2 as a Tier 2 system, and
I had a Raspberry Pi 2 sitting around collecting dust, so I put it to
use.
I wrote the image to an SD card, and for a few days I stretched my
legs on this new system. I cloned a couple dozen of my own git
repositories, ran the builds and the tests, and just got a feel for
things. I tried out the ports system for the first time, mainly to
discover that the low-powered Raspberry Pi 2 takes days to build some
of the packages I want to try.
I mostly program in Vim these days, so it’s some days before I
even set up Emacs. Eventually I do build Emacs, clone my
configuration, fire it up, and give Elfeed a spin.
And that’s when the “search failed” bug strikes! Not just once, but
dozens of times. Perfect! This low-powered platform is the jackpot for
this particular bug, triggering it left and right. Given that I’ve got
DTrace at my disposal, it’s the perfect place to debug this.
Something is lying to Elfeed and DTrace will play the judge.
Before I dive in I see three possibilities:
curl is reporting success but truncating its output.
Emacs is quietly truncating curl’s output.
Emacs is misinterpreting curl’s exit status.
With Dtrace I can observe what every curl process writes to Emacs, and
I can also double check curl’s exit status. I come up with the
following (newbie) DTrace script:
The /execname == "curl"/ is a predicate that (obviously) causes the
behavior to only fire for curl processes. The first probe has DTrace
print a line for every write(2) from curl. arg0, arg1, and
arg2 correspond to the arguments of write(2): fd, buf, count. It
logs the process ID (pid) of the write, the length of the write, and
the actual contents written. Remember that these curl processes are
run in parallel by Emacs, so the pid allows me to associate the
separate writes and the exit status.
The second probe prints the pid and the exit status (the first argument
to exit(2)).
I also want to compare this to exactly what is delivered to Elfeed when
curl exits, so I modify the process sentinel — the callback
that handles a subprocess exiting — to call write-file before any
action is taken. I can compare these buffer dumps to the logs produced
by DTrace.
There are two important findings.
First, when the “search failed” bug occurs, the buffer was completely
empty (95% of the time) or truncated at the end of the HTTP headers
(5% of the time), right at the blank line. DTrace indicates that curl
did its job to the full, so it’s Emacs who’s the liar. It’s not
delivering all of curl’s data to Elfeed. That’s pretty annoying.
Second, curl was line-buffered. Each line was a separate,
independent write(2). I was certainly not expecting this. Normally
the C library only does line buffering when the output is a terminal.
That’s because it’s guessing a user may be watching, expecting the
output to arrive a line at a time.
Here’s a sample of what it looked like in the log:
Oh. That’s right. This is the same problem I ran into four years
ago when writing EmacSQL. By default Emacs connects to
subprocesses through a pseudo-terminal (pty). I called this a mistake
in Emacs back then, and I still stand by that claim. The pty causes
weird, annoying problems for little benefit:
Interpreting control characters. Hope you weren’t transferring binary
data!
Subprocesses will generally get line buffered. This makes them
slower, though in some situations it might be desirable.
Stdout and stderr get mixed together. (Optional since Emacs 25.)
New! There’s a bug somewhere in Emacs that causes truncation when
ptys are used heavily in parallel.
Just from eyeballing the DTrace log I knew what to do: dump the pty
and switch to a pipe. This is controlled with the
process-connection-type variable, and fixing it is a
one-liner.
Not only did this completely resolve the truncation issue, Elfeed is
noticeably faster at fetching feeds on all machines. It’s no longer
receiving mountains of XML one line at a time, like sucking pudding
through a straw. It’s now quite zippy even on my Raspberry Pi 2, which
had never been the case before (without the “search failed” bug).
Even if you were never affected by this bug, you will benefit from the
fix.
I haven’t officially reported this as an Emacs bug yet because
reproducibility is still an issue. It needs something better than
“fire off a bunch of HTTP requests across the internet in parallel
from a Raspberry Pi.”
The fix reminds me of that old boilermaker story about
charging a lot of money just to swing a hammer. Once the problem
arose, DTrace quickly helped to identify the place to hit Emacs with
the hammer.
Finally, a big thanks to alphapapa for originally taking the time to
report this bug months ago.
Besides binding 0..9 to complete their
corresponding candidate, it also un-binds RET and binds
SPC to close the company popup.
Actual code
(defunora-company-number()"Forward to `company-complete-number'.Unless the number is potentially part of the candidate.In that case, insert the number."(interactive)(let*((k(this-command-keys))(re(concat"^"company-prefixk)))(if(cl-find-if(lambda(s)(string-matchres))company-candidates)(self-insert-command1)(company-complete-number(string-to-numberk)))))
Initially, I would just bind company-complete-number. The problem
with that was that if my candidate list was ("var0" "var1" "var2"),
then entering 1 means:
select the first candidate (i.e. "var0"), instead of:
insert "1", resulting in "var1", i.e. the second candidate.
My customization will now check company-candidates—the list of
possible completions—for the above mentioned conflict. And if
it's detected, the key pressed will be inserted instead of being used
to select a candidate.
Outro
Looking at git-log, I've been using company-complete-number for at
least 3 years now. It's quite useful, and now also more seamless,
since I don't have to type e.g. C-q 2 any more. In any
case, thanks to the author and the contributors of
company-mode. Merry Christmas and happy hacking in the New Year!
There was recently some interesting discussion about correctly
using backquotes to express a mixture of data and code. Since lambda
expressions seem to evaluate to themselves, what’s the difference?
For example, an association list of operations:
It looks like it would work, and indeed it does work in this case.
However, there are good reasons to actually evaluate those lambda
expressions. Eventually invoking the lambda expressions in the quoted
form above are equivalent to using eval. So, instead, prefer the
backquote form:
There are a lot of interesting things to say about this, but let’s
first reduce it to two very simple cases:
(lambda(x)x)'(lambda(x)x)
What’s the difference between these two forms? The first is a lambda
expression, and it evaluates to a function object. The other is a quoted
list that looks like a lambda expression, and it evaluates to a list —
a piece of data.
A naive evaluation of these expressions in *scratch* (C-x C-e)
suggests they are are identical, and so it would seem that quoting a
lambda expression doesn’t really matter:
However, there are two common situations where this is not the case:
byte compilation and lexical scope.
Lambda under byte compilation
It’s a little trickier to evaluate these forms byte compiled in the
scratch buffer since that doesn’t happen automatically. But if it did,
it would look like this:
The #[...] is the syntax for a byte-code function object. As
discussed in detail in my byte-code internals article, it’s a
special vector object that contains byte-code, and other metadata, for
evaluation by Emacs’ virtual stack machine. Elisp is one of very few
languages with readable function objects, and this feature is
core to its ahead-of-time byte compilation.
The quote, by definition, prevents evaluation, and so inhibits byte
compilation of the lambda expression. It’s vital that the byte compiler
does not try to guess the programmer’s intent and compile the expression
anyway, since that would interfere with lists that just so happen to
look like lambda expressions — i.e. any list containing the lambda
symbol.
There are three reasons you want your lambda expressions to get byte
compiled:
Byte-compiled functions are significantly faster. That’s the main
purpose for byte compilation after all.
The compiler performs static checks, producing warnings and errors
ahead of time. This lets you spot certain classes of problems before
they occur. The static analysis is even better under lexical scope due
to its tighter semantics.
Under lexical scope, byte-compiled closures may use less memory. More
specifically, they won’t accidentally keep objects alive longer than
necessary. I’ve never seen a name for this implementation issue, but I
call it overcapturing. More on this later.
While it’s common for personal configurations to skip byte compilation,
Elisp should still generally be written as if it were going to be byte
compiled. General rule of thumb: Ensure your lambda expressions are
actually evaluated.
Under lexical scope, lambda expressions evaluate to closures.
Closures capture their lexical environment in their closure object —
nothing in this particular case. It’s a type of function object,
making it a valid first argument to funcall.
Since the quote prevents the second expression from being evaluated,
semantically it evaluates to a list that just so happens to look like
a (non-closure) function object. Invoking a data object as a
function is like using eval — i.e. executing data as code.
Everyone already knows eval should not be used lightly.
It’s a little more interesting to look at a closure that actually
captures a variable, so here’s a definition for constantly, a
higher-order function that returns a closure that accepts any number of
arguments and returns a particular constant:
(defunconstantly(x)(lambda(&rest_)x))
Without byte compiling it, here’s an example of its return value:
The environment has been captured as an association list (with a
trailing t), and we can plainly see that the variable x is bound to
the symbol :foo in this closure. Consider that we could manipulate
this data structure (e.g. setcdr or setf) to change the binding of
x for this closure. This is essentially how closures mutate their own
environment. Moreover, closures from the same environment share
structure, so such mutations are also shared. More on this later.
Semantically, closures are distinct objects (via eq), even if the
variables they close over are bound to the same value. This is because
they each have a distinct environment attached to them, even if in
some invisible way.
(eq(constantly:foo)(constantly:foo));; => nil
Without byte compilation, this is true even when there’s no lexical
environment to capture:
The byte compiler is smart, though. As an optimization, the
same closure object is reused when possible, avoiding unnecessary
work, including multiple object allocations. Though this is a bit of
an abstraction leak. A function can (ab)use this to introspect whether
it’s been byte compiled:
(defunhave-i-been-compiled-p()(let((funcs(vectornilnil)))(dotimes(i2)(setf(areffuncsi)(lambda())))(eq(areffuncs0)(areffuncs1))))(have-i-been-compiled-p);; => nil(byte-compile'have-i-been-compiled-p)(have-i-been-compiled-p);; => t
The trick here is to evaluate the exact same non-capturing lambda
expression twice, which requires a loop (or at least some sort of
branch). Semantically we should think of these closures as being
distinct objects, but, if we squint our eyes a bit, we can see the
effects of the behind-the-scenes optimization.
Don’t actually do this in practice, of course. That’s what
byte-code-function-p is for, which won’t rely on a subtle
implementation detail.
Overcapturing
I mentioned before that one of the potential gotchas of not byte
compiling your lambda expressions is overcapturing closure variables in
the interpreter.
To evaluate lisp code, Emacs has both an interpreter and a virtual
machine. The interpreter evaluates code in list form: cons cells,
numbers, symbols, etc. The byte compiler is like the interpreter, but
instead of directly executing those forms, it emits byte-code that, when
evaluated by the virtual machine, produces identical visible results to
the interpreter — in theory.
What this means is that Emacs contains two different implementations
of Emacs Lisp, one in the interpreter and one in the byte compiler.
The Emacs developers have been maintaining and expanding these
implementations side-by-side for decades. A pitfall to this approach
is that the implementations can, and do, diverge in their behavior.
We saw this above with that introspective function, and it comes up
in practice with advice.
Another way they diverge is in closure variable capture. For example:
Notice that the closure captured y even though it’s unnecessary.
This is because the interpreter doesn’t, and shouldn’t, take the time
to analyze the body of the lambda to determine which variables should
be captured. That would need to happen at run-time each time the
lambda is evaluated, which would make the interpreter much slower.
Overcapturing can get pretty messy if macros are introducing their own
hidden variables.
On the other hand, the byte compiler can do this analysis just once at
compile-time. And it’s already doing the analysis as part of its job.
It can avoid this problem easily:
It’s clear that :some-big-value isn’t present in the closure.
But… how does this work?
How byte compiled closures are constructed
Recall from the internals article that the four core elements of a
byte-code function object are:
Parameter specification
Byte-code string (opcodes)
Constants vector
Maximum stack usage
While a closure seems like compiling a whole new function each time
the lambda expression is evaluated, there’s actually not that much to
it! Namely, the behavior of the function remains the same. Only
the closed-over environment changes.
What this means is that closures produced by a common lambda
expression can all share the same byte-code string (second element).
Their bodies are identical, so they compile to the same byte-code.
Where they differ are in their constants vector (third element), which
gets filled out according to the closed over environment. It’s clear
just from examining the outputs:
constantly has three of the four components of the closure in its own
constant pool. Its job is to construct the constants vector, and then
assemble the whole thing into a byte-code function object (#[...]).
Here it is with M-x disassemble:
(Note: since byte compiler doesn’t produce perfectly optimal code, I’ve
simplified it for this discussion.)
It pushes most of its constants on the stack. Then the stack-ref 5 (5)
puts x on the stack. Then it calls vector to create the constants
vector (6). Finally, it constructs the function object (#[...]) by
calling make-byte-code (8).
Since this might be clearer, here’s the same thing expressed back in
terms of Elisp:
To see the disassembly of the closure’s byte-code:
(disassemble(constantly:x))
The result isn’t very surprising:
0 constant :x
1 return
Things get a little more interesting when mutation is involved. Consider
this adder closure generator, which mutates its environment every time
it’s called:
In theory, this closure could operate by mutating its constants vector
directly. But that wouldn’t be much of a constants vector, now would
it!? Instead, mutated variables are boxed inside a cons cell. Closures
don’t share constant vectors, so the main reason for boxing is to share
variables between closures from the same environment. That is, they have
the same cons in each of their constant vectors.
There’s no equivalent Elisp for the closure in adder, so here’s the
disassembly:
It puts two references to boxed integer on the stack (constant,
dup), unboxes the top one (car-safe), increments that unboxed
integer, stores it back in the box (setcar) via the bottom reference,
leaving the incremented value behind to be returned.
This all gets a little more interesting when closures interact:
This is starting to resemble object oriented programming, with methods
acting upon fields stored in a common, closed-over environment.
All three closures share a common variable, total. Since I didn’t
use print-circle, this isn’t obvious from the last result, but each
of those (0) conses are the same object. When one closure mutates
the box, they all see the change. Here’s essentially how fancy-adder
is transformed by the byte compiler:
The backquote in the original fancy-adder brings this article full
circle. This final example wouldn’t work correctly if those lambdas
weren’t evaluated properly.
On December 7, Patreon made an announcement about the change in their
transaction fee structure. The results as of December 10 speak for
themselves:
December 2017 summary: -$29 in pledges, -6 patrons
All leaving patrons marked "I'm not happy with Patreon's features or
services." as the reason for leaving, with quotes ranging from:
The billing changes are not great.
to:
Patreon's new fees are unacceptable
In this article, I will explore the currently available methods for
supporting sustainable Free Software development and compare their
transaction fees.
Here's what I collected so far, vs the actual money spent by the
contributors:
2017-11-01: $140.42 / $162.50 = 86.41%
2017-12-01: $163.05 / $187.50 = 86.96%
The numbers here are using the old Patreon rules that are going away
this month.
Real numbers
method
formula
charged
donated
fee
old Patreon
???
$1.00
$0.86
14%
new Patreon
7.9% + $0.35
$1.38
$0.95
31%
$2.41
$1.90
21%
$5.50
$4.75
14%
OpenCollective
12.9% + $0.30
$1.33
$0.90
32%
$2.36
$1.80
24%
$5.45
$4.50
18%
Flattr
16.5%
$1.00
$0.84
17%
$2.00
$1.67
17%
$5.00
$4.18
17%
Liberapay
0.585%
$1.00
$0.99
1%
On Patreon
Just like everyone else, I'm not happy with the incoming change to the
Patreon fees. But even after the change, it's still a better deal than
OpenCollective, which is used quite successfully e.g. by
CIDER.
Just to restate the numbers in the table, if all backers give $1
(which is the majority currently, and I actually would generally
prefer 5 new $1 backers over 1 new $5 backer), with the old system I
get $0.86, while with the new system it's $0.69. That's more than 100%
increase in transaction fees.
On OpenCollective
It's more expensive than the new Patreon fees in every category or
scenario.
On Flattr
Flattr is in the same bucket as Patreon, except with slightly lower
fees currently. Their default plan sounds absolutely ridiculous to me:
you install a browser plug-in so that a for-profit corporation can
track which websites you visit most often in order to distribute the
payments you give them among those websites.
If it were a completely local tool which doesn't upload any data on
the internet and instead gives you a monthly report to adjust your
donations, it would have been a good enough tool. Maybe with some
adjustments for mind-share bubbles, which result in prominent projects
getting more rewards than they can handle, while small projects fade
away into obscurity without getting a chance. But right now it's
completely crazy. Still, if you don't install the plug-in, you can
probably still use Flattr and it will work similarly to Patreon.
I made an account, just in case, but I
wouldn't recommend going to Flattr unless you're already there, or the
first impression it made on me is wrong.
On Paypal
Paypal is OK in a way, since a lot of the time the organizations like
Patreon are just middle men on top of Paypal. On the other hand,
there's no way to set up recurring donations. And it's harder for me
to plan decisions regarding my livelihood if I don't know at least
approximately the sum I'll be getting next month.
My account, in case you want to make a lump sum donation:
paypal.me/aboabo.
On Bitcoin
Bitcoin is similar to Paypal, except it also:
has a very bad impact on the environment,
is a speculative bubble that supports either earning or losing money
without actually providing value to the society.
I prefer to stay away from Bitcoin.
Summary
Liberapay sounds almost too good to
be true. At the same time, their fees are very realistic, you could
almost say optimal, since there are no fees for transfers between
members. So you can spend either €20.64 (via card) or €20.12 (via bank
wire) to charge €20 into your account and give me €1 per month at no
further cost. If you change your mind after one month, you can
withdraw your remaining €19 for free if you use a SEPA (Single Euro
Payments Area) bank.
If I set out today to set up a service similar to Liberapay, even with
my best intentions and the most optimistic expectations, I don't see
how a better offer could be made. I recommend anyone who wants to
support me to try it out. And, of course, I will report back with real
numbers if anything comes out of it.
Thanks to all my patrons for their former and ongoing support. At one
point we were at 30% of the monthly goal (25% atm.). This made me very
excited and optimistic about the future. Although I'm doing Free
Software for almost 5 years now, it's actually 3 years in academia and
2 years in industry. Right now, I'm feeling a burnout looming over the
horizon, and I was really hoping to avoid it by spending less time
working at for-profit corporations. Any help, either monetary or
advice is appreciated. If you're a part of a Software Engineering or a
Research collective that makes you feel inspired instead of exhausted
in the evening and you have open positions in EU or on remote, have a
look at my LinkedIn -
maybe we could become colleagues in the future. I'll accept
connections from anyone - if you're reading this blog, we probably
have a lot in common; and it's always better together.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
Diego Berrocal websitetwittergithub will be
presenting Getting Closer to Using Emacs as an IDE(or better)
I’ll demonstrate how my setup has become more of an IDE gradually
because of the different plugins I use and how it has changed
dramatically thanks to the Language Server Protocol by Microsoft
(which will be the focus of this talk).
Ivy is a completion method that's
similar to Ido, but with emphasis on simplicity and customizability.
Overview
The current release constitutes of 280 commits and 8 months of
progress since 0.9.0. Many issues ranging from
#952 to
#1336 were fixed.
The number of people who contributed code as grown to
91; thanks,
everyone!
Details on changes
Changelog.org has been a part of the repository since 0.6.0, you
can get the details of the current and past changes:
Many improvements are incremental and don't require any extra code to
enable. I'll go over a few selected features that require a bit of
information to make a good use of them.
Selectable prompt
Off by default. You can turn it on like so:
(setqivy-use-selectable-promptt)
After this, your current input becomes selectable as a
candidate. Press C-p when you're on the first candidate to
select your input instead.
This solves the long standing issue of e.g. creating a file or
a directory foo when a file foobar already exists. Previously, the
only solution was to use C-M-j. It's still available, but
now you can also select your input with C-p and press
RET.
New global actions for ivy
ivy-set-actions was used to enable the following bindings:
Press M-o w to copy the current candidate to the kill ring.
Press M-o i to insert the current candidate into the buffer.
These bindings are valid for any completion session by default.
Use C-d in ivy-occur buffers
Here's an example use-case: search your source code for a variable
name with e.g. counsel-rg and call ivy-occur (C-c C-o).
Suppose you get 10 results, only 4 of which are interesting. You can
now delete the uninteresting ones with C-d. Then maybe
check off the others with C-d as well as you complete them
one by one. A sort of a TODO list.
Similarly, if you want to go over variables to customize, you can call
counsel-describe-variable with input ^counsel-[^-] and then check
off the ones you have already examined with C-d.
Defcustoms to play with
Here's the list of new defcustom or defvar that might be
interesting to review:
counsel-async-filter-update-time
counsel-async-ignore-re
counsel-describe-function-function
counsel-describe-function-preselect
counsel-find-file-ignore-regexp
counsel-fzf-dir-function
counsel-git-grep-skip-counting-lines
counsel-git-log-split-string-re
counsel-url-expansions
ivy-auto-select-single-candidate
ivy-magic-slash-non-match-action
ivy--preferred-re-builders
ivy-truncate-lines
New Commands
14 new commands were added by me and many contributors. Here's the list:
If you didn't know, is a nifty tool that lets you encrypt things for use with Ansible. For some reason that I've not yet discovered, ansible-vault doesn't care if you've set EDITOR /usr/bin/emacs in your fish.config. To work around this, I use the following alias function:
I think ivy-occur (C-c C-o) is one of the coolest
features in ivy. It allows you
to save your current search into a new buffer. This has many uses:
get a full overview of all candidates
many useful modal bindings (q, j,
k, f) and mouse support
ability to manipulate candidates as text
save the search for later, with the option to refresh the search with g
go over candidates as a TODO list, using C-d to remove elements
Everything above works for any ivy-read session. But the most
powerful features come into play when ivy-occur gets customized for
a specific collection.
ivy-occur for grep-like functions
(ivy-set-occur'swiper'swiper-occur)
Thanks to this default customization, the resulting *ivy-occur
swiper* buffer is in ivy-occur-grep-mode which inherits from
grep-mode. Additionally, you can use
ivy-wgrep-change-to-wgrep-modeC-x C-q to edit the result
in-place - pressing C-x C-s will save the changes.
Similar customizations are available for counsel-git-grep,
counsel-ag, counsel-rg, and counsel-grep.
This makes C-c C-o open your candidates in the powerful
ibuffer, which adds additional info to your buffer list and allows
you to manipulate buffers easily.
For instance, to delete all matching buffers you can do C-c C-o
tD.
The source code is short enough to be included here:
(defunivy-switch-buffer-occur()"Occur function for `ivy-switch-buffer' using `ibuffer'."(ibuffernil(buffer-name)(list(cons'nameivy--old-re))))
The interface is quite simple: ivy-occur is responsible for
generating a new buffer, and the occur function
e.g. ivy-switch-buffer-occur is to fill that buffer with useful
info, based on the current search parameters like ivy-text and
ivy--old-re.
ivy-occur for counsel-find-file-like functions
This is a brand new feature that works for counsel-find-file,
counsel-git, and counsel-fzf (which itself is quite new, thanks to
@jojojames for contributing it).
Since these functions are used to complete file names, we obviously
want ivy-occur to open a Dired buffer.
Example 1
To delete all *.elc files in the current folder do:
C-x C-felc$C-c C-otDy.
Example 2
To copy all Org files in a Git project to some directory do:
M-xcounsel-gitorg$C-c C-otC.
Example 3
To get a list of videos to watch do:
M-xcounsel-fzfmp4$C-c C-o.
I can further e.g. mark 3 files with m and use r
to send these 3 files to vlc as a list. See this
post for my dired setup
that makes r work this way.
You can remove some files afterwards with the usual D or
dx. And to redisplay the buffer use g.
Outro
I hope you like the new feature. I had a really good few hours
figuring out how it should work exactly. Please consider joining my
72 patrons to give me the
opportunity to work on Free Software a lot more. Happy hacking!
Since docker.org has * Tasks, I just click the capture button in
Firefox and follow up with an immediate C-c C-c in Emacs.
The link is already in the right position, no need for an extra refile
step.
Using customize with orca
You can set up the capture rules using M-xcustomize-groupRETorcaRET.
Here's a screenshot:
As you see, the customization is a list of an arbitrary length, with
each element falling into one of three categories, each backed by an
Elisp function (orca-handler-current-buffer, orca-handler-file,
and orca-handler-match-url). Each function takes a different number
of arguments (one, two, and three, respectively) - they are all
annotated by the interface.
Here's the code that describes the expected :type to customize:
I hope you enjoy orca. I've submitted it to MELPA. Hopefully, it
will be available for an easy install very soon.
Org-mode is a beautiful thing, but my previous attempts to configure
it were huge config files of loosely related (i.e. the only thing in
common was Org-mode) stuff spanning hundreds of lines. Orca is an
improvement in this respect, since it focuses on a very narrow
domain. It still tries to be flexible (just like org-capture) - you
can plug in your own functions into orca-handler-list. But
initially, the flexibility can be constrained into the
customize-group interface, to allow for a self-documenting solution
that's easy to get started with. Happy hacking!
PS. Thanks to all my patrons for advancing my
Patreon campaign!
As of this writing, we're almost at the 25% mark
with 61 contributors.
Do you long for the days before Emacs 24.3 when flet was dynamically
scoped? Well, you probably shouldn’t since there are some very good
reasons lexical scope. But, still, a dynamically scoped flet
is situationally really useful, particularly in unit testing. The good
news is that it’s trivial to get this original behavior back without
relying on deprecated functions nor third-party packages.
But first, what is flet and what does it mean for it to be
dynamically scoped? The name stands for “function let” (or something
to that effect). It’s a macro to bind named functions within a local
scope, just as let binds variables within some local scope. It’s
provided by the now-deprecated cl package.
However, a gotcha here is that square is visible not just to the body
of norm but also to any function called directly or indirectly from
the flet body. That’s dynamic scope.
(flet((sqrt(v)(/v2))); close enough(norm22));; -> 4
Note: This works because sqrt hasn’t (yet?) been assigned a bytecode
opcode. One weakness with flet is that, due to being dynamically
scoped, it is unable to define or override functions whose calls
evaporate under byte compilation. For example, addition:
Since + has its own opcode, the function call is eliminated under
byte-compilation and flet can’t do its job. This is similar these
same functions being unadvisable.
cl-lib and cl-flet
The cl-lib package introduced in Emacs 24.3, replacing cl, adds a
namespace prefix, cl-, to all of these Common Lisp style functions.
In most cases this was the only change. One exception is cl-flet,
which has different semantics: It’s lexically scoped, just like in
Common Lisp. Its bindings aren’t visible outside of the cl-flet
body.
In most cases this is what you actually want. The old flet subtly
changes the environment for all functions called directly or
indirectly from its body.
Besides being cleaner and less error prone, cl-flet also doesn’t
have special exceptions for functions with assigned opcodes. At
macro-expansion time it walks the body, taking its action before the
byte-compiler can interfere.
In order for it to work properly, it’s essential that functions are
quoted with sharp-quotes (#') so that the macro can tell the
difference between functions and symbols. Just make a general habit of
sharp-quoting functions.
In unit testing, temporarily overriding functions for all of Emacs is
useful, so flet still has some uses. But it’s deprecated!
Unit testing with flet
Since Emacs can do anything, suppose there is an Emacs package that
makes sandwiches. In this package there’s an interactive function to
set the default sandwich cheese.
Since it’s interactive, it uses completing-read to prompt the user
for input. A unit test could call this function non-interactively, but
perhaps we’d also like to test the interactive path. The code inside
interactive occasionally gets messy and may warrant testing. It
would obviously be inconvenient to prompt the user for input during
testing, and it wouldn’t work at all in batch mode (-batch).
With flet we can stub out completing-read just for the unit test:
;;; -*- lexical-binding: t; -*-(ert-deftesttest-set-default-cheese();; protect original with dynamic binding(let(default-cheese);; simulate user entering "american"(flet((completing-read(&rest_)"american"))(call-interactively#'set-default-cheese)(should(eq'americandefault-cheese)))))
Since default-cheese was defined with defvar, it will be
dynamically scoped despite let normally using lexical scope in this
example. Both of the side effects of the tested function — setting a
global variable and prompting the user — are captured using a
combination of let and flet.
Since cl-flet is lexically scoped, it cannot serve this purpose. If
flet is deprecated and cl-flet can’t do the job, what’s the right
way to fix it? The answer lies in generalized variables.
cl-letf
What’s really happening inside flet is it’s globally binding a
function name to a different function, evaluating the body, and
rebinding it back to the original definition when the body completes.
It macro-expands to something like this:
The unwind-protect ensures the original function is rebound even if
the body of the call were to fail. This is very much a let-like
pattern, and I’m using symbol-function as a generalized variable via
setf. Is there a generalized variable version of let?
Yes! It’s called cl-letf! In this case the f suffix is analogous
to the f suffix in setf. That form above can be reduced to a more
general form:
Keep in mind that this suffers the exact same problem with
bytecode-assigned functions as flet, and for exactly the same
reasons. If completing-read were to ever be assigned its own opcode
then cl-letf would no longer work for this particular example.
Normally when I want to do anything with Emacs it's a matter of writing some Elisp code and poof, I've got what I want. Anything can be realized, it's usually just a matter of knowing which internal to tweak or what to implement. With that in mind, now that I'm getting into the swing of using I thought it would be great to have org-agenda-list as my default buffer when I open Emacs. As it turns out, simply doing something like (setq initial-buffer-choice 'org-agenda-list) won't yield the same results between daemon and non-daemon Emacs. Wheee!
In light of the recent success of the
Magit Kickstarter
(congratulations to @tarsius, by the way), I got a lot more optimistic about Free Software crowdfunding.
So I opened a Patreon account where you can support my work:
https://www.patreon.com/abo_abo.
The goal I set there is both optimistic and (hopefully) realistic: I'd
like to hack on Free Software 1 day per week indefinitely, reducing my
real world job days to 4 per week.
Ideally, I'd like to work on Free Software full time (one can dream),
but it doesn't look like that level of donations is attainable right
now. But I think I could accomplish a lot working a full day per week:
improve the level of maintenance of my current projects
polish and release a few projects I have in a semi-complete unreleased state
If you are a user of my work, don't feel any pressure to donate. We
are all here voluntarily: I publish because I enjoy it, you use the
software because you find it useful. But out there is the real world,
and, although I like my real world job enough, I can't say that would
I do it voluntarily if I had enough money to meet my needs.
If you do what you love, you'll never work a day in your life
I'd like to do what I love, and I wish you all the same. Happy hacking!
Although fine-tuning your slack notifications is already reason enough to run
slack in Emacs, that’s only the beginning. Once everything is up and running
you get to decide what you want out of your slack. Some of the snippets below
simply make up for missing functionality, other customize the package beyond
what you can do on the Slack Webapp.
Priorities first. The most important improvement you can implement is install
emojify-mode and turn it on for slack chats.
(add-hook'slack-mode-hook#'emojify-mode)
Secondly, make sure you customize the chat faces to your liking. Just open a
chat buffer, place your cursor on a piece of text whose face you want to
customize, and call customize-face.
In order to keep track of new messages in the mode-line, slack.el uses a package
called tracking, which is the same one circe uses for IRC chats. The command
tracking-next-buffer is a fantastic way to cycle through your pending
messages, bind it to something short.
(with-eval-after-load'tracking(define-keytracking-mode-map[f11]#'tracking-next-buffer));; Ensure the buffer exists when a message arrives on a;; channel that wasn't open.(setqslack-buffer-create-on-notifyt)
I’ll never know who thought user statuses were a good idea for Slack. But, thanks
to a tip by _asummers on HackerNews, I can live in a world where they don’t exist.
(defunslack-user-status(_id_team)"")
I like notifications with minimal titles, and the package is kind enough to make
these configurable.
;;; Channels(setqslack-message-notification-title-format-function(lambda(_teamroomthreadp)(concat(ifthreadp"Thread in #%s")room)))(defunendless/-cleanup-room-name(room-name)"Make group-chat names a bit more human-readable."(replace-regexp-in-string"--"" "(replace-regexp-in-string"#mpdm-"""room-name)));;; Private messages and group chats(setqslack-message-im-notification-title-format-function(lambda(_teamroomthreadp)(concat(ifthreadp"Thread in %s")(endless/-cleanup-room-nameroom))))
Slack.el uses lui for the chat buffers. If you, like me, are a heavy user of
abbrevs in Emacs, you’ll find it annoying that the final word of each message
won’t get expanded unless you explicitly hit SPC before RET. That’s easy to
remedy with an advice.
Finally, the biggest missing feature from this package is that it displays the
author on every message output.
Never mind, this feature has now been implemented by the package author!
You don’t have to stop here, of course. Want to fine-tune which buffers get
tracked on the mode-line? Hack into tracking.el. Want to change the face used
for your own messages, or even align them to the right? Redefine
slack-buffer-insert. Your workflow is yours to build.
Update 17 mar 2019
Noted that the message-merging is now part of the package.
When creating documents, context aware completion is a powerful
mechanism that can help you improve the speed, correctness and
discoverability.
Emacs provides context aware completion via the complete-symbol
command, bound to C-M-i by default. In order for it to do
something useful, completion-at-point-functions has to be set up.
Documentation:
Special hook to find the completion table for the thing at point.
Each function on this hook is called in turn without any argument and should
return either nil to mean that it is not applicable at point,
or a list of the form (START END COLLECTION) where
START and END delimit the entity to complete and should include
point, COLLECTION is the completion table to use to complete it.
For each major-mode, a different value of
completion-at-point-functions can (and probably should) apply. One
of the modes that's set up nicely by default is emacs-lisp-mode:
press C-M-i to get completion for Elisp variable and
function names. Org-mode, on the other hand, is quite lacking in this
regard: nothing useful happens with C-M-i.
When I write about code in Org-mode, I quote items like this:
=/home/oleh/=, =HammerFactoryFactory=, etc.
Quoting has several advantages:
It looks nice, since it's in a different face,
flyspell doesn't need to check it, which makes sense since it would
fail on most variable and class names,
Prevents Org from confusing directory names for italics mark up.
Completion has one more advantage on top of that: if I refer to a
symbol name multiple times within a document, completion helps me to
enter it quickly and correctly. Here's the corresponding completion
source:
First of all, it checks if the point is e.g. after =A, i.e. we
are in fact entering a new quoted symbol. If that's not the case,
return nil and let the other completion sources have a go.
Next, it looks through the current buffer for each =foo= and
=bar=, accumulating them into a list.
Finally, it returns the bounds of what we've got so far, plus the
found candidates. It's important that the bounds are passed to the
completion engine, so that it can delete everything inside the
bounds before inserting the whole selected symbol.
I usually enter ~, so that ffap-string-at-point recognizes it as a
path. Then complete each part of the path with C-M-i. It's
very similar to counsel-find-file. In fact, I could just use
counsel-find-file for this, with M-o i to insert the file
name instead of opening the selected file.
org-completion-refs
org-completion-refs is very similar to org-completion-symbols: it
will collect all instances of e.g. \label{foo}, and offer them for
completion when you enter \ref{. If you want to look at the code,
it's available in my config.
Outro
I hope I convinced you about the usefulness of completion at
point. It's especially cool since it's a universal interface for
major-mode-specific completion. So any IDE-like package for any
language could provide its own completion using the familiar
interface. That could go a long way towards providing a "just works"
experience, particularly when dealing with a new language.
Keeping a REPL (or a console) always by your side is never a bad habit, and if
you use an IDE-package (like Robe for Ruby, or Cider for Clojure) it’s nigh
unavoidable. Being an essential part of your environment, it would be ridiculous
not to invest some time optimizing it.
One obvious optimization is to bind a key to your “start console” command, but
that’s just the start. You pretty much never need two running consoles for the
same project, so why not have the same key switch to it if it’s already running?
But we can go a bit farther with very little work. I have a file where I define
a lot of small helper methods for my Ruby console, so let’s require it
automatically whenever a new console is started.
(defcustomendless/ruby-extensions-file"../console_extensions.rb""File loaded when a ruby console is started.
Name is relative to the project root.");; Skip ENV prompt that shows up in some cases.(setqinf-ruby-console-environment"development")(defunendless/run-ruby()(interactive)(require'inf-ruby)(let((default-directory(projectile-project-root))(was-running(get-buffer-processinf-ruby-buffer)));; This function automatically decides between starting;; a new console or visiting an existing one.(inf-ruby-console-auto)(when(and(notwas-running)(get-buffer-process(current-buffer))(file-readable-pendless/ruby-extensions-file));; If this brand new buffer has lots of lines then;; some exception probably happened.(send-string(get-buffer-process(current-buffer))(concat"require '"endless/ruby-extensions-file"'\n")))));; CIDER users might recognize this key.(define-keyruby-mode-map(kbd"C-c M-j")#'endless/run-ruby)
If you use Projectile and want to go even faster, check out the j key
on my post about Projectile.
There’s nothing I can praise about Magit that hasn’t been written in a dozen
blogs already, but since Jonas started a kickstarter campaign for it I knew I
had to say something. If you use Magit, you already know the greatness of it.
And if you don’t, hopefully I can convince you to try it in time to back the
campaign.
I could go on and on about the virtues of this gem. It’s probably the package
that most saves me time, and has taught me more about git than I learned reading
a whole book on it. But that’s all just sparks and glitter on top of the real
show. For Magit is a magic show, and its real feature is making you the
magician.
Controlling Magit feels like putting on a performance. Move your fingers quickly
enough, and you’ll be rebasing faster than the eye can see. But if your crowd is
not impressed yet, you move on to tagging, cherry-picking, rewriting, reflogging
until they’re left staring unblinkingly at your monitor, their jaws unknowingly
open in awe.
But, if you’re one of those that only cares about practical benefits, then here
are few Magit commands I use just about everyday.
f a Fetch all remotes
Usually how I start my day. Updates information
about all remote branches, without messing with your local branch. Really
helps that it’s a one-handed combo, so I can do it while sipping the
morning coffee.
r u Rebase upstream
Upstream usually means origin/master. If fetch-all
indicates that my local branch is out of date with upstream, this will
quickly bring it up to date. If I don’t want to rebase this branch, I can
m m and merge instead. Since fetch-all has already done all of the slow
networking, both merge and rebase are super quick.
b s Branch spin-off
Creates and checks out a new branch, while carrying
over any ongoing (and possibly committed) changes. It even undoes the
commits on the previous branch. Super useful if you realise you forgot to
branch-off from master before starting to work.
b c Branch create
Over time I’ve been gradually using less b s and
more b c. When I need a new working branch, instead of having to switch
to master, pull origin, and then spin-off, I simply b c straight from
origin/master (it never really matters if my local master is outdated).
P … Push!
Feels silly that pushing is so far down my list, but here you
go. P p is 95% of my usage, but P t is also useful and
you’ll occasionally need a -f in there.
c … Commit powers
c c is your basic commit. But I couldn’t live without
c w (for quickly fixing a commit message) or c e (for throwing more
changes into the last commit).
l l Log current
Magit is pretty good at keeping you informed of the
commit difference between local and remote, but l l is a quick way to get
your bearings if you ever feel lost.
y Branch viewer
Concise list of all refs. Great for finding and removing
old branches with k (both local and remote). Magit even warns if you try
to delete a branch that isn’t merged.
And a few commands I don’t really use everyday, but rather eagerly look forward
to the next chance to use them.
l r Reflog
Don’t know what reflog is? Neither did I! Magit taught me
about reflog without saying a word, and you should learn it
too (it feels like a superpower).
r s Rebase subset
At work, when I need to hotfix something I branch-off
from origin/production instead of origin/master. Occasionally though,
I’ll forget to do that, commit my hotfix, only to realise my branch is now
100 commits ahead of production. Rebase subset solves this is a blink, by
taking only the commit I want and placing it on top of another branch.
r i Rebase interactively
I admit, I was a little scared when I first tried
this feature. I had no idea what it did, but that “interactively” menu
option had been teasing me for months. If I explain it here I feel like I’d
be robing you of the discovery, so I’ll just describe it as “omnipotence
over history”.
All of this arsenal combines into some fast and powerful git manipulation. Just
yesterday a colleague asked for help on one of his branches. The whole checkout,
edit, commit, push process took less than 30 seconds. One more b b and I was
back in my branch doing my own stuff.
What’s more, it all feels right. My editor is where I manipulate the source
code, that’s where I ought to be changing branches, not alt-tabbing to a terminal.
There’s no denying slack is a useful tool for intrateam communication, but it’s
also a powerful source of distractions. Though I can’t just turn it off all
day, I can certainly keep the spam in check.
Slack’s Webapp does allow you to partially mute certain channels, but that’s
about as far as it goes. On the other hand, with the power of Emacs and the
Alert package, we can perfectly filter out anything we don’t care about.
Start by installing the Slack package from Melpa. It takes a bit of effort to
get it set up, but once you’re done following the instructions you should have a
(surprisingly featureful) Slack client running inside Emacs.
I’ll probably write further posts on how you can perfect your workflow with this
package, but for now we can just go over some basic keybinds.
;;; Big QOL changes.(setqslack-completing-read-function#'ido-completing-read)(setqslack-buffer-function#'switch-to-buffer)(setqslack-prefer-current-teamt)(setqslack-display-team-namenil);;; Go to any channel with `C-x j'.(define-keyctl-x-map"j"#'slack-select-rooms);;; Quick 'n dirty way of opening the most recent link;;; in the current chat room.(define-keyslack-mode-map(kbd"M-o")(kbd"<backtab> RET M->"));;; I thumbs-up a lot. Don't judge me.(define-keyslack-mode-map(kbd"C-;")":+1:");;; Bring up the mentions menu with `@', and insert a;;; space afterwards.(define-keyslack-mode-map"@"(defunendless/slack-message-embed-mention()(interactive)(call-interactively#'slack-message-embed-mention)(insert" ")));;; Pretty straightforward.(define-keyslack-mode-map(kbd"C-c C-d")#'slack-message-delete)(define-keyslack-mode-map(kbd"C-c C-e")#'slack-message-edit)(define-keyslack-mode-map(kbd"C-c C-k")#'slack-channel-leave)
Now you might think I’ve got it all backwards. Connecting Emacs with Slack could
only bring the distractions closer to me. But that’s where Alert comes in.
The Slack package automatically uses Alert for sending notifications, so you
have full control over them by customizing alert-user-configuration.
That’s super easy to do via the customization interface (M-xcustomize-variable). But the examples below use plain Elisp. Just keep in mind
that the first element is an alist determining which messages to match, and the
second element is a symbol specifying what to do (the third is not important
here).
For instance, let’s start by telling alert not to notify anything. Sounds
blissful, doesn’t it?
Both of the rules above are more-or-less supported by Slack already (although
you can’t really mute channels completely without leaving them). Below is one
example where Alert really shines.
We use Rollbar for exception tracking, and I like being notified whenever
something explodes in the server. However, it makes no sense to notify me
whenever someone resolves an issue. That can be resolved with two short rules.
Gap buffers are a common data structure for representing a text buffer
in a text editor. Emacs famously uses gap buffers — long-standing proof
that gap buffers are a perfectly sufficient way to represent a text
buffer.
Gap buffers are very easy to implement. A bare minimum
implementation is about 60 lines of C.
Gap buffers are especially efficient for the majority of typical
editing commands, which tend to be clustered in a small area.
Except for the gap, the content of the buffer is contiguous, making
the search and display implementations simpler and more efficient.
There’s also the potential for most of the gap buffer to be
memory-mapped to the original file, though typical encoding and
decoding operations prevent this from being realized.
Due to having contiguous content, saving a gap buffer is basically
just two write(2) system calls. (Plus fsync(2), etc.)
A gap buffer is really a pair of buffers where one buffer holds all of
the content before the cursor (or point for Emacs), and the other
buffer holds the content after the cursor. When the cursor is moved
through the buffer, characters are copied from one buffer to the
other. Inserts and deletes close to the gap are very efficient.
Typically it’s implemented as a single large buffer, with the
pre-cursor content at the beginning, the post-cursor content at the
end, and the gap spanning the middle. Here’s an illustration:
The top of the animation is the display of the text content and cursor
as the user would see it. The bottom is the gap buffer state, where
each character is represented as a gray block, and a literal gap for
the cursor.
Ignoring for a moment more complicated concerns such as undo and
Unicode, a gap buffer could be represented by something as simple as
the following:
structgapbuf{char*buf;size_ttotal;/* total size of buf */size_tfront;/* size of content before cursor */size_tgap;/* size of the gap */};
This is close to how Emacs represents it. In the structure
above, the size of the content after the cursor isn’t tracked directly,
but can be computed on the fly from the other three quantities. That is
to say, this data structure is normalized.
As an optimization, the cursor could be tracked separately from the
gap such that non-destructive cursor movement is essentially free. The
difference between cursor and gap would only need to be reconciled for
a destructive change — an insert or delete.
Multiple cursors is fairly recent text editor invention that
has gained a lot of popularity recent years. It seems every major
editor either has the feature built in or a readily-available
extension. I myself used Magnar Sveen’s well-polished package
for several years. Though obviously the concept didn’t originate in
Emacs or else it would have been called multiple points, which
doesn’t quite roll off the tongue quite the same way.
The concept is simple: If the same operation needs to done in many
different places in a buffer, you place a cursor at each position, then
drive them all in parallel using the same commands. It’s super flashy
and great for impressing all your friends.
However, as a result of improving my typing skills, I’ve
come to the conclusion that multiple cursors is all hat and no
cattle. It doesn’t compose well with other editing commands, it
doesn’t scale up to large operations, and it’s got all sorts of flaky
edge cases (off-screen cursors). Nearly anything you can do with
multiple cursors, you can do better with old, well-established editing
paradigms.
Somewhere around 99% of my multiple cursors usage was adding a common
prefix to a contiguous serious of lines. As similar brute force
options, Emacs already has rectangular editing, and Vim already has
visual block mode.
The most sophisticated, flexible, and robust alternative is a good old
macro. You can play it back anywhere it’s needed. You can zip it across
a huge buffer. The only downside is that it’s less flashy and so you’ll
get invited to a slightly smaller number of parties.
But if you don’t buy my arguments about multiple cursors being
tasteless, there’s still a good technical argument: Gap buffers are
not designed to work well in the face of multiple cursors!
For example, suppose we have a series of function calls and we’d like to
add the same set of arguments to each. It’s a classic situation for a
macro or for multiple cursors. Here’s the original code:
foo();
bar();
baz();
The example is tiny so that it will fit in the animations to come.
Here’s the desired code:
foo(x, y);
bar(x, y);
baz(x, y);
With multiple cursors you would place a cursor inside each set of
parenthesis, then type x, y. Visually it looks something like this:
Text is magically inserted in parallel in multiple places at a time.
However, if this is a text editor that uses a gap buffer, the
situation underneath isn’t quite so magical. The entire edit doesn’t
happen at once. First the x is inserted in each location, then the
comma, and so on. The edits are not clustered so nicely.
From the gap buffer’s point of view, here’s what it looks like:
For every individual character insertion the buffer has to visit each
cursor in turn, performing lots of copying back and forth. The more
cursors there are, the worse it gets. For an edit of length n with
m cursors, that’s O(n * m) calls to memmove(3). Multiple cursors
scales badly.
Compare that to the old school hacker who can’t be bothered with
something as tacky and modern (eww!) as multiple cursors, instead
choosing to record a macro, then play it back:
The entire edit is done locally before moving on to the next location.
It’s perfectly in tune with the gap buffer’s expectations, only needing
O(m) calls to memmove(3). Most of the work flows neatly into the
gap.
So, don’t waste your time with multiple cursors, especially if you’re
using a gap buffer text editor. Instead get more comfortable with your
editor’s macro feature. If your editor doesn’t have a good macro
feature, get a new editor.
If you want to make your own gap buffer animations, here’s the source
code. It includes a tiny gap buffer implementation:
Vim and Emacs have different internals models for the current working
directory, and these models influence the overall workflow for each
editor. They decide how files are opened, how shell commands are
executed, and how the build system is operated. These effects even reach
outside the editor to influence the overall structure of the project
being edited.
In the traditional unix model, which was eventually adopted
everywhere else, each process has a particular working directory
tracked by the operating system. When a process makes a request to the
operating system using a relative path — a path that doesn’t begin
with a slash — the operating system uses the process’ working
directory to convert the path into an absolute path. When a process
forks, its child starts in the same directory. A process can change
its working directory at any time using chdir(2), though
most programs never need to do it. The most obvious way this system
call is exposed to regular users is through the shell’s built-in cd
command.
Vim’s spiritual heritage is obviously rooted in vi, one of the classic
unix text editors, and the most elaborate text editor standardized by
POSIX. Like vi, Vim closely follows the unix model for working
directories. At any given time Vim has exactly one working directory.
Shell commands that are run within Vim will start in Vim’s working
directory. Like a shell, the cd ex command changes and queries Vim’s
working directory.
Emacs eschews this model and instead each buffer has its own working
directory tracked using a buffer-local variable, default-directory.
Emacs internally simulates working directories for its buffers like an
operating system, resolving absolute paths itself, giving credence to
the idea that Emacs is an operating system (“lacking only a decent
editor”). Perhaps this model comes from ye olde lisp machines?
In contrast, Emacs’ M-x cd command manipulates the local variable
and has no effect on the Emacs process’ working directory. In fact,
Emacs completely hides its operating system working directory from
Emacs Lisp. This can cause some trouble if that hidden working
directory happens to be sitting on filesystem you’d like to unmount.
Vim can be configured to simulate Emacs’ model with its autochdir
option. When set, Vim will literally chdir(2) each time the user
changes buffers, switches windows, etc. To the user, this feels just
like Emacs’ model, but this is just a convenience, and the core
working directory model is still the same.
Single instance editors
For most of my Emacs career, I’ve stuck to running a single,
long-lived Emacs instance no matter how many different tasks I’m
touching simultaneously. I start the Emacs daemon shortly after
logging in, and it continues running until I log out — typically only
when the machine is shut down. It’s common to have multiple Emacs
windows (frames) for different tasks, but they’re all bound to the
same daemon process.
While with care it’s possible to have a complex, rich Emacs
configuration that doesn’t significantly impact Emacs’ startup time, the
general consensus is that Emacs is slow to start. But since it has a
really solid daemon, this doesn’t matter: hardcore Emacs users only ever
start Emacs occasionally. The rest of the time they’re launching
emacsclient and connecting to the daemon. Outside of system
administration, it’s the most natural way to use Emacs.
The case isn’t so clear for Vim. Vim is so fast that many users fire
it up on demand and exit when they’ve finished the immediate task. At
the other end of the spectrum, others advocate using a single
instance of Vim like running a single Emacs daemon. In my
initial dive into Vim, I tried the single-instance, Emacs way of
doing things. I set autochdir out of necessity and pretended each
buffer had its own working directory.
At least for me, this isn’t the right way to use Vim, and it all comes
down to working directories. I want Vim to be anchored at the
project root with one Vim instance per project. Everything is
smoother when it happens in the context of the project’s root
directory, from opening files, to running shell commands (ctags in
particular), to invoking the build system. With autochdir, these
actions are difficult to do correctly, particularly the last two.
Invoking the build
I suspect the Emacs’ model of per-buffer working directories has, in a
Sapir-Whorf sort of way, been responsible for leading developers
towards poorly-designed, recursive Makefiles. Without a global
concept of working directory, it’s inconvenient to invoke the build
system (M-x compile) in some particular grandparent directory that
is the root of the project. If each directory has its own Makefile, it
usually makes sense to invoke make in the same directory as the file
being edited.
Over the years I’ve been reinventing the same solution to this
problem, and it wasn’t until I spent time with Vim and its alternate
working directory model that I truly understood the problem. Emacs
itself has long had a solution lurking deep in its bowels, unseen by
daylight: dominating files. The function I’m talking about is
locate-dominating-file:
(locate-dominating-file FILE NAME)
Look up the directory hierarchy from FILE for a directory containing
NAME. Stop at the first parent directory containing a file NAME, and
return the directory. Return nil if not found. Instead of a string,
NAME can also be a predicate taking one argument (a directory) and
returning a non-nil value if that directory is the one for which we’re
looking.
The trouble of invoking the build system at the project root is that
Emacs doesn’t really have a concept of a project root. It doesn’t know
where it is or how to find it. The vi model inherited by Vim is to
leave the working directory at the project root. While Vim can
simulate Emacs’ working directory model, Emacs cannot (currently)
simulate Vim’s model.
Instead, by identifying a file name unique to the project’s root (i.e.
a “dominating” file) such as Makefile or build.xml, then
locate-dominating-file can discover the project root. All that’s
left is wrapping M-x compile so that default-directory is
temporarily adjusted to the project’s root.
That looks very roughly like this (and needs more work):
It’s a pattern I’ve used again and again and
again, working against the same old friction. By running one
Vim instance per project at the project’s root, I get the correct
behavior for free.
Lately, due to working with a large code base, I've grown more and
more fond of counsel-rg. It's an Elisp wrapper
around ripgrep - a relatively
new recursive grep tool that aims to be faster than the competition
(ag, git grep, pt, ack etc).
Besides being really fast, rg also has some really
nice
command switches.
One such switch is especially useful for Emacs:
-M, --max-columns NUM
: Don't print lines longer than this limit in bytes. Longer lines are omitted,
and only the number of matches in that line is printed.
The -M switch is useful twofold:
Emacs is slow when dealing with long lines (by long I mean thousands of chars per line)
Emacs is slow at accepting a huge amount of output from a process
For each character you add to your input, counsel-rg starts a new
shell command to recalculate the matches with the new input. This
means that in order to avoid keyboard lag there's only about 0.1
seconds available for both:
Running the shell command.
Accepting output from the shell command.
So I'm quite happy that rg speeds up both steps. Less time spent on
these steps provides for much smoother searching.
counsel-grep-or-swiper
I also work with large log files, one file at a time. For a long time,
I've used counsel-grep-or-swiper as my main search command:
(global-set-key(kbd"C-s")'counsel-grep-or-swiper)
But for a 40Mb log file with really long lines
counsel-grep-or-swiper started to lag a bit. I tried counsel-rg,
and it was actually faster than grep, although it was searching the
whole directory. So I thought, why not use rg instead of grep? The
switch is actually really easy and required only a simple user
customization:
(setqcounsel-grep-base-command"rg -i -M 120 --no-heading --line-number --color never '%s' %s")
Outro
If you haven't tried ripgrep so far, I suggest you give it a go. Happy hacking!
And if you're a C hacker and have some free time on your hands, why
not look at the long lines and the process output issues in Emacs? I'd
be very grateful:)
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
Ray Puzio will be presenting A Tour of Arxana
The topography of knowledge can be quite intricate, with assertions
linking not only to objects of knowledge, but also to other assertions
and even linking to links. In order to adequately represent and
reason about such situations, Joe Corneli and Ray Puzio have been
developing a hypertext system called Arxana.
Inspired by Ted Nelson’s projects Xanadu and implemented in Emacs
Lisp, this system is based upon representing hypergraphs with a
generalization of CONS cells. In this talk, Ray will give a tour of
Arxana both under the hood and behind the wheel and round it off with
a scenic drive through some applications to the study of mathematical
exposition which he and Joe are working on.
Ivy is a completion method that's
similar to Ido, but with emphasis on simplicity and customizability.
Overview
The current release constitutes of 339 commits and almost a full
year of progress since 0.8.0. Many issues ranging
from #493
to #946 were fixed.
The number of people who contributed code as grown
to 63;
thanks, everyone!
Details on changes
Changelog.org has been a part of the repository since 0.6.0, you
can get the details of the current and past changes:
Many improvements are incremental and don't require any extra code to
enable. I'll go over a few selected features that require a bit of
information to make a good use of them.
A better action choice interface
For all ivy completions, pressing M-o allows to execute
one of the custom actions for the current command. Now you have an
option to use hydra for selecting
an action. Use this code to turn on the feature:
(require'ivy-hydra)
One big advantage of the new interface is that you can peak at the
action list with M-o without dismissing the candidate
list. Press M-o again to go back to candidate selection
without selecting an action.
Here's some code from my config that ensures that I always have some extra actions to choose from:
The new counsel-rg joins the group of grepping commands in counsel
(counsel-ag, counsel-git-grep, counsel-grep, counsel-pt). It
wraps around the newly popular and very
fast ripgrep shell tool.
A nice improvement to the grepping commands is the ability to specify
extra flags when you press C-u (universal-argument)
before the
command. See this gif
for an example of excluding *.el from the files searched by ag.
counsel-find-file
Press M-o b to change the current directory to one of the
virtual buffers' directories. You continue to select a file from that
directory.
Press M-o r to find the current file as root.
counsel-git-log
You can now customize
counsel-git-log-cmd. See
#652 for using this to
make counsel-git-log work on Windows.
counsel-mode
counsel-info-lookup-symbol now substitutes the built in info-lookup-symbol.
Pressing C-r while in the minibuffer of eval-expression
or shell-command now gives you completion of your previous
history.
counsel-yank-pop
Use the new counsel-yank-pop-separator variable to make
counsel-yank-pop look
like this.
ivy
There was breaking change for alist type collections some months
ago. Right now the action functions receive an item from the
collection, instead of (cdr item) like before. If anything breaks,
the easy fix is to add an extra cdr to the action function.
Unique index for alist completion was added. The uniqueness assumption
is that the completion system is passed a list of unique strings, of
which one (or more) are selected. Unlike plain string completion,
alists may require violating the uniqueness assumption: there may be
two elements with the same car but different cdr. Example: C
function declaration and definition for tag completion. Until now,
whenever two equal strings were sent to ivy-read, only the first one
could be selected. Now, each alist car gets an integer index assigned
to it as a text property 'idx. So it's possible to differentiate two
alist items with the same key.
Action functions don't require using with-ivy-window anymore. This
allows for a lot of simplification, e.g. use insert instead of
(lambda (x) (with-ivy-window (insert x))).
ivy-switch-buffer
You can now customize faces in ivy-switch-buffer by the mode of each buffer.
Here's a snippet from my config:
Given the title, the publication date of this article is probably
really confusing. This was deliberate.
Three weeks ago I made a conscious decision to improve my typing
habits. You see, I had a dirty habit. Despite spending literally
decades typing on a daily basis, I’ve been a weak typist. It wasn’t
exactly finger pecking, nor did it require looking down at the
keyboard as I typed, but rather a six-finger dance I developed
organically over the years. My technique was optimized towards Emacs’
frequent use of CTRL and ALT combinations, avoiding most of the hand
scrunching. It was fast enough to keep up with my thinking most of the
time, but was ultimately limiting due to its poor accuracy. I was
hitting the wrong keys far too often.
My prime motivation was to learn Vim — or, more specifically, to learn
modal editing. Lots of people swear by it, including people whose
opinions I hold in high regard. The modal editing community is without
a doubt larger than the Emacs community, especially since, thanks to
Viper and Evil, a subset of the Emacs community is also part
of the modal editing community. There’s obviously something
significantly valuable about it, and I wanted to understand what that
was.
But I was a lousy typist who couldn’t hit the right keys often enough to
make effective use of modal editing. I would need to learn touch typing
first.
Touch typing
How would I learn? Well, the first search result for “online touch
typing course” was Typing Club, so that’s what I went with. By
the way, here’s my official review: “Good enough not to bother
checking out the competition.” For a website it’s pretty much the
ultimate compliment, but it’s not exactly the sort of thing you’d want
to hear from your long-term partner.
My hard rule was that I would immediately abandon my old habits cold
turkey. Poor typing is a bad habit just like smoking, minus the cancer
and weakened sense of smell. It was vital that I unlearn all that old
muscle memory. That included not just my six-finger dance, but also my
NetHack muscle memory. NetHack uses “hjkl” for navigation just
like Vim. The problem was that I’d spent a couple hundred hours in
NetHack over the past decade with my index finger on “h”, not the
proper home row location. It was disorienting to navigate around Vim
initally, like riding a bicycle with inverted controls.
Based on reading other people’s accounts, I determined I’d need
several days of introductory practice where I’d be utterly
unproductive. I took a three-day weekend, starting my touch typing
lessons on a Thursday evening. Boy, they weren’t kidding about it
being slow going. It was a rough weekend. When checking in on my
practice, my wife literally said she pitied me. Ouch.
By Monday I was at a level resembling a very slow touch typist. For
the rest of the first week I followed all the lessons up through the
number keys, never progressing past an exercise until I had exceeded
the target speed with at least 90% accuracy. This was now enough to
get me back on my feet for programming at a glacial, frustrating pace.
Programming involves a lot more numbers and symbols than other kinds
of typing, making that top row so important. For a programmer, it
would probably be better for these lessons to be earlier in the
series.
Modal editing
For that first week I mostly used Emacs while I was finding my feet
(or finding my fingers?). That’s when I experienced first hand what
all these non-Emacs people — people who I, until recently, considered
to be unenlightened simpletons — had been complaining about all these
years: Pressing CTRL and ALT key combinations from the home row is a
real pain in in the ass! These complaints were suddenly making
sense. I was already seeing the value of modal editing before I even
started really learning Vim. It made me look forward to it even more.
During the second week of touch typing I went though Derek Wyatt’s
Vim videos and learned my way around the :help system enough
to bootstrap my Vim education. I then read through the user manual,
practicing along the way. I’ll definitely have to pass through it a
few more times to pick up all sorts of things that didn’t stick. This
is one way that Emacs and Vim are a lot alike.
Update: Practical Vim: Edit Text at the Speed of Thought was
recommended in the comments, and it’s certainly a better place to
start than the Vim user manual. Unlike the manual, it’s opinionated
and focuses on good habits, which is exactly what a newbie needs.
One of my rules when learning Vim was to resist the urge to remap
keys. I’ve done it a lot with Emacs: “Hmm, that’s not very convenient.
I’ll change it.” It means my Emacs configuration is fairly
non-standard, and using Emacs without my configuration is like using
an unfamiliar editor. This is both good and bad. The good is that I’ve
truly changed Emacs to be my editor, suited just for me. The bad is
that I’m extremely dependent on my configuration. What if there was a
text editing emergency?
With Vim as a sort of secondary editor, I want to be able to fire it
up unconfigured and continue to be nearly as productive. A pile of
remappings would prohibit this. In my mind this is like a form of
emergency preparedness. Other people stock up food and supplies. I’m
preparing myself to sit at a strange machine without any of my
configuration so that I can start the rewrite of the software lost in
the disaster, so long as that machine has vi, cc, and
make. If I can’t code in C, then what’s the point in surviving
anyway?
The other reason is that I’m just learning. A different mapping might
seem more appropriate, but what do I know at this point? It’s better
to follow the beaten path at first, lest I form a bunch of bad habits
again. Trust in the knowledge of the ancients.
Future directions
I am absolutely sticking with modal editing for the long term. I’m
really enjoying it so far. At three weeks of touch typing and two
weeks of modal editing, I’m around 80% caught back up with my old
productivity speed, but this time I’ve got a lot more potential for
improvement.
For now, Vim will continue taking over more and more of my text
editing work. My last three articles were written in Vim. It’s really
important to keep building proficiency. I still rely on Emacs for
email and for syndication feeds, and that’s not
changing any time soon. I also really like Magit as a Git
interface. Plus I don’t want to abandon years of accumulated
knowledge and leave the users of my various Emacs packages out
to dry. Ultimately I believe will end up using Evil, to get what seems
to be the best of both worlds: modal editing and Emacs’ rich
extensibility.
Recently, I've had to code some C++ at work. And I saw it as a good
opportunity to step up my Emacs' IDE game. I've eschewed clang-based
tools until now, but GCC isn't adding AST support any time soon, and
CEDET is too slow and too clumsy with macros for the particular
project that I
had. Here's
the line in
Eigen that
broke the camel's back. Basically it's 30 lines of macros that expand
to 30 lines of typedefs. Maybe it's a valid implementation choice, I'd
rather avoid the macros altogether, but in any case I couldn't get
CEDET to parse that.
Use Rtags for navigation
The first thing I tried
was rtags. My project was
CMake-based, so I just put this line in my subdirectory Makefile:
cmake:
cd ../build && cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 ..
The -DCMAKE_EXPORT_COMPILE_COMMANDS=1 causes a
compile_commands.json file to be emitted during the actual
compilation. This file describes the compile flags for every source
file. These flags are essential in helping the parser understand
what's going on.
Then, in the build directory I start:
rdm & rc -J .
Finally, rtags-find-symbol-at-point should work now. I still like to
use CEDET as backup, it's pretty good at tracking variables
defined in the current function:
For my other C++ projects which aren't CMake-based, I use the
excellent bear tool to emit the
compile_commands.json file. It's as easy as:
make clean
bear make
Use Irony for completion
It didn't take long to figure out that rtags isn't great at
completion. I almost accepted that's just the way it is. But this
morning I decided to make some changes and
try irony-mode. And it
worked beautifully for completion! What's ironic, is that irony-mode
doesn't have goto-symbol, so the time spent to figure out rtags
was worth it.
Here's my Irony setup; I only changed the C-M-i binding to
the newly written counsel-irony, now available in the counsel
package on MELPA:
First of all, the completion is displayed inline, similarly to modern IDEs.
You can use all of Ivy's regex tricks to complete your candidate:
Note how the power of regex matching allows me to narrow the initial
1622 candidates to only 22 functions that have src1 and src2 as
arguments. One of the candidates is cut off for being longer than the
window width. You can still match against the invisible text, but you
won't see it. It's possible to use C-c C-o (ivy-occur) to
store the current candidates into a buffer:
Clicking the mouse on any of the lines in the new buffer will insert
the appropriate symbol into the C++ buffer.
Outro
I'd like to thank the authors of rtags and irony-mode for these
nice packages. Hopefully, counsel-irony is a nice addition. Happy
hacking!
ediff.el --- a comprehensive visual interface to diff & patch
I wrote about
ediff years ago.
Today, I'll just reference a useful ediff snippet
from my config that I've added
some time ago and refined only recently.
The premise is quite simple: press e in dired-mode to
immediately ediff two marked files, no questions asked:
(define-keydired-mode-map"e"'ora-ediff-files)
And here's the code, with a few bells and whistles:
;; -*- lexical-binding: t -*-(defunora-ediff-files()(interactive)(let((files(dired-get-marked-files))(wnd(current-window-configuration)))(if(<=(lengthfiles)2)(let((file1(carfiles))(file2(if(cdrfiles)(cadrfiles)(read-file-name"file: "(dired-dwim-target-directory)))))(if(file-newer-than-file-pfile1file2)(ediff-filesfile2file1)(ediff-filesfile1file2))(add-hook'ediff-after-quit-hook-internal(lambda()(setqediff-after-quit-hook-internalnil)(set-window-configurationwnd))))(error"no more than 2 files should be marked"))))
Some notes on how the extra code adds convenience:
In case no files are marked, the file at point is used as the first
file, and read-file-name is used for the second file. Since I
have the magic (setq dired-dwim-target t) in my config, in case a
second dired buffer is open, dired-dwim-target-directory will
offer it as the starting directory during completion. Very useful
to compare two files in two different directories.
Depending on the order of the arguments to ediff-files, the
changes will appear either as added or removed;
file-newer-than-file-p tries to put the arguments in a logical
order by looking at the files' last change times.
ediff-after-quit-hook-internal is used to restore the previous
window configuration after I quit ediff with q.
That's about it. Hopefully, it's useful. Happy hacking.
make-it-so is an old package
of mine that I haven't yet highlighted on the blog. This package helps
you manage a collection of makefiles that are used to generate new
files from existing files using shell commands.
You can think of these makefiles as a directory of shell functions,
arranged by the extension of the files that they operate on:
When you call make-it-so on a particular file, you get completion
for the recipes that are available for that file extension, along with
an option to create a new recipe.
Example 1: convert pdf to txt
Suppose you want to convert a PDF file test.pdf to a text file test.txt.
In case the recipe is in your collection, you don't have to remember
the command or the command switches to do it anymore:
Navigate to test.pdf in dired and press , (bound to make-it-so).
Select the recipe you want using completion: to-txt is already provided.
Your file and the makefile recipe are moved to the staging area:
The makefile is opened in a new buffer with the following bindings:
f5 (mis-save-and-compile) will run compile,
creating test.txt in the current directory.
C-, (mis-finalize) will finalize the operation,
moving test.pdf and test.txt to the parent directory (where
test.pdf was before), and deleting the staging directory.
C-M-, (mis-abort) will move test.pdf back to its
initial location and delete all generated files. This command is
effectively an undo for make-it-so.
It takes a large chunk of text to describe everything, but the key sequence for doing all this is quite short:
, - make-it-so.
RET - select to-txt.
f5 - create test.txt.
C-, - finalize.
Example 2: make a gif from a series of png images
I'll describe the process of creating a high quality gif like this
one, which describes the effect of the C key
in lispy:
First, I use kazam to take two png
screenshots of my Emacs screen:
$ ls -1 *.png
Screenshot 2017-02-25 16:14:49.png
Screenshot 2017-02-25 16:15:10.png
I plan to use gifsicle to sequence
the still images into a gif. But it only takes gif as the input
format, so first I have to convert my png files to non-animated
gif files.
I open the dired buffer where they are located and mark them with
m (dired-mark). Then call make-it-so with ,
and select to-gif recipe. This recipe has no parameters, so there's
nothing else to do but f5 C-,. Two new files are created:
Note that the file names (the defaults of kazam) are problematic
when used with makefiles, since they contain spaces and colons. The
Elisp layer of make-it-so takes care of that. It renames the files
back and forth so that the logic in the makefiles remains simple.
Next, I mark the two gif files using *%
(dired-mark-files-regexp), press , once more and select
the gifsicle recipe. I'm presented a makefile with the following
contents:
# ——— parameters —————————————————————————————————# delay between frames in hundredths of a seconddelay=60# ——— implementation —————————————————————————————DIRGIF=$(shell ls *.gif | grep -v anime.gif)all:anime.gifanime.gif:Makefile$(DIRGIF)
rm -f anime.gif
gifsicle --delay=$(delay) --colors=256 --loop $(DIRGIF) > $@echo$@ >> provide
clean:
rm -f anime.gif provide
install-tools:
sudo apt-get install gifsicle
.PHONY:allinstall-toolsclean
The most commonly useful parameter, the delay between frames, is
nicely documented at the top. I don't have to remember that the switch
name is --delay or that the switch style --delay=60 is used. I
simply change the number above until I get the result that I want.
Example 3: add a new recipe
As a sample scenario, assume you want to convert *.svg to *.png.
Step 1
An internet search leads to Stack Overflow and this command:
inkscape -z -e test.png -w 1024 -h 1024 test.svg
Navigate to the file(s) in dired and call make-it-so with
,. No default actions are available, so just type "to-png"
and hit RET. The "to-" prefix signifies that this is a
conversion, adapting the Makefile to this form:
# This is a template for the Makefile.# Parameters should go in the upper half as:# width = 200# and be referenced in the command as $(width)# ____________________________________________DIRSVG=$(shell dir *.svg)DIRPNG=$(DIRSVG:.svg=.png)all:cleanMakefile$(DIRPNG)%.png: %.svgecho"add command here"echo$@ >> provide
clean:
rm -f *.png provide
# Insert the install command here.# e.g. sudo apt-get install ffmpeginstall-tools:echo"No tools required".PHONY:allinstall-toolsclean
If the action name doesn't have a "to-" prefix, the transformation is
assumed to be e.g. "svg" -> "out.svg". You can change this of course
by editing the Makefile.
Step 2
In case the command needs additional packages in order to work you
might want to change echo "No tools required" to the appropriate
package install instruction, e.g. sudo apt-get install inkscape.
When you're on a new system, this will serve as a reminder of what you
should install in order for the Makefile to work. Simply call:
make install-tools
Step 3
Replace echo "add command here" with:
inkscape -z -e $@ -w $(width) -h $(height) $^
The parameters width and height will go to the top of the
Makefile, where they can be customized.
$@ refers to the output file, test.png in this case.
$^ refers to the input file, test.svg in this case.
That's it. You can see the final
Makefile here.
Test if the command works with f5 from the Makefile. If
you're happy with it, call mis-finalize with C-, from
dired. The Makefile will be saved for all future calls to
make-it-so.
Outro
To summarize the advantages of make-it-so:
Write the recipe one time, never have to look up how to do the
same thing a few months from now.
A chance to write the recipe zero times, if someone in the
community has already done it and shared the recipe with you.
The Elisp layer takes care of hairy file names.
Parallel commands on multiple files, i.e. make -j8, are provided for free.
The most important usage tip: until you're sure that the command and
the Makefile work properly make backups. In fact, make backups period.
Happy hacking!
A few months ago I had a discussion with Vladimir Kazanov about his
Orgfuse project: a Python script that exposes an Emacs
Org-mode document as a FUSE filesystem. It permits other
programs to navigate the structure of an Org-mode document through the
standard filesystem APIs. I suggested that, with the new dynamic
modules in Emacs 25, Emacs itself could serve a FUSE filesystem. In
fact, support for FUSE services in general could be an package of his
own.
So that’s what he did: Elfuse. It’s an old joke that
Emacs is an operating system, and here it is handling system calls.
However, there’s a tricky problem to solve, an issue also present my
joystick module. Both modules handle asynchronous events —
filesystem requests or joystick events — but Emacs runs the event loop
and owns the main thread. The external events somehow need to feed
into the main event loop. It’s even more difficult with FUSE because
FUSE also wants control of its own thread for its own event loop.
This requires Elfuse to spawn a dedicated FUSE thread and negotiate a
request/response hand-off.
When a filesystem request or joystick event arrives, how does Emacs
know to handle it? The simple and obvious solution is to poll the
module from a timer.
Blocking directly on the module’s event pump with Emacs’ thread would
prevent Emacs from doing important things like, you know, being a
text editor. The timer allows it to handle its own events
uninterrupted. It gets the job done, but it’s far from perfect:
It imposes an arbitrary latency to handling requests. Up to the
poll period could pass before a request is handled.
Polling the module 100 times per second is inefficient. Unless you
really enjoy recharging your laptop, that’s no good.
The poll period is a sliding trade-off between latency and battery
life. If only there was some mechanism to, ahem, signal the Emacs
thread, informing it that a request is waiting…
SIGUSR1
Emacs Lisp programs can handle the POSIX SIGUSR1 and SIGUSR2 signals,
which is exactly the mechanism we need. The interface is a “key”
binding on special-event-map, the keymap that handles these kinds of
events. When the signal arrives, Emacs queues it up for the main event
loop.
The module blocks on its own thread on its own event pump. When a
request arrives, it queues the request, rings the bell for Emacs to
come handle it (raise()), and waits on a semaphore. For illustration
purposes, assume the module reads requests from and writes responses
to a file descriptor, like a socket.
intevent_fd=/* ... */;structrequestrequest;sem_init(&request.sem,0,0);for(;;){/* Blocking read for request event */read(event_fd,&request.event,sizeof(request.event));/* Put request on the queue */queue_lock(requests);queue_push(requests,&request);queue_unlock(requests);raise(SIGUSR1);// TODO: Should raise() go inside the lock?/* Wait for Emacs */while(sem_wait(&request.sem));/* Reply with Emacs' response */write(event_fd,&request.response,sizeof(request.response));}
The sem_wait() is in a loop because signals will wake it up
prematurely. In fact, it may even wake up due to its own signal on the
line before. This is the only way this particular use of sem_wait()
might fail, so there’s no need to check errno.
If there are multiple module threads making requests to the same
global queue, the lock is necessary to protect the queue. The
semaphore is only for blocking the thread until Emacs has finished
writing its particular response. Each thread has its own semaphore.
When Emacs is done writing the response, it releases the module thread
by incrementing the semaphore. It might look something like this:
This SIGUSR1+semaphore mechanism is roughly how Elfuse currently
processes requests.
Making it work on Windows
Windows doesn’t have signals. This isn’t a problem for Elfuse since
Windows doesn’t have FUSE either. Nor does it matter for Joymacs since
XInput isn’t event-driven and always requires polling. But someday
someone will need this mechanism for a dynamic module on Windows.
Fortunately there’s a solution: input language change events,
WM_INPUTLANGCHANGE. It’s also on special-event-map:
Instead of raise() (or pthread_kill()), broadcast the window event
with PostMessage(). Outside of invoking the language-change key
binding, Emacs will ignore the event because WPARAM is 0 — it doesn’t
belong to any particular window. We don’t really want to change the
input language, after all.
Naturally you’ll also need to replace the POSIX threading primitives
with the Windows versions (CreateThread(), CreateSemaphore(),
etc.). With a bit of abstraction in the right places, it should be
pretty easy to support both POSIX and Windows in these asynchronous
dynamic module events.
Not everything written in Emacs Lisp needs to be fast. Most of Emacs
itself — around 82% — is written in Emacs Lisp because those parts
are generally not performance-critical. Otherwise these functions
would be built-ins written in C. Extensions to Emacs don’t have a
choice and — outside of a few exceptions like dynamic modules
and inferior processes — must be written in Emacs Lisp, including
their performance-critical bits. Common performance hot spots are
automatic indentation, AST parsing, and interactive
completion.
Here are 5 guidelines, each very specific to Emacs Lisp, that will
result in faster code. The non-intrusive guidelines could be applied
at all times as a matter of style — choosing one equally expressive
and maintainable form over another just because it performs better.
There’s one caveat: These guidelines are focused on Emacs 25.1 and
“nearby” versions. Emacs is constantly evolving. Changes to the
virtual machine and byte-code compiler may transform
currently-slow expressions into fast code, obsoleting some of these
guidelines. In the future I’ll add notes to this article for anything
that changes.
(1) Use lexical scope
This guideline refers to the following being the first line of every
Emacs Lisp source file you write:
;;; -*- lexical-binding: t; -*-
This point is worth mentioning again and again. Not only will your
code be more correct, it will be measurably faster. Dynamic
scope is still opt-in through the explicit use of special variables,
so there’s absolutely no reason not to be using lexical scope. If
you’ve written clean, dynamic scope code, then switching to lexical
scope won’t have any effect on its behavior.
Along similar lines, special variables are a lot slower than local,
lexical variables. Only use them when necessary.
(2) Prefer built-in functions
Built-in functions are written in C and are, as expected,
significantly faster than the equivalent written in Emacs Lisp.
Complete as much work as possible inside built-in functions, even if
it might mean taking more conceptual steps overall.
For example, what’s the fastest way to accumulate a list of items?
That is, new items go on the tail but, for algorithm reasons, the list
must be constructed from the head.
You might be tempted to keep track of the tail of the list, appending
new elements directly to the tail with setcdr (via setf below).
It might not look it, but nreverse is very fast. Not only is it a
built-in, it’s got its own opcode. Using push in a loop, then
finishing with nreverse is the canonical and fastest way to
accumulate a list of items.
In fib-track-tail, the added complexity of tracking the tail in
Emacs Lisp is much slower than zipping over the entire list a second
time in C.
Listen, I know you love dash.el and higher order functions,
but this habit ain’t cheap. The byte-code compiler does not know how
to inline these lambdas, so there’s an additional per-element function
call overhead.
Worse, if you’re using lexical scope like I told you, the above
example forms a closure over e. This means a new function object
is created (e.g. make-byte-code) each time expt-list is called. To
be clear, I don’t mean that the lambda is recompiled each time — the
same byte-code string is shared between all instances of the same
lambda. A unique function vector (#[...]) and constants vector are
allocated and initialized each time expt-list is invoked.
Related mini-guideline: Don’t create any more garbage than strictly
necessary in performance-critical code.
Compare to an implementation with an explicit loop, using the
nreverse list-accumulation technique.
The cl-loop macro will expand into essentially the previous
definition, making them practically equivalent. It takes some getting
used to, but writing efficient loops is a whole lot less tedious with
cl-loop.
In Emacs 24.4 and earlier, catch/throw is implemented by
converting the body of the catch into a lambda function and calling
it. If code inside the catch accesses a variable outside the catch
(very likely), then, in lexical scope, it turns into a closure,
resulting in the garbage function object like before.
In Emacs 24.5 and later, the byte-code compiler uses a new opcode,
pushcatch. It’s a whole lot more efficient, and there’s no longer a
reason to shy away from catch/throw in performance-critical code.
This is important because it’s often the only way to perform an early
bailout.
(4) Prefer using functions with dedicated opcodes
When following the guideline about using built-in functions, you might
have several to pick from. Some built-in functions have dedicated
virtual machine opcodes, making them much faster to invoke. Prefer
these functions when possible.
How can you tell when a function has an assigned opcode? Take a peek
at the byte-defop listings in bytecomp.el. Optimization often
involves getting into the weeds, so don’t be shy.
For example, the assq and assoc functions search for a matching
key in an association list (alist). Both are built-in functions, and
the only difference is that the former compares keys with eq (e.g.
symbol or integer keys) and the latter with equal (typically string
keys). The difference in performance between eq and equal isn’t as
important as another factor: assq has its own opcode (158).
This means in performance-critical code you should prefer assq,
perhaps even going as far as restructuring your alists specifically to
have eq keys. That last step is probably a trade-off, which means
you’ll want to make some benchmarks to help with that decision.
Another example is eq, =, eql, and equal. Some macros and
functions use eql, especially cl-lib which inherits eql as a
default from Common Lisp. Take cl-case, which is like switch from
the C family of languages. It compares elements with eql.
It turns out eql is pretty much always the worst choice for
cl-case. Of the four equality functions I listed, the only one
lacking an opcode is eql. A faster definition would use eq. (In
theory, cl-casecould have done this itself because it knows all
the keys are symbols.)
Fortunately eq can safely compare integers in Emacs Lisp. You only
need eql when comparing symbols, integers, and floats all at once,
which is unusual.
(5) Unroll loops using and/or
Consider the following function which checks its argument against a
list of numbers, bailing out on the first match. I used % instead of
mod since the former has an opcode (166) and the latter does not.
The byte-code compiler doesn’t know how to unroll loops. Fortunately
that’s something we can do for ourselves using and and or. The
compiler will turn this into clean, efficient jumps in the byte-code.
In Emacs 24.4 and earlier with the old-fashioned lambda-based catch,
the unrolled definition is seven times faster. With the faster
pushcatch-based catch it’s about twice as fast. This means the
loop overhead accounts for about half the work of the first definition
of this function.
Update: It was pointed out in the comments that this particular
example is equivalent to a cond. That’s literally true all the way
down to the byte-code, and it would be a clearer way to express the
unrolled code. In real code it’s often not quite equivalent.
Unlike some of the other guidelines, this is certainly something you’d
only want to do in code you know for sure is performance-critical.
Maintaining unrolled code is tedious and error-prone.
I’ve had the most success with this approach by not by unrolling these
loops myself, but by using a macro, or similar, to
generate the unrolled form.
How can I find more optimization opportunities myself?
Use M-x disassemble to inspect the byte-code for your own hot spots.
Observe how the byte-code changes in response to changes in your
functions. Take note of the sorts of forms that allow the byte-code
compiler to produce the best code, and then exploit it where you can.
Using my Emacs configuration. C/C++ are the focus of this clip, but you'll be set up for much more. Most of the clip is sort of boring (waiting for stuff to download from MELPA and etc) but everything comes together at around 0:57 seconds in.
Last night I pushed another performance enhancement for Elfeed, this
time reducing the time spent parsing feeds. It’s accomplished by
compiling, during macro expansion, a jQuery-like domain-specific
language within Elfeed.
Heuristic parsing
Given the nature of the domain — an under-specified standard
and a lack of robust adherence — feed parsing is much more heuristic
than strict. Sure, everyone’s feed XML is strictly conforming since
virtually no feed reader tolerates invalid XML (thank you, XML
libraries), but, for the schema, the situation resembles the de
facto looseness of HTML. Sometimes important or required information
is missing, or is only available in a different namespace.
Sometimes, especially in the case of timestamps, it’s in the wrong
format, or encoded incorrectly, or ambiguous. It’s real world data.
To get a particular piece of information, Elfeed looks in a number of
different places within the feed, starting with the preferred source
and stopping when the information is found. For example, to find the
date of an Atom entry, Elfeed first searches for elements in this
order:
<published>
<updated>
<date>
<modified>
<issued>
Failing to find any of these elements, or if no parsable date is
found, it settles on the current time. Only the updated element is
required, but published usually has the desired information, so it
goes first. The last three are only valid for another namespace, but
are useful fallbacks.
Before Elfeed even starts this search, the XML text is parsed into an
s-expression using xml-parse-region — a pure Elisp XML parser
included in Emacs. The search is made over the resulting s-expression.
<?xml version="1.0" encoding="utf-8"?><feedxmlns="http://www.w3.org/2005/Atom"><title>Example Feed</title><linkhref="http://example.org/"/><updated>2003-12-13T18:30:02Z</updated><author><name>John Doe</name></author><id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id><entry><title>Atom-Powered Robots Run Amok</title><linkrel="alternate"href="http://example.org/2003/12/13/atom03"/><id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id><updated>2003-12-13T18:30:02Z</updated><summary>Some text.</summary></entry></feed>
Which is parsed to into this s-expression.
((feed((xmlns."http://www.w3.org/2005/Atom"))(title()"Example Feed")(link((href."http://example.org/")))(updated()"2003-12-13T18:30:02Z")(author()(name()"John Doe"))(id()"urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6")(entry()(title()"Atom-Powered Robots Run Amok")(link((rel."alternate")(href."http://example.org/2003/12/13/atom03")))(id()"urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a")(updated()"2003-12-13T18:30:02Z")(summary()"Some text."))))
Each XML element is converted to a list. The first item is a symbol
that is the element’s name. The second item is an alist of attributes
— cons pairs of symbols and strings. And the rest are its children,
both string nodes and other elements. I’ve trimmed the extraneous
string nodes from the sample s-expression.
A subtle detail is that xml-parse-region doesn’t just return the
root element. It returns a list of elements, which always happens to
be a single element list, which is the root element. I don’t know why
this is, but I’ve built everything to assume this structure as input.
Elfeed strips all namespaces stripped from both elements and
attributes to make parsing simpler. As I said, it’s heuristic rather
than strict, so namespaces are treated as noise.
A domain-specific language
Coding up Elfeed’s s-expression searches in straight Emacs Lisp would
be tedious, error-prone, and difficult to understand. It’s a lot of
loops, assoc, etc. So instead I invented a jQuery-like, CSS
selector-like, domain-specific language (DSL) to express these
searches concisely and clearly.
For example, all of the entry links are “selected” using this
expression:
(feedentrylink[rel"alternate"]:href)
Reading right-to-left, this matches every href attribute under every
link element with the rel="alternate" attribute, under every
entry element, under the feed root element. Symbols match element
names, two-element vectors match elements with a particular attribute
pair, and keywords (which must come last) narrow the selection to a
specific attribute value.
Imagine hand-writing the code to navigate all these conditions for
each piece of information that Elfeed requires. The RSS parser makes
up to 16 such queries, and the Atom parser makes as many as 24. That
would add up to a lot of tedious code.
The package (included with Elfeed) that executes this query is called
“xml-query.” It comes in two flavors: xml-query and xml-query-all.
The former returns just the first match, and the latter returns all
matches. The naming parallels the querySelector() and
querySelectorAll() DOM methods in JavaScript.
That date search I mentioned before looks roughly like this. The *
matches text nodes within the selected element. It must come last just
like the keyword matcher.
Over the past three years, Elfeed has gained more and more of these
selectors as it collects more and more information from feeds. Most
recently, Elfeed collects author and category information provided by
feeds. Each new query slows feed parsing a little bit, and it’s a
perfect example of a program slowing down as it gains more features
and capabilities.
But I don’t want Elfeed to slow down. I want it to get faster!
Optimizing the domain-specific language
Just like the primary jQuery function ($), both xml-query and
xml-query-all are functions. The xml-query engine processes the
selector from scratch on each invocation. It examines the first
element, dispatches on its type/value to apply it to the input, and
then recurses on the rest of selector with the narrowed input,
stopping when it hits the end of the list. That’s the way it’s worked
from the start.
However, every selector argument in Elfeed is a static, quoted list.
Unlike user-supplied filters, I know exactly what I want to
execute ahead of time. It would be much better if the engine didn’t
have to waste time reparsing the DSL for each query.
This is the classic split between interpreters and compilers. An
interpreter reads input and immediately executes it, doing what the
input tells it to do. A compiler reads input and, rather than execute
it, produces output, usually in a simpler language, that, when
evaluated, has the same effect as executing the input.
Rather than interpret the selector, it would be better to compile it
into Elisp code, compile that into byte-code, and then have the
Emacs byte-code virtual machine (VM) execute the query each time it’s
needed. The extra work of parsing the DSL is performed ahead of time,
the dispatch is entirely static, and the selector ultimately executes
on a much faster engine (byte-code VM). This should be a lot faster!
So I wrote a function that accepts a selector expression and emits
Elisp source that implements that selector: a compiler for my DSL.
Having a readily-available syntax tree is one of the big advantages
of homoiconicity, and this sort of function makes perfect sense
in a lisp. For the external interface, this compiler function is
called by a new pair of macros, xml-query* and xml-query-all*.
These macros consume a static selector and expand into the compiled
Elisp form of the selector.
To demonstrate, remember that link query from before? Here’s the macro
version of that selection, but only returning the first match. Notice
the selector is no longer quoted. This is because it’s consumed by the
macro, not evaluated.
As soon as it finds a match, it’s thrown to the top level and
returned. Without the DSL, the expansion is essentially what would
have to be written by hand. This is exactly the sort of leverage
you should be getting from a compiler. It compiles to around 130
byte-code instructions.
The xml-query-all* form is nearly the same, but instead of a
throw, it pushes the result into the return list. Only the prologue
(the outermost part) and the epilogue (the innermost part) are
different.
Parsing feeds is a hot spot for Elfeed, so I wanted the compiler’s
output to be as efficient as possible. I had three goals for this:
No extraneous code. It’s easy for the compiler to emit
unnecessary code. The byte-code compiler might be able to eliminate
some of it, but I don’t want to rely on that. Except for the
identifiers, it should basically look like a human wrote it.
Avoid function calls. I don’t want to pay function call
overhead, and, with some care, it’s easy to avoid. In the
xml-query* expansion, the only function call is throw, which is
unavoidable. The xml-query-all* version makes no function calls
whatsoever. Notice that I used assq rather than assoc. First, it
only needs to match symbols, so it should be faster. Second, assq
has its own byte-code instruction (158) and assoc does not.
No unnecessary memory allocations. The xml-query* expansion
makes no allocations. The xml-query-all* version only conses
once per output, which is the minimum possible.
The end result is at least as optimal as hand-written code, but
without the chance of human error (typos, fat fingering) and sourced
from an easy-to-read DSL.
Performance
In my tests, the xml-query macros are a full order of magnitude
faster than the functions. Yes, ten times faster! It’s an even
bigger gain than I expected.
In the full picture, xml-query is only one part of parsing a feed.
Measuring the time starting from raw XML text (as delivered by
cURL) to a list of database entry objects, I’m seeing an
overall 25% speedup with the macros. The remaining time is
dominated by xml-parse-region, which is mostly out of my control.
With xml-query so computationally cheap, I don’t need to worry about
using it more often. Compared to parsing XML text, it’s virtually
free.
When it came time to validate my DSL compiler, I was really happy
that Elfeed had a test suite. I essentially rewrote a core component
from scratch, and passing all of the unit tests was a strong sign that
it was correct. Many times that test suite has provided confidence in
changes made both by me and by others.
I’ll end by describing another possible application: Apply this
technique to regular expressions, such that static strings containing
regular expressions are compiled into Elisp/byte-code via macro
expansion. I wonder if situationally this would be faster than Emacs’
own regular expression engine.
I recently had a discussion with Xah Lee about lexical scope in
Emacs Lisp. The topic was why lexical-binding exists at a file-level
when there was already lexical-let (from cl-lib), prompted by my
previous article on JIT byte-code compilation. The specific
context is Emacs Lisp, but these concepts apply to language design in
general.
Until Emacs 24.1 (June 2012), Elisp only had dynamically scoped
variables — a feature, mostly by accident, common to old lisp
dialects. While dynamic scope has some selective uses, it’s widely
regarded as a mistake for local variables, and virtually no other
languages have adopted it.
Way back in 1993, Dave Gillespie’s deviously clever lexical-let
macro was committed to the cl package, providing a rudimentary
form of opt-in lexical scope. The macro walks its body replacing local
variable names with guaranteed-unique gensym names: the exact same
technique used in macros to create “hygienic” bindings that aren’t
visible to the macro body. It essentially “fakes” lexical scope within
Elisp’s dynamic scope by preventing variable name collisions.
For example, here’s one of the consequences of dynamic scope.
The “local” variable v in outer is visible to its callee, inner,
which can access and manipulate it. The meaning of the free variablev in inner depends entirely on the run-time call stack. It might
be a global variable, or it might be a local variable for a caller,
direct or indirect.
Using lexical-let deconflicts these names, giving the effect of
lexical scope.
But there’s more to lexical scope than this. Closures only make sense
in the context of lexical scope, and the most useful feature of
lexical-let is that lambda expressions evaluate to closures. The
macro implements this using a technique called closure
conversion. Additional parameters are added to the original
lambda function, one for each lexical variable (and not just each
closed-over variable), and the whole thing is wrapped in another
lambda function that invokes the original lambda function with the
additional parameters filled with the closed-over variables — yes, the
variables (e.g. symbols) themselves, not just their values, (e.g.
pass-by-reference). The last point means different closures can
properly close over the same variables, and they can bind new values.
To roughly illustrate how this works, the first lambda expression
below, which closes over the lexical variables x and y, would be
converted into the latter by lexical-let. The #: is Elisp’s syntax
for uninterned variables. So #:x is a symbol x, but not the
symbol x (see print-gensym).
;; Before conversion:(lambda()(+xy));; After conversion:(lambda(&restargs)(apply(lambda(xy)(+(symbol-valuex)(symbol-valuey)))'#:x'#:yargs))
I’ve said on multiple occasions that lexical-binding: t has
significant advantages, both in performance and static analysis, and
so it should be used for all future Elisp code. The only reason it’s
not the default is because it breaks some old (badly written) code.
However, lexical-let doesn’t realize any of these advantages! In
fact, it has worse performance than straightforward dynamic scope with
let.
New symbol objects are allocated and initialized (make-symbol) on
each run-time evaluation, one per lexical variable.
Since it’s just faking it, lexical-let still uses dynamic
bindings, which are more expensive than lexical bindings. It varies
depending on the C compiler that built Emacs, but dynamic variable
accesses (opcode varref) take around 30% longer than lexical
variable accesses (opcode stack-ref). Assignment is far worse,
where dynamic variable assignment (varset) takes 650% longer than
lexical variable assignment (stack-set). How I measured all this
is a topic for another article.
The “lexical” variables are accessed using symbol-value, a full
function call, so they’re even slower than normal dynamic
variables.
Because converted lambda expressions are constructed dynamically at
run-time within the body of lexical-let, the resulting closure is
only partially byte-compiled even if the code as a whole has been
byte-compiled. In contrast, lexical-binding: t closures are fully
compiled. How this works is worth its own article.
Converted lambda expressions include the additional internal
function invocation, making them slower.
While lexical-let is clever, and occasionally useful prior to Emacs
24, it may come at a hefty performance cost if evaluated frequently.
There’s no reason to use it anymore.
Constraints on code generation
Another reason to be weary of dynamic scope is that it puts needless
constraints on the compiler, preventing a number of important
optimization opportunities. For example, consider the following
function, bar:
(defunbar()(let((x1)(y2))(foo)(+xy)))
Byte-compile this function under dynamic scope (lexical-binding:
nil) and disassemble it to see what it looks like.
(byte-compile#'bar)(disassemble#'bar)
That pops up a buffer with the disassembly listing:
0 constant 1
1 constant 2
2 varbind y
3 varbind x
4 constant foo
5 call 0
6 discard
7 varref x
8 varref y
9 plus
10 unbind 2
11 return
It’s 12 instructions, 5 of which deal with dynamic bindings. The
byte-compiler doesn’t always produce optimal byte-code, but this just
so happens to be nearly optimal byte-code. The discard (a very
fast instruction) isn’t necessary, but otherwise no more compiler
smarts can improve on this. Since the variables x and y are
visible to foo, they must be bound before the call and loaded after
the call. While generally this function will return 3, the
compiler cannot assume so since it ultimately depends on the behavior
foo. Its hands are tied.
Compare this to the lexical scope version (lexical-binding: t):
It’s only 8 instructions, none of which are expensive dynamic variable
instructions. And this isn’t even close to the optimal byte-code. In
fact, as of Emacs 25.1 the byte-compiler often doesn’t produce the
optimal byte-code for lexical scope code and still needs some work.
Despite not firing on all cylinders, lexical scope still manages to
beat dynamic scope in performance benchmarks.
Here’s the optimal byte-code, should the byte-compiler become smarter
someday:
0 constant foo
1 call 0
2 constant 3
3 return
It’s down to 4 instructions due to computing the math operation at
compile time. Emacs’ byte-compiler only has rudimentary constant
folding, so it doesn’t notice that x and y are constants and
misses this optimization. I speculate this is due to its roots
compiling under dynamic scope. Since x and y are no longer exposed
to foo, the compiler has the opportunity to optimize them out of
existence. I haven’t measured it, but I would expect this to be
significantly faster than the dynamic scope version of this function.
Optional dynamic scope
You might be thinking, “What if I really do want x and y to be
dynamically bound for foo?” This is often useful. Many of Emacs’ own
functions are designed to have certain variables dynamically bound
around them. For example, the print family of functions use the global
variable standard-output to determine where to send output by
default.
Have no fear: With lexical-binding: t you can have your cake and
eat it too. Variables declared with defvar, defconst, or
defvaralias are marked as “special” with an internal bit flag
(declared_special in C). When the compiler detects one of these
variables (special-variable-p), it uses a classical dynamic binding.
Declaring both x and y as special restores the original semantics,
reverting bar back to its old byte-code definition (next time it’s
compiled, that is). But it would be poor form to mark x or y as
special: You’d de-optimize all code (compiled after the declaration)
anywhere in Emacs that uses these names. As a package author, only do
this with the namespace-prefixed variables that belong to you.
The only way to unmark a special variable is with the undocumented
function internal-make-var-non-special. I expected makunbound to
do this, but as of Emacs 25.1 it does not. This could possibly be
considered a bug.
Accidental closures
I’ve said there are absolutely no advantages to lexical-binding: nil.
It’s only the default for the sake of backwards-compatibility. However,
there is one case where lexical-binding: t introduces a subtle issue
that would otherwise not exist. Take this code for example (and
nevermind prin1-to-string for a moment):
This creates and serializes a closure, which is one of Elisp’s unique
features. It doesn’t close over any variables, so it should be
pretty simple. However, this function will only work correctly under
lexical-binding: t when byte-compiled.
The interpreter doesn’t analyze the closure, so just closes over
everything. This includes the hidden variable temp-buffer created by
the with-temp-buffer macro, resulting in an abstraction leak.
Buffers aren’t readable, so this will signal an error if an attempt is
made to read this function back into an s-expression. The
byte-compiler fixes this by noticing temp-buffer isn’t actually
closed over and so doesn’t include it in the closure, making it work
correctly.
Under lexical-binding: nil it works correctly either way:
(function-as-string);; -> "(lambda nil :example)"
This may seem contrived — it’s certainly unlikely — but it has come
up in practice. Still, it’s no reason to avoid lexical-binding: t.
Use lexical scope in all new code
As I’ve said again and again, always use lexical-binding: t. Use
dynamic variables judiciously. And lexical-let is no replacement. It
has virtually none of the benefits, performs worse, and it only
applies to let, not any of the other places bindings are created:
function parameters, dotimes, dolist, and condition-case.
Today I pushed an update for Elfeed that doubles the speed
of the search filter in the worse case. This is the user-entered
expression that dynamically narrows the entry listing to a subset that
meets certain criteria: published after a particular date,
with/without particular tags, and matching/non-matching zero or more
regular expressions. The filter is live, applied to the database as
the expression is edited, so it’s important for usability that this
search completes under a threshold that the user might notice.
The typical workaround for these kinds of interfaces is to make
filtering/searching asynchronous. It’s possible to do this well, but
it’s usually a terrible, broken design. If the user acts upon the
asynchronous results — say, by typing the query and hitting enter to
choose the current or expected top result — then the final behavior is
non-deterministic, a race between the user’s typing speed and the
asynchronous search. Elfeed will keep its synchronous live search.
For anyone not familiar with Elfeed, here’s a filter that finds all
entries from within the past year tagged “youtube” (+youtube) that
mention Linux or Linus (linu[sx]), but aren’t tagged “bsd” (-bsd),
limited to the most recent 15 entries (#15):
@1-year-old +youtube linu[xs] -bsd #15
The database is primarily indexed over publication date, so filters on
publication dates are the most efficient filters. Entries are visited
in order starting with the most recently published, and the search can
bail out early once it crosses the filter threshold. Time-oriented
filters have been encouraged as the solution to keep the live search
feeling lively.
Filtering Overview
The first step in filtering is parsing the filter text entered by the
user. This string is broken into its components using the
elfeed-search-parse-filter function. Date filter components are
converted into a unix epoch interval, tags are interned into symbols,
regular expressions are gathered up as strings, and the entry limit is
parsed into a plain integer. Absence of a filter component is
indicated by nil.
Previously, the next step was to apply the elfeed-search-filter
function with this structured filter representation to the database.
Except for special early-bailout situations, it works left-to-right
across the filter, checking each condition against each entry. This is
analogous to an interpreter, with the filter being a program.
Thinking about it that way, what if the filter was instead compiled
into an Emacs byte-code function and executed directly by the Emacs
virtual machine? That’s what this latest update does.
Benchmarks
With six different filter components, the actual filtering routine is
a bit too complicated for an article, so I’ll set up a simpler, but
roughly equivalent, scenario. With a reasonable cut-off date, the
filter was already sufficiently fast, so for benchmarking I’ll focus
on the worst case: no early bailout opportunities. An entry will be
just a list of tags (symbols), and the filter will have to test every
entry.
My real-world Elfeed database currently has 46,772 entries with
36 distinct tags. For my benchmark I’ll round this up to a nice
100,000 entries, and use 26 distinct tags (A–Z), which has the nice
alphabet property and more closely reflects the number of tags I still
care about.
First, here’s make-random-entry to generate a random list of 1–5
tags (i.e. an entry). The state parameter is the random state,
allowing for deterministic benchmarks on a randomly-generated
database.
Here’s my old time macro. An important change I’ve made since
years ago is to call garbage-collect before starting the clock,
eliminating bad samples from unlucky garbage collection events.
Depending on what you want to measure, it may even be worth disabling
garbage collection during the measurement by setting
gc-cons-threshold to a high value.
Finally, the benchmark harness. It uses a hard-coded seed to generate
the same pseudo-random database. The test is run against the a filter
function, f, 100 times in search for the same 6 tags, and the timing
results are averaged.
The baseline will be memq (test for membership using identity,
eq). There are two lists of tags to compare: the list that is the
entry, and the list from the filter. This requires a nested loop for
each entry, one explicit (cl-loop) and one implicit (memq), both
with early bailout.
Byte-code compiling everything and running the benchmark on my laptop
I get:
(benchmark#'memq-count);; => 0.041 seconds
That’s actually not too bad. One of the advantages of this definition
is that there are no function calls. The memq built-in function has
its own opcode (62), and the rest of the definition is special forms
and macros expanding to special forms (cl-loop). It’s exactly the
thing I need to exploit to make filters faster.
As a sanity check, what would happen if I used member instead of
memq? In theory it should be slower because it uses equal for
tests instead of eq.
It’s only slightly slower because member, like many other
built-ins, also has an opcode (157). It’s just a tiny bit
more overhead.
(benchmark#'member-count);; => 0.047 seconds
To test function call overhead while still using the built-in (e.g.
written in C) memq, I’ll alias it so that the byte-code compiler is
forced to emit a function call.
So the function call adds about 27% overhead. This means it would be a
good idea to avoid calling functions in the filter if I can help
it. I should rely on these special opcodes.
Suppose memq was written in Emacs Lisp rather than C. How much would
that hurt performance? My version of my-memq below isn’t quite the
same since it returns t rather than the sublist, but it’s good enough
for this purpose. (I’m using cl-loop because writing early bailout
in plain Elisp without recursion is, in my opinion, ugly.)
Oof! It’s more than 3 times slower than the opcode. This means I
should use built-ins as much as possible in the filter.
Dynamic vs. lexical scope
There’s one last thing to watch out for. Everything so far has been
compiled with lexical scope. You should really turn this on by default
for all new code that you write. It has three important advantages:
It allows the compiler to catch more mistakes.
It eliminates a class of bugs related to dynamic scope: Local
variables are exposed to manipulation by callees.
It halves the performance in this benchmark, and for no benefit. Under
dynamic scope, local variables use the varref opcode — a global
variable lookup — instead of the stack-ref opcode — a simple array
index.
(defunnorm(ab)(*(-ab)(-ab)))
Under dynamic scope, this compiles to:
0 varref a
1 varref b
2 diff
3 varref a
4 varref b
5 diff
6 mult
7 return
And under lexical scope (notice the variable names disappear):
So far I’ve been moving in the wrong direction, making things slower
rather than faster. How can I make it faster than the straight memq
version? By compiling the filter into byte-code.
I won’t write the byte-code directly, but instead generate Elisp code
and use the byte-code compiler on it. This is safer, will work
correctly in future versions of Emacs, and leverages the optimizations
performed by the byte-compiler. This sort of thing recently got a bad
rap on Emacs Horrors, but I was happy to see that this
technique is already established.
It dynamically builds the code as an s-expression, runs that through
the byte-code compiler, executes it, and throws it away. It’s
“just-in-time,” though compiling to byte-code and not native
code. For the benchmark tags of (A B C D E F), this builds
the following:
Due to its short-circuiting behavior, or is a special form, so this
function is just special forms and memq in its opcode form. It’s as
fast as Elisp can get.
Having s-expressions is a real strength for lisp, since the
alternative (in, say, JavaScript) would be to assemble the function by
concatenating code strings. By contrast, this looks a lot like a
regular lisp macro. Invoking the byte-code compiler does add some
overhead compared to the interpreted filter, but it’s insignificant.
How much faster is this?
(benchmark#'jit-count);; => 0.017s
It’s more than twice as fast! The big gain here is through loop
unrolling. The outer loop has been unrolled into the or expression.
That section of byte-code looks like this:
In Elfeed, not only does it unroll these loops, it completely
eliminates the overhead for unused filter components. Comparing to
this benchmark, I’m seeing roughly matching gains in Elfeed’s worst
case. In Elfeed, I also bind lexical-binding around the
byte-compile call to force lexical scope, since otherwise it just
uses the buffer-local value (usually nil).
Filter compilation can be toggled on and off by setting
elfeed-search-compile-filter. If you’re up to date, try out live
filters with it both enabled and disabled. See if you can notice the
difference.
Result summary
Here are the results in a table, all run with Emacs 24.4 on x86-64.
And the same benchmarks on Aarch64 (Emacs 24.5, ARM Cortex-A53), where
I also occasionally use Elfeed, and where I have been very interested
in improving performance.
I have created a fortune file for the all-time top 10,000
/r/Showerthoughts posts, as of October 2016. As a word of
warning: Many of these entries are adult humor and may not be
appropriate for your work computer. These fortunes would be
categorized as “offensive” (fortune -o).
The copyright status of this file is subject to each of its thousands
of authors. Since it’s not possible to contact many of these authors —
some may not even still live — it’s obviously never going to be under
an open source license (Creative Commons, etc.). Even more, some
quotes are probably from comedians and such, rather than by the
redditor who made the post. I distribute it only for fun.
Installation
To install this into your fortune database, first process it with
strfile to create a random-access index, showerthoughts.dat, then
copy them to the directory with the rest.
$ strfile showerthoughts
"showerthoughts.dat" created
There were 10000 strings
Longest string: 343 bytes
Shortest string: 39 bytes
$ cp showerthoughts* /usr/share/games/fortunes/
Alternatively, fortune can be told to use this file directly:
$ fortune showerthoughts
Not once in my life have I stepped into somebody's house and
thought, "I sure hope I get an apology for 'the mess'."
―AndItsDeepToo, Aug 2016
If you didn’t already know, fortune is an old unix utility that
displays a random quotation from a quotation database — a digital
fortune cookie. I use it as an interactive login shell greeting on
my ODROID-C2 server:
if shopt-q login_shell;then
fortune ~/.fortunes
fi
How was it made?
Fortunately I didn’t have to do something crazy like scrape reddit for
weeks on end. Instead, I downloaded the pushshift.io submission
archives, which is currently around 70 GB compressed. Each file
contains one month’s worth of JSON data, one object per submission,
one submission per line, all compressed with bzip2.
Unlike so many other datasets, especially when it’s made up of
arbitrary inputs from millions of people, the format of the
/r/Showerthoughts posts is surprisingly very clean and requires
virtually no touching up. It’s some really fantastic data.
A nice feature of bzip2 is concatenating compressed files also
concatenates the uncompressed files. Additionally, it’s easy to
parallelize bzip2 compression and decompression, which gives it an
edge over xz. I strongly recommend using lbzip2 to
decompress this data, should you want to process it yourself.
cat RS_*.bz2 | lbunzip2 > everything.json
jq is my favorite command line tool for processing JSON (and
rendering fractals). To filter all the /r/Showerthoughts posts,
it’s a simple select expression. Just mind the capitalization of the
subreddit’s name. The -c tells jq to keep it one per line.
However, you’ll quickly find that jq is the bottleneck, parsing all
that JSON. Your cores won’t be exploited by lbzip2 as they should. So
I throw grep in front to dramatically decrease the workload for
jq.
This will let some extra things through, but it’s a superset. The -a
option is necessary because the data contains some null bytes. Without
it, grep switches into binary mode and breaks everything. This is
incredibly frustrating when you’ve already waited half an hour for
results.
To further reduce the workload further down the pipeline, I take
advantage of the fact that only four fields will be needed: title,
score, author, and created_utc. The rest can — and should, for
efficiency’s sake — be thrown away where it’s cheap to do so.
This gathers all 1,199,499 submissions into a 185 MB JSON file (as of
this writing). Most of these submissions are terrible, so the next
step is narrowing it to the small set of good submissions and putting
them into the fortune database format.
It turns out reddit already has a method for finding the best
submissions: a voting system. Just pick the highest scoring posts.
Through experimentation I arrived at 10,000 as the magic cut-off
number. After this the quality really starts to drop off. Over time
this should probably be scaled up with the total number of
submissions.
I did both steps at the same time using a bit of Emacs Lisp, which is
particularly well-suited to the task:
This Elisp program reads one JSON object at a time and sticks each
into a AVL tree sorted by score (descending), then timestamp
(ascending), then title (ascending). The AVL tree is limited to 10,000
items, with the lowest items being dropped. This was a lot faster than
the more obvious approach: collecting everything into a big list,
sorting it, and keeping the top 10,000 items.
Formatting
The most complicated part is actually paragraph wrapping the
submissions. Most are too long for a single line, and letting the
terminal hard wrap them is visually unpleasing. The submissions are
encoded in UTF-8, some with characters beyond simple ASCII. Proper
wrapping requires not just Unicode awareness, but also some degree of
Unicode rendering. The algorithm needs to recognize grapheme
clusters and know the size of the rendered text. This is not so
trivial! Most paragraph wrapping tools and libraries get this wrong,
some counting width by bytes, others counting width by codepoints.
Emacs’ M-x fill-paragraph knows how to do all these things — only
for a monospace font, which is all I needed — and I decided to
leverage it when generating the fortune file. Here’s an example that
paragraph-wraps a string:
For the file format, items are delimited by a % on a line by itself.
I put the wrapped content, followed by a quotation dash, the
author, and the date. A surprising number of these submissions have
date-sensitive content (“on this day X years ago”), so I found it was
important to include a date.
April Fool's Day is the one day of the year when people critically
evaluate news articles before accepting them as true.
―kellenbrent, Apr 2015
%
Of all the bodily functions that could be contagious, thank god
it's the yawn.
―MKLV, Aug 2015
%
There’s the potential that a submission itself could end with a lone
% and, with a bit of bad luck, it happens to wrap that onto its own
line. Fortunately this hasn’t happened yet. But, now that I’ve
advertised it, someone could make such a submission, popular enough
for the top 10,000, with the intent to personally trip me up in a
future update. I accept this, though it’s unlikely, and it would be
fairly easy to work around if it happened.
The strfile program looks for the % delimiters and fills out a
table of file offsets. The header of the .dat file indicates the
number strings along with some other metadata. What follows is a table
of 32-bit file offsets.
struct{uint32_tstr_version;/* version number */uint32_tstr_numstr;/* # of strings in the file */uint32_tstr_longlen;/* length of longest string */uint32_tstr_shortlen;/* shortest string length */uint32_tstr_flags;/* bit field for flags */charstr_delim;/* delimiting character */}
Note that the table doesn’t necessarily need to list the strings in
the same order as they appear in the original file. In fact, recent
versions of strfile can sort the strings by sorting the table, all
without touching the original file. Though none of this important to
fortune.
Now that you know how it all works, you can build your own fortune
file from your own inputs!
Two months ago Emacs 25 was released and introduced a new dynamic
module feature. Emacs can now load shared libraries built
against Emacs’ module API, defined in emacs-module.h. What’s
interesting about this API is that it doesn’t require linking against
Emacs or any sort of library. Instead, at run time Emacs supplies the
module’s initialization function with function pointers for the entire
API.
As a demonstration, in this article I’ll build an Emacs joystick
interface (Linux only) using a dynamic module. It will allow Emacs to
read events from any joystick on the system. All the source code is
here:
It includes a calibration interface (M-x joydemo) within Emacs:
Currently, Emacs’ emacs-module.h header is the entirety of the module
documentation. It’s a bit thin and leaves ambiguities that requires
some reading of the Emacs source code. Even reading the source, it’s
not clear which behaviors are a reliable part of the interface. For
example, if there’s a pending non-local exit, it’s safe for a function
to return NULL since the return value is never inspected (Emacs
25.1), but will this always be the case? While mistakes are
unforgiving (a hard crash), the API is mostly intuitive and it’s been
pretty easy to feel my way around it.
All Emacs values — integers, floats, cons cells, vectors, strings,
etc. — are represented as the polymorphic, pointer-valued type,
emacs_value. Despite being a pointer, NULL is not a valid value,
as convenient as that would be. The API includes functions for
creating and extracting the fundamental types: integers, floats,
strings. Almost all other object types can only be accessed by making
Lisp function calls to regular Emacs functions from the module.
Modules also introduce a brand new Emacs object type: a user
pointer. These are non-readable, opaque pointer values
returned by modules, typically representing a handle to some resource,
be it a memory block, database connection, or a joystick. These
objects include a finalizer function pointer — which, surprisingly, is
not permitted to be NULL — and their lifetime is managed by Emacs’
garbage collector.
User pointers are a somewhat dangerous feature since there’s little to
stop Emacs Lisp code from misusing them. A Lisp program can take a
user pointer from one module and pass it to a function in a different
module. Since it’s just a pointer, there’s no way to type check it. At
best, a module could maintain a table of all its live pointers,
checking all user pointer arguments against the table before
dereferencing. But I don’t expect this to be normal practice.
Module Initialization
After loading the module through the platform’s mechanism, the first
thing Emacs does is check for the symbol plugin_is_GPL_compatible.
While tacky, this is not surprising given the culture around Emacs.
Next it calls emacs_module_init(), passing it the first function
pointer. From this, the module can get a Lisp environment and start
doing Emacs things, such as binding module functions to Lisp symbols.
In a real module, it’s common to create function objects for native
functions, then fetch the fset symbol and make a Lisp call on it to
bind the newly-created function object to a name. You’ll see this in
action later.
Joystick API
The joystick API will closely resemble Linux’s own joystick API,
making for a fairly thin wrapper. It’s so thin that Emacs almost
doesn’t even need a dynamic module. This is because, on Linux,
joysticks are just files under /dev/input/. Want to see the input
events on the first joystick? Just read /dev/input/js0. So Plan 9.
Emacs already knows how to read files, but these virtual files are a
little too special for that. The header linux/joystick.h defines a
struct js_event:
structjs_event{uint32_ttime;/* event timestamp in milliseconds */int16_tvalue;uint8_ttype;uint8_tnumber;/* axis/button number */};
The idea is to read from the joystick device into this structure. The
first several reads are initialization that define the axes and
buttons of the joystick and their initial state. Further events are
queued up for the file descriptor. This all means that the file can’t
just be opened each time joystick input is needed. It has to be held
open for the duration, and is typically configured non-blocking.
The Emacs package will be called joymacs and there will be three
functions:
The joymacs-open function will take an integer, opening the Nth
joystick (/dev/input/jsN). It will create a file descriptor for the
joystick device, returning it as a user pointer. Think of it as a sort
of “joystick handle.” Now, it could instead return the file
descriptor as an integer, but the user pointer has two significant
benefits:
The resource will be garbage collected. If the caller loses
track of a file descriptor returned as an integer, the joystick
device will be held open until Emacs shuts down, using up one of
Emacs’ file descriptors. By putting it in a user pointer, the
garbage collector will have the module to release the file
descriptor if the user loses track of it.
It should be difficult for the user to make a dangerous call.
Emacs Lisp can’t create user pointers — they only come from modules
— and so the module is less likely to get passed the wrong thing.
In the case of joystick-close, the module will be calling
close(2) on the argument. We definitely don’t want to make that
system call on file descriptors owned by Emacs. Further, since user
pointers are mutable, the module can ensure it doesn’t call
close(2) twice.
Here’s the implementation for joymacs-open. I’ll over over each part
in detail.
The C function name doesn’t matter to Emacs. It’s static because it
doesn’t even matter if the function visible to Emacs. It will get the
function pointer later as part of initialization.
This is the prototype for all functions callable by Emacs Lisp,
regardless of its arity. It has four arguments:
It gets an environment, env, through which to call back into
Emacs.
It gets n, the number of arguments. This is guaranteed to be the
correct number of arguments, as specified later when creating the
function object, so only variadic functions need to inspect this
argument.
The Lisp arguments are passed as an array of values, args.
There’s no type declaration when declaring a function object, so
these may be of the wrong type. I’ll go over how to deal with this.
Finally, it gets an arbitrary pointer, supplied at function object
creation time. This allows the module to create closures, but will
usually be ignored.
The first thing the function does is extract its integer argument.
This is actually an intmax_t, but I don’t think anyone has that many
USB ports. An int will suffice.
As for not underestimating fools, what if the user passed a value that
isn’t an integer? Will the world come crashing down? Fortunately Emacs
checks that in extract_integer and, if there’s a mismatch, sets a
pending error signal in the environment. This is really great because
checking types directly in the module is a real pain the ass. So,
before committing to anything further, such as opening a file, I check
for this signal and bail out early if necessary. In Emacs 25.1 it’s
safe to return NULL since the return value will be completely ignored,
but I’d rather hedge my bets.
By the way, the nil here is a global variable set in initialization.
You don’t just get that for free!
The next step is opening the joystick device, read-only and
non-blocking. The non-blocking is vital because the module would
otherwise hang Emacs later if there are no events (well, except for
the read being quickly interrupted by a POSIX signal).
If the joystick fails to open (e.g. it doesn’t exist, or the user
lacks permission), manually set an error signal for a non-local exit.
I chose the file-error signal and I’m just using the filename as the
signal data.
Otherwise create the user pointer. No need to allocate any memory;
just stuff it in the pointer itself. If the user mistakenly passes it
to another module, it will sure be in for a surprise when it tries to
dereference it.
The garbage collector will call this function when the user pointer is
lost. If the user closes it early with joymacs-close, that function
will set the user pointer to -1, an invalid file descriptor, so that
it doesn’t get closed a second time here.
The joymacs-read function is doing something a little unusual for an
Emacs Lisp function. It takes two arguments: the joystick handle and a
5-element vector. Instead of returning the event in some
representation, it fills the vector with the event details. The are
two reasons for this:
The API has no function for creating vectors … though the module
could get the make-symbol vector and call it to create a
vector.
The idiom for event pumps is for the caller to supply a buffer to
the pump. This has better performance by avoiding lots of
unnecessary allocations, especially since events tend to be
message-like objects with a short, well-defined extent.
Here’s the full definition:
staticemacs_valuejoymacs_read(emacs_env*env,ptrdiff_tn,emacs_value*args,void*ptr){(void)n;(void)ptr;intfd=(intptr_t)env->get_user_ptr(env,args[0]);if(env->non_local_exit_check(env)!=emacs_funcall_exit_return)returnnil;structjs_evente;intr=read(fd,&e,sizeof(e));if(r==-1&&errno==EAGAIN){/* No more events. */returnnil;}elseif(r==-1){/* An actual read error (joystick unplugged, etc.). */emacs_valuesignal=env->intern(env,"file-error");constchar*error=strerror(errno);size_tlen=strlen(error);emacs_valuemessage=env->make_string(env,error,len);env->non_local_exit_signal(env,signal,message);returnnil;}else{/* Fill out event vector. */emacs_valuev=args[1];emacs_valuetype=e.type&JS_EVENT_BUTTON?button:axis;emacs_valuevalue;if(type==button)value=e.value?t:nil;elsevalue=env->make_float(env,e.value/(double)INT16_MAX);env->vec_set(env,v,0,env->make_integer(env,e.time));env->vec_set(env,v,1,type);env->vec_set(env,v,2,value);env->vec_set(env,v,3,env->make_integer(env,e.number));env->vec_set(env,v,4,e.type&JS_EVENT_INIT?t:nil);returnargs[1];}}
As before, extract the first argument and check for a signal. Then
call read(2) to get an event. If the read fails with EAGAIN, it’s
not a real failure. There are just no more events, so return nil.
structjs_evente;intr=read(fd,&e,sizeof(e));if(r==-1&&errno==EAGAIN){/* No more events. */returnnil;}
If the read failed with something else — perhaps the joystick was
unplugged — signal an error. The strerror(3) string is used for the
signal data.
if(r==-1){/* An actual read error (joystick unplugged, etc.). */emacs_valuesignal=env->intern(env,"file-error");constchar*error=strerror(errno);emacs_valuemessage=env->make_string(env,error,strlen(error));env->non_local_exit_signal(env,signal,message);returnnil;}
Otherwise fill out the event vector. If the second argument isn’t a
vector, or if it’s too short, the signal will automatically get raised
by Emacs. The module can keep plowing through the vec_set() calls
safely since it’s not committing to anything.
/* Fill out event vector. */emacs_valuev=args[1];emacs_valuetype=e.type&JS_EVENT_BUTTON?button:axis;emacs_valuevalue;if(type==button)value=e.value?t:nil;elsevalue=env->make_float(env,e.value/(double)INT16_MAX);env->vec_set(env,v,0,env->make_integer(env,e.time));env->vec_set(env,v,1,type);env->vec_set(env,v,2,value);env->vec_set(env,v,3,env->make_integer(env,e.number));env->vec_set(env,v,4,e.type&JS_EVENT_INIT?t:nil);returnargs[1];
The Linux event struct has four fields and the function fills out five
values of the vector. This is because the type field has a bit flag
indicating initialization events. This is split out into an extra
t/nil value. It also normalizes axis values and converts button values
into t/nil, which makes more sense for Emacs Lisp. The event itself is
returned since it’s a truthy value and it’s convenient for the caller.
The astute programmer might notice that the negative side of the axis
could go just below -1.0, since INT16_MIN has one extra value over
INT16_MAX (two’s complement). It doesn’t seem to be documented, but
the joystick drivers I’ve seen never exactly return INT16_MIN, so
this is in fact the correct way to normalize it.
Initialization
Update 2021: In a previous version of this article, I talked about
interning symbols during initialziation so that they do not need to be
re-interned each time the module is called. This no longer works,
and it was probably never intended to be work in the first place. The
lesson is simple: Do not reuse Emacs objects between module calls.
First grab the fset symbol since this function will be needed to bind
names to the module’s functions.
emacs_valuefset=env->intern(env,"fset");
Using fset, bind the functions. The second and third arguments to
make_function are the minimum and maximum number of arguments, which
may look familiar. The last argument is that closure pointer
I mentioned at the beginning.
The source repository now includes a port to Windows (XInput). If
you’re on Linux or Windows, have Emacs 25 with modules enabled, and a
joystick is plugged in, then make run in the repository should bring
up Emacs running a joystick calibration demonstration. The module
can’t poke at Emacs when events are ready, so instead there’s a timer
that polls the module for events.
I’d like to someday see an Emacs Lisp game well-suited for a joystick.
Four Saturdays ago, on September 17, Emacs 25 was finally released. Almost two
years before that, I wrote a post predicting a few Big things to expect from
Emacs 25. Throughout the months since then, I’ve also been reporting interesting
new features as they arrived on the dev builds. Today, we compile a list of all
of those news posts, and review which predictions I actually got right.
News posts
For the convenience of those looking for actually practical information, I feel
we should start with a list of the posts. I must strive to be clear that this
is not a comprehensive list of new features. Not by a mile! If you want
everything, you can have a look at the news file that ships with Emacs,
/etc/NEWS, or you can read Mickey Petersen’s commented version.
The list below merely sums up most (not even all) of the features that piqued
my interest over the last couple of years.
If you’re short on time, I personally recommend checking out char-folding
search (1), archive priorities (17), and the comment-line command (4), which are all
features you might not realize are there. The round-quotes in Help buffers (2) and
the improved query-replace history (3) are also among my favorites, but you don’t
need to do anything special to see them in action.
Reviewing predictions
Now, for the indulgence of those looking for more than just practical
information. On the aforementioned post, I talked about 6 things I hoped to see
by the time Emacs 25 came out.
Moving the code base to git
This one even had a schedule date when I wrote
the post, so it obviously doesn’t count as a prediction. Still, I’m
extremely glad it happened as I would never have contributed myself if not
for this change.
Dynamic library loading
Now this is exciting. I originally wrote it was
“looking almost in shape for release”, but I have no idea what led me to
say that. This feature actually almost didn’t make it into release. It only
got merged to the emacs-25 branch more than one year later, after the
feature-freeze took place!
Nonetheless, it’s finally out and about, and I’m eager to see what comes
out of it. If you’re interested, it’s actually very easy to get started
(well… easy by C-coding standards). Aurélien Aptel has a super short
tutorial on how to write, compile, and use a dynamic module in Emacs.
Concurrency in Elisp
I’ll count this as a half point. We don’t have proper
concurrency in Elisp yet, and whether that’s something we want is still an
ongoing discussion. Still, most of the points in my rant have already been
addressed. The packages list does refresh asynchronously now (despite the
lack of concurrency), and several optimizations have been made which speed
up the actual rendering of the list.
A Better Package Menu
I shouldn’t have to say much here. The last 5 links on
list above are all improvements to the package menu. While there’s still
room for improvement, it’s quite ahead of where it was 2 years ago. Better
dependency management, asynchronous refreshing, and more filtering options
were all much needed improvements. Meanwhile, archive priorities are a nice
cherry to top of it all off.
A More Robust Customize Interface
Out of the two issues that led me to
write this point, one of them has been fixed (custom-theme-load-path is no
longer a defcustom). Still, I’m counting this as a miss. Over the last
couple of years I’ve seen other fundamental issues with the Customize
interface be reported, and there’s still quite a bit of work to be done here.
Namespaces
Well, OK. We still don’t have namespaces. Honestly, though, it
doesn’t bother me anymore. I’ve released two packages, Names and Nameless,
which are available on Elpa and help mitigate the the lack of namespaces. I
use Nameless for all my Elisp coding now and that’s the end of the story
for me.
Summarizing, even if we exclude the Git point (which wasn’t really a
prediction), it’s a 2½ out of 5 (or 3½ if we count namespaces). That’s a
pretty good outcome considering I had never even made a single contribution to
Emacs core when I wrote that post. I’d love to write another one of those, but
I’m not nearly as involved in the news as I was 2 years ago. Maybe in a couple
of months I’ll manage to catch up.
This is a follow-up to , as well as the start of a series on using Emacs effectively with a number of languages.
I'll go into what I use for editing python with Emacs and why, as well as how I set it all up. Let's go!
Recently, I've been looking at
libigl. I
didn't manage to fully figure out their CMake build system for
tutorials: although each tutorial has a CMakeLists.txt, it's only
possible to build them all at once.
So I decided to replace CMakeLists.txt with a good-old Makefile;
how hard can it be? Concerning includes, not at all hard: the missing
files are found with counsel-locate and added to the include path.
But I had some trouble matching a missing ld dependency to a library
file. Fixed it with a bunch of googling and guesswork; I still wonder
if there's a better way. But in the process, I've found this useful
command:
readelf --syms libGL.so
which produces e.g.:
Symbol table '.dynsym' contains 2732 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 000000000004faf0 0 SECTION LOCAL DEFAULT 8
2: 00000000000e8f20 0 FUNC GLOBAL DEFAULT 11 glGetIntegerui64i_vNV
3: 00000000000e13e0 0 FUNC GLOBAL DEFAULT 11 glGetMultiTexEnvfvEXT
4: 00000000000d7440 0 FUNC GLOBAL DEFAULT 11 glProgramUniform2uiv
5: 00000000000cfdc0 0 FUNC GLOBAL DEFAULT 11 glMultiTexCoord3sv
This is a pretty good representation of a binary file: in this
example, instead of one megabyte of gibberish I see a bit more than
2732 lines describing the functions this file uses and provides.
Viewing the symbol list automatically
I liked the above representation so much that I want to see it by
default. In Emacs, it's pretty easy to do with auto-mode-alist:
The idea is very simple: elf-mode is a toggle function that replaces
the buffer contents with the shell command output. It carefully uses
read-only-mode and set-buffer-modified-p so that the file will not
be overwritten by accident with the symbol names.
Using autoload to avoid overhead
As you might imagine, looking at binaries isn't really a common
task. Is it worth to be dragging this code around from now on, loading
it on each start? The answer is yes, of course. Since the actual cost
is negligible until the feature is used.
If you look above, elf-mode has an ;;;###autoload cookie before
it. The cookie results in this line in my loaddefs.el:
(autoload'elf-mode"modes/ora-elf"""tnil)
My init.el always loads loaddefs.el, but never loads ora-elf.el
where the function is defined. That file is only loaded when the
function elf-mode is called for the first time. The above autoload
statement simply instructs Emacs to load a particular file when
elf-mode needs to be called.
When you use the package manager, the autoloads file is generated and
loaded for you automatically:
Here, the package manager will always load ace-link-autoloads.el,
which instructs Emacs to load ace-link.el when one of the
;;;###autoload functions is called and ace-link.el isn't yet
loaded.
As an example of how useful delayed loading is: my 6000 line
config starts in 1.8 seconds.
About 40% of that time is spent on (package-initialize), which I
assume is the package manager loading all those *-autoloads.el files
that I have in my elpa/.
Outro
Let me know if there's interest to have elf-mode on MELPA. Also, if
anyone knows how to set mode automatically based on the first few
chars of the file (all binaries seem to start with ^?ELF), I'd like
to know that as well. Happy hacking!
The end of the month marks Elfeed’s third birthday. Surprising
to nobody, it’s also been three years of heavy, daily use by me. While
I’ve used Elfeed concurrently on a number of different machines over
this period, I’ve managed to keep an Elfeed database index
with a lineage going all the way back to the initial development
stages, before the announcement. It’s a large, organically-grown
database that serves as a daily performance stress test. Hopefully
this means I’m one of the first people to have trouble if an invisible
threshold is ever exceeded.
I’m also the sort of person who gets excited when I come across an
interesting dataset, and I have this gem sitting right in front of me.
So a couple of days ago I pushed a new Elfeed function,
elfeed-csv-export, which exports a database index into three CSV
files. These are intended to serve as three tables in a SQL database,
exposing the database to interesting relational queries and joins.
Entry content (HTML, etc.) has always been considered volatile, so
this is not exported. The export function isn’t interactive (yet?), so
if you want to generate your own you’ll need to (require
'elfeed-csv) and evaluate it yourself.
All the source code for performing the analysis below on your own
database can be found here:
Web authors are notoriously awful at picking actually-unique entry
IDs, even when using the smarter option, Atom. I still simply
don’t trust that entry IDs are unique, so, as usual, I’ve qualified
them by their source feed URL, hence the primary key on both columns
in entries.
At this point I wish I had collected a lot more information. If I were
to start fresh today, Elfeed’s database schema would not only fully
match Atom’s schema, but also exceed it with additional logging:
When was each entry actually fetched?
How did each entry change since the last fetch?
When and for what reason did a feed fetch fail?
When did an entry stop appearing in a feed?
How long did fetching take?
How long did parsing take?
Which computer (hostname) performed the fetch?
What interesting HTTP headers were included?
Even if not kept for archival, how large was the content?
I may start tracking some of these. If I don’t, I’ll be kicking myself
three years from now when I look at this again.
A look at my index
So just how big is my index? It’s 25MB uncompressed, 2.5MB
compressed. I currently follow 117 feeds, but my index includes
43,821 entries from 309 feeds. These entries are marked with
53,360 tags from a set of 35 unique tags. Some of these datapoints
are the result of temporarily debugging Elfeed issues and don’t
represent content that I actually follow. I’m more careful these days
to test in a temporary database as to avoid contamination. Some are
duplicates due to feeds changing URLs over the years. Some are
artifacts from old bugs. This all represents a bit of noise, but
should be negligible. During my analysis I noticed some of these
anomalies and took a moment to clean up obviously bogus data (weird
dates, etc.), all by adjusting tags.
The first thing I wanted to know is the weekday frequency. A number of
times I’ve blown entire Sundays working on Elfeed, and, as if to
frustrate my testing, it’s not unusual for several hours to pass
between new entries on Sundays. Is this just my perception or are
Sundays really that slow?
Here’s my query. I’m using SQLite’s strftime to shift the
result into my local time zone, Eastern Time. This time zone is the
source, or close to the source, of a large amount of the content. This
also automatically accounts for daylight savings time, which can’t be
done with a simple divide and subtract.
The most frequent tag (13,666 appearances) is “youtube”, which marks
every YouTube video, and I’ll use gnuplot to visualize it. The input
“file” is actually a command since gnuplot is poor at filtering data
itself, especially for histograms.
plot '< grep ^youtube, weekdays.csv' using 2:3 with boxes
Wow, things do quiet down dramatically on weekends! From the
glass-half-full perspective, this gives me a chance to catch up when I
inevitably fall behind on these videos during the week.
The same is basically true for other types of content, including
“comic” (12,465 entries) and “blog” (7,505 entries).
However, “emacs” (2,404 entries) is a different story. It doesn’t slow
down on the weekend, but Emacs users sure love to talk about Emacs on
Mondays. In my own index, this spike largely comes from Planet
Emacsen. Initially I thought maybe this was an artifact of
Planet Emacsen’s date handling — i.e. perhaps it does a big fetch on
Mondays and groups up the dates — but I double checked: they pass the
date directly through from the original articles.
Conclusion: Emacs users love Mondays. Or maybe they hate Mondays and
talk about Emacs as an escape.
I can reuse the same query to look at different time scales. When
during the day do entries appear? Adjusting the time zone here becomes
a lot more important.
Emacs bloggers tend to follow a nice Eastern Time sleeping schedule.
(I wonder how Vim bloggers compare, since, as an Emacs user, I
naturally assume Vim users’ schedules are as undisciplined as their
bathing habits.) However, this also might be prolific the
Irreal breaking the curve.
The YouTube channels I follow are a bit more erratic, but there’s
still a big drop in the early morning and a spike in the early
afternoon. It’s unclear if the timestamp published in the feed is the
upload time or the publication time. This would make a difference in
the result (e.g. overnight video uploads).
December is a big drop across all tags, probably for the holidays.
Both “comic” and “blog” also have an interesting drop in August. For
brevity, I’ll only show one. This might be partially due my not
waiting until the end of this month for this analysis, since there are
only 2.5 Augusts in my 3-year dataset.
Unfortunately the timestamp is the only direct numerical quantity in
the data. So far I’ve been binning data points and counting to get a
second numerical quantity. Everything else is text, so I’ll need to
get more creative to find other interesting relationships.
So let’s have a look a the lengths of entry titles.
The shortest are the webcomics. I’ve complained about poor webcomic
titles before, so this isn’t surprising. The spikes are from
comics that follow a strict (uncreative) title format.
Emacs article titles follow a nice distribution. You can tell these
are programmers because so many titles are exactly 32 characters long.
Picking this number is such a natural instinct that we aren’t even
aware of it. Or maybe all their database schemas have VARCHAR(32)
title columns?
Blogs in general follow a nice distribution. The big spike is from the
Dwarf Fortress development blog, which follows a strict date
format.
The longest on average are YouTube videos. This is largely due to the
kinds of videos I watch (“Let’s Play” videos), which tend to have
long, predictable names.
And finally, here’s the most interesting-looking graph of them all.
This is the title length versus time of day (not binned). Each point
is one of the 53,360 posts.
set style fill transparent solid 0.25 noborder
set style circle radius 0.04
plot 'length-vs-daytime.csv' using 1:2 with circles
(This is a good one to follow through to the full size image.)
Again, all Eastern Time since I’m self-centered like that. Vertical
lines are authors rounding their post dates to the hour. Horizontal
lines are the length spikes from above, such as the line of entries at
title length 10 in the evening (Dwarf Fortress blog). There’s a the
mid-day cloud of entries of various title lengths, with the shortest
title cloud around mid-morning. That’s probably when many of the
webcomics come up.
Additional analysis could look further at textual content, beyond
simply length, in some quantitative way (n-grams? soundex?). But
mostly I really need to keep track of more data!
If you've ever tried the swiper-all from
swiper, forget everything about
it. The command was super-awkward, since it had to parse all your
open files before giving you a chance enter anything, resulting in
dozens of seconds before the prompt.
Recently, I've had some time to examine and improve it and the result
looks very promising. The new command is now async, which means
there's no delay before the prompt comes up. Here's a result I got
with no delay while having around 50 buffers open:
The shortcut I'm using:
(global-set-key(kbd"C-c u")'swiper-all)
For efficiency's sake a small trade off had to be made: the line
numbers are no longer displayed. This actually results in an advantage
that you can select different candidates on the same line.
There are still a few things I plan to try for the new command, like
adding file-less buffers, caching for incremental regexes and maybe
even newlines in wild cards, but even now it seems very usable. So
give it a try, enjoy and happy hacking!
One of the questions we get most often about CIDER is “can I configure X on a
per-project basis?”. Occasionally, you find someone suggesting (or even
implementing) some sophisticated configurable variable trying to account for
multiple simultaneous use-cases. Fortunately that’s one effort we don’t need to
make. Emacs already has that built-in in the form of directory-local variables
(dir-local for short).
As the name implies, dir-local variable values (can) apply to all files inside a
given directory (also applying recursively to its sub-directories), though you
can also restrict them by major-mode or by subdirectory. To configure it:
This will prompt you for a major-mode, then a variable name, and finally a
variable value.
Finally, it will drop you in a (newly-created) file called .dir-locals.el
in the current directory. Just save this file and you’re done!
By doing this, you have configured the given variable to always have the
provided value anywhere inside the current directory, so make sure you’re at
the root of your project when you call the command. If you use Projectile,
there’s the projectile-edit-dir-locals command for doing just that.
Also worth noting:
If you type nil for the major-mode, it applies to all major-modes.
There is (of course) tab-completion in the variable-name prompt.
If you answer eval for the variable name, it will prompt you for a lisp
expression, instead of a value. This expression will be saved and will be
evaluated every time a file is visited.
After doing this, call revert-buffer on any previously-open files to
apply the new value.
Update 21 Jul 2016
It’s worth mentioning that Joel McCracken posted a similar quick guide a few
years ago. Some of the information is the same, but some is complementary, so
you might want to have a look.
Data processing time will becomes longer and longer as the increasing
rate of data volumes. Users may check-in frequently to see the whether
it is finished. Most of the time they will found it hasn't, doing this
the users make contact switch which break the flow of whatever the
using was doing.
Sometimes the user can't stop doing so, either because they are
impatient, or because they really have a deadline to catch. Also, with
less likelihood, they might find errors in the processing, either
because of the QC check fails, or running out of computational
resources.
Giving this fact, it really makes sens to have your program actively
inform the user on the process so that they doesn't need to check-in
at all. Because users will be notified immediately whenever the
whole progressing is completed, or there's error that the user needs
to take action onup.
This blog posts walk though the basics of sending Emails in Python,
composing and sending out Emails. Each component is broken down into
small piceses. It helps you debug/tests Email program, and personalise
your Emails. In the end, you should be able to build a email robot.
Prerequisite
Before going into the technical details, you have to check that you
are able to send out emails. You need the
[ ] SMTP server,
[ ] user name,
[ ] password,
[ ] port number, and
[ ] communication protocol.
You could easily found out these information from your Email service
provider. An example of Gmail is at Here.
To check if you have all the information correct, run the following
snippet. It will try to send out an empty Email to yourself. Make
sure you fill in the username and password before hit go.
importsmtplibusername=<FillIn>password=<FillIn>conn=smtplib.SMTP("smtp.gmail.com")conn.starttls()# set connection to TLS mode
conn.login(username,password,)# Log in to the remote server
conn.sendmail(username,[username],'For testing')# Send emails
conn.quit()# close connection.
If you don't see error messages, that's great, you are all set.
There should be an Email in your Inbox. It has no subject and for
testing in the main body.
If you do, you need to double check the information, and try
again. If you are sure that the information are correct, btu still
can't send out, check your network configuration, maybe the firewall
block the connection.
Once you are able to send out an empty email the next is to compose
an full Email.
Compose An Email in Python
An Email is consisted of multiple parts, the Subject, Body,
attachments, Signature, and also some meta-data including from, to,
and date.
Firstly, create an object of MIMEMutiple() class. It will be the
building block of your Email. The approached described here is to
add each components into it.
body_txt='''
Hello,
Just to tell you Python is awesome.
'''plain_body=MIMEText(body_txt,"plain")
You could also try to compose an complex HTML email in Python, or
use diffenret tool to generate the HTML and import in Python.
html_txt=r'''\
<html>
<head></head>
<body>
<p>Hi!<br>
How are you?<br>
Here is the <a href="http://www.python.org">link</a> you wanted.
</p>
</body>
</html>
'''html_body=email.mime.text.MIMEText(html_txt,'html')
Attachment
You can attach files of any type in an Email, specially, for image,
you could use MIMEImage, and for audio, you could use
MIMEAudio.
But you don't have to be specific. MIMEApplication would be
sufficeincy for all cases. It will configure the file type of the
attached file. Use it as follows:
withopen(fpath,'rb')asfp:part=MIMEApplication(fp.read(),Name=basename(fpath))# file content as string
part['Content-Disposition']='attachment; filename="%s"'%basename(f)# attachment description.
msg.attach(part)# attach to the msg.
Put Everything Together
fromos.pathimportbasenameimportsmtplibfromemail.mime.applicationimportMIMEApplicationfromemail.mime.multipartimportMIMEMultipartfromemail.mime.textimportMIMETextfromemail.utilsimportformatdate# Configure your email
username=<Fillin>password=<Fillin>recipents=<Fillin>subject='Hey'attachments=[]# add attachment here.
body_txt='''
Hello,
Just to tell you Python is awesome.
'''# Email - meta data
msg=MIMEMultipart()msg['From']=usernamemsg['To']=', '.join(recipents)msg['Subject']=subjectmsg['Date']=formatdate(localtime=True)# standard.
# Email - main body
plain_body=MIMEText(body_txt,"plain")msg.attach(plain_body)# attachments
forfpathinattachmentsor[]:withopen(fpath,'rb')asfp:part=MIMEApplication(fp.read(),Name=basename(fpath))# file content as string
part['Content-Disposition']='attachment; filename="%s"'%basename(fpath)# attachment description.
msg.attach(part)# attach to the msg.
# send out email
conn=smtplib.SMTP("smtp.gmail.com")conn.starttls()# set connection to TLS mode
conn.login(username,password)# Log in to the remote server
conn.sendmail(username,[username],msg.as_string())# Send emails
conn.quit()# close connection.
Wrap everything in a Class
At JBARML, we are planing to send users emails for a progress
update. Many of the data generating process are lured together and
automated by a workflow manager. The whole programcan takes upto
weeks to complete. Actively sending update progeress to user is
much more senssible then user loggin in to a remote machien and
check it now and then.
In this case,
email to notify user the progress of the data
generating workflow.
Once you learn paredit, you really can’t go back. Even when you’re not editing
lisp you crave for the ease of manipulating everything as balanced sexps.
Although that’s (sadly) not always possible, you can still hack your way into a
bit of guilty paren-pleasure in pretty much any editing session.
While not all programming languages are lisps, most of them do have brackets and
quotes. And paredit should have no problem moving you forward-up out of a C
string, wrapping curly brackets around a work in LaTeX, or even splicing out a
pair of parenthesis in plain prose.
Below are a few keys I find useful pretty much everywhere, so I’ve allowed them
to take over the global keymap.
(global-set-key(kbd"C-M-u")#'paredit-backward-up)(global-set-key(kbd"C-M-n")#'paredit-forward-up);; This one's surpisingly useful for writing prose.(global-set-key"\M-S"#'paredit-splice-sexp-killing-backward)(global-set-key"\M-R"#'paredit-raise-sexp)(global-set-key"\M-("#'paredit-wrap-round)(global-set-key"\M-["#'paredit-wrap-square)(global-set-key"\M-{"#'paredit-wrap-curly)
Today's post is about the newest feature related to the
ivy-switch-buffer command. If you
use ivy-mode, you're probably already using ivy-switch-buffer
since it overwrites the built-in switch-to-buffer.
The cool thing about ivy-switch-buffer is that it's not only buffers
that are offered for completion. Other buffer-like entities can be
there as well: bookmarks, recently opened files (recentf), and
finally, window layouts. Since all of those are relatively the same
concept, it's very convenient to have them all in one place available
for completion.
Here are the relevant settings:
;; Enable bookmarks and recentf(setqivy-use-virtual-bufferst);; Example setting for ivy-views(setqivy-views`(("dutch + notes {}"(vert(file"dutch.org")(buffer"notes")))("ivy.el {}"(horz(file,(find-library-name"ivy"))(buffer"*scratch*")))))
I did mention ivy-views before in the
ivy-0.8.0 release post.
But now, instead of setting ivy-views by hand, you can also bind
ivy-push-view to a key and store as many window configurations as
you like, really fast.
What gets stored:
The window list - all windows open on the current frame.
The window splits relative to each other as a tree. Currently, the
size of the split isn't saved, all windows are split equally.
The point positions in each window. If you use just one window,
you've got something similar to bookmark-set.
Suppose I have two files open: the file 2016-06-23-ivy-push-view.md
and the _posts directory. By pressing C-c v I am prompted
for a view name with the default being e.g.
{} 2016-06-23-ivy-push-view.md _posts 2.
I can still name the view however I want, but I typically just press
RET. The parts of the automatic view name are:
{} - this is a simple string marker to distinguish the views in
the buffer view. If I enter only {} into ivy-switch-buffer
prompt, the candidates will normally filter to only views, since
very rarely will a file or a buffer name match {}.
2016-06-23-ivy-push-view.md _posts is the list of buffers stored
in the view. This view has only two buffers, but ivy-push-view can
handle as many windows as you can cram into a single frame.
2 means that I already have two views with the same buffers, each
new view with the same buffers gets an increased number for the
suggested name. And it's not useless to have many views for the same
buffers, since the views also store point positions, not just the
window list.
Here's the beauty of it for me: when I type _posts into
ivy-switch-buffer I can chose to open the _posts directory in a
variety of ways:
If the buffer is currently open, I can just switch there.
If the buffer is currently closed, I can re-open it, thanks to recentf.
I can open the buffer as part of a stored view(s) in ivy-views.
Finally, if I decide that I don't need a particular view any more, I
can delete it with C-c V (ivy-pop-view). It's possible to
delete many views at once by pressing C-M-m (ivy-call),
as usual with most ivy completion functions.
Breaking API change
While implementing ivy-set-view I decided that the current way alist
collections are handled together with actions is sub-optimal. Here's
the new way of working:
Previously, the return result would be 3, i.e. the cdr of the
selected candidate. Any code using ivy-read with an alist-type
collection will break. I fixed all instances in counsel.el, and
there actually aren't too many uses in the published third party
packages.
A simple fix to the problem is to use cdr in the action function.
Additionally, having more information available in the action function
will serve to improve the code.
After last week's post, Clément Pit-Claudel informed us of an alternative method
for providing input to compilations. I have no idea how I’d never learned about
that before, but I figure that other people might be in the same situation so
it’s worth a post. Have a look at the Update at the end of the post.
This morning I pushed out an important update to Elfeed, my
web feed reader for Emacs. The update should be available in MELPA by
the time you read this. Elfeed now has support for fetching feeds
using a cURL through a curl inferior process. You’ll need
the program in your PATH or configured through
elfeed-curl-program-name.
I’ve been using it for a couple of days now, but, while I work out the
remaining kinks, it’s disabled by default. So in addition to having
cURL installed, you’ll need to set elfeed-use-curl to non-nil.
Sometime soon it will be enabled by default whenever cURL is
available. The original url-retrieve fetcher will remain in place
for time time being. However, cURL may become a requirement someday.
Fetching with a curl inferior process has some huge advantages.
It’s much faster
The most obvious change is that you should experience a huge speedup
on updates and better responsiveness during updates after the first
cURL run. There are important two reasons:
Asynchronous DNS and TCP: Emacs 24 and earlier performs DNS
queries synchronously even for asynchronous network processes. This is
being fixed on some platforms (including Linux) in Emacs 25, but now
we don’t have to wait.
On Windows it’s even worse: the TCP connection is also established
synchronously. This is especially bad when fetching relatively small
items such as feeds, because the DNS look-up and TCP handshake dominate
the overall fetch time. It essentially makes the whole process
synchronous.
Conditional GET: HTTP has two mechanism to avoid transmitting
information that a client has previously fetched. One is the
Last-Modified header delivered by the server with the content. When
querying again later, the client echos the date back like a
token in the If-Modified-Since header.
The second is the “entity tag,” an arbitrary server-selected token
associated with each version of the content. The server delivers it
along with the content in the ETag header, and the client hands it
back later in the If-None-Match header, sort of like a cookie.
This is highly valuable for feeds because, unless the feed is
particularly active, most of the time the feed hasn’t been updated
since the last query. This avoids sending anything other hand a
handful of headers each way. In Elfeed’s case, it means it doesn’t
have to parse the same XML over and over again.
Both of these being outside of cURL’s scope, Elfeed has to manage
conditional GET itself. I had no control over the HTTP headers until
now, so I couldn’t take advantage of it. Emacs’ url-retrieve
function allows for sending custom headers through dynamically binding
url-request-extra-headers, but this isn’t available when calling
url-queue-retrieve since the request itself is created
asynchronously.
Both the ETag and Last-Modified values are stored in the database and
persist across sessions. This is the reason the full speedup isn’t
realized until the second fetch. The initial cURL fetch doesn’t have
these values.
Fewer bugs
As mentioned previously, Emacs has a built-in URL retrieval library
called url. The central function is url-retrieve which
asynchronously fetches the content at an arbitrary URL (usually HTTP)
and delivers the buffer and status to a callback when it’s ready.
There’s also a queue front-end for it, url-queue-retrieve which
limits the number of parallel connections. Elfeed hands this function
a pile of feed URLs all at once and it fetches them N at a time.
Unfortunately both these functions are incredibly buggy. It’s been a
thorn in my side for years.
It takes a URL and a callback. Seeing this, the sane, unsurprising
expectation is the callback will be invoked exactly once for time
url-retrieve was called. In any case where the request fails, it
should report it through the callback. This is not the case.
The callback may be invoked any number of times, including zero.
In this example, suppose you have a webserver that will return an HTTP
404 for a requested URL. Below, I fire off 10 asynchronous requests in a
row.
What would you guess is the length of results? It’s initially 0
before any requests complete and over time (a very short time) I would
expect this to top out at 10. On Emacs 24, here’s the real answer:
(lengthresults);; => 46
The same error is reported multiple times to the callback. At least
the pattern is obvious.
What’s the length of results? This time it’s zero. Remember how DNS
is synchronous? Because of this, DNS failures are reported
synchronously as a signaled error. This gets a lot worse with
url-queue-retrieve. Since the request is put off until later, DNS
doesn’t fail until later, and you get neither a callback nor an error
signal. This also puts the queue in a bad state and necessitated
elfeed-unjam for manually clear it. This one should get fixed in
Emacs 25 when DNS is asynchronous.
This last one assumes you don’t have anything listening on port 57432
(pulled out of nowhere) so that the connection fails.
On Linux, we finally get the sane result of 10. However, on Windows,
it’s zero. The synchronous TCP connection will fail, signaling an
error just like DNS failures. Not only is it broken, it’s broken in
different ways on different platforms.
There are many more cases of callback weirdness which depend on the
connection and HTTP session being in various states when thing go
awry. These were just the easiest to demonstrate. By using cURL, I get
to bypass this mess.
No more GnuTLS issues
At compile time, Emacs can optionally be linked against GnuTLS, giving
it robust TLS support so long as the shared library is available.
url-retrieve uses this for fetching HTTPS content. Unfortunately,
this library is noisy and will occasionally echo non-informational
messages in the minibuffer and in *Messages* that cannot be
suppressed.
When not linked against GnuTLS, Emacs will instead run the GnuTLS
command line program as an inferior process, just like Elfeed now does
with cURL. Unfortunately this interface is very slow and frequently
fails, basically preventing Elfeed from fetching HTTPS feeds. I
suspect it’s in part due to an improper coding-system-for-read.
cURL handles all the TLS negotation itself, so both these problems
disappear. The compile-time configuration doesn’t matter.
Windows is now supported
Emacs’ Windows networking code is so unstable, even in Emacs 25, that
I couldn’t make any practical use of Elfeed on that platform. Even the
Cygwin emacs-w32 version couldn’t cut it. It hard crashes Emacs every
time I’ve tried to fetch feeds. Fortunately the inferior process code
is a whole lot more stable, meaning fetching with cURL works great. As
of today, you can now use Elfeed on Windows. The biggest obstable is
getting cURL installed and configured.
Interface changes
With cURL, obviously the values of url-queue-timeout and
url-queue-parallel-processes no longer have any meaning to Elfeed.
If you set these for yourself, you should instead call the functions
elfeed-set-timeout and elfeed-set-max-connections, which will do
the appropriate thing depending on the value of elfeed-use-curl.
Each also comes with a getter so you can query the current value.
The deprecated elfeed-max-connections has been removed.
Feed objects now have meta tags :etag, :last-modified, and
:canonical-url. The latter can identify feeds that have been moved,
though it needs a real UI.
See any bugs?
If you use Elfeed, grab the current update and give the cURL fetcher a
shot. Please open a ticket if you find problems. Be sure to report
your Emacs version, operating system, and cURL version.
As of this writing there’s just one thing missing compared to
url-queue: connection reuse. cURL supports it, so I just need to code
it up.
The Emacs compile command is a severely underused tool. It allows you to run
any build tool under the sun and provides error-highlighting and jump-to-error
functionality for dozens of programming languages, but many an Emacser is still
in the habit of switching to a terminal in order to run make, lein test, or
bundle exec. It does have one limitation, though. The compilation buffer is
not a real shell, so if the command being run asks for user input (even a simple
y/n confirmation) there’s no way to provide it.
(Since posting this, I’ve learned that part of it is mildly futile. Read the
update below for more information.)
Fortunately, that’s not hard to fix. The snippet below defines two commands. The
first one prompts you for input and then sends it to the underlying terminal
followed by a newline, designed for use with prompts and momentary REPLs. The
second is a command that simply sends the key that was pressed to invoke it,
designed for easily replying to y/n questions or quickly quitting REPLs with
C-d or C-j.
(defunendless/send-input(input&optionalnl)"Send INPUT to the current process.
Interactively also sends a terminating newline."(interactive"MInput: \nd")(let((string(concatinput(ifnl"\n"))));; This is just for visual feedback.(let((inhibit-read-onlyt))(insert-before-markersstring));; This is the important part.(process-send-string(get-buffer-process(current-buffer))string)))(defunendless/send-self()"Send the pressed key to the current process."(interactive)(endless/send-input(apply#'string(append(this-command-keys-vector)nil))))(dolist(key'("\C-d""\C-j""y""n"))(define-keycompilation-mode-mapkey#'endless/send-self))
This is something I’ve run into for years, but I finally decided to fix it
because it meant I couldn’t run Ruby’s rspec in the compilation buffer if my
code contained a binding.pry (which spawns a REPL). Now I can actually interact
with this REPL via C-c i or just quickly get rid of it with C-d. If you run
into the same situation, you should also set the following option in your
.pryrc file.
Pry.config.pager=falseifENV["INSIDE_EMACS"]
Update 17 Jun 2016
As Clément points out in the comments, you can run compilation commands in
comint-mode by providing the C-u prefix to M-xcompile. You still have
all of the usual compilation-mode features (like next/previous-error), with
the additional benefit that the buffer accepts input like a regular shell does.
The only caveat is that, since the buffer is modifiable, you lose some
convenience keys like q to quit the buffer or g to recompile, so you’ll need
to bind them somewhere else
I still like it that the previous solution gives me quick access to C-d and
y/n for those cases when I forget to use comint-mode, but the solution I had
for inputting long strings is definitely redundant now. Instead, we can have a
key that restarts the current compilation in comint-mode.
(require'cl-lib)(defunendless/toggle-comint-compilation()"Restart compilation with (or without) `comint-mode'."(interactive)(cl-callf(lambda(mode)(if(eqmodet)nilt))(eltcompilation-arguments1))(recompile))(define-keycompilation-mode-map(kbd"C-c i")#'endless/toggle-comint-compilation)(define-keycompilation-minor-mode-map(kbd"C-c i")#'endless/toggle-comint-compilation)(define-keycompilation-shell-minor-mode-map(kbd"C-c i")#'endless/toggle-comint-compilation)
I'd like to show off a certain Elisp productivity booster that I've
had in an unfinished state for a while and finished just today.
A large part of tweaking Elisp is simply setting variables. The new
command, counsel-set-variable,
allows to set them quite a bit faster.
Completion stage 1:
First of all, you get completion for all variables that you have defined:
Completion stage 2:
Once a symbol is selected, the code checks whether the symbol is a
defcustom with type 'boolean or 'radio. Since then it is
possible to offer all values that the symbol is allowed to become for
completion.
For example, here's a typical 'radio-type definition:
(defcustomavy-style'at-full"The default method of displaying the overlays.Use `avy-styles-alist' to customize this per-command.":type'(choice(const:tag"Pre"pre)(const:tag"At"at)(const:tag"At Full"at-full)(const:tag"Post"post)(const:tag"De Bruijn"de-bruijn)))
And here's a completion screen offered for this variable:
It is worth noting that the current value of the variable is
pre-selected, to give a nice reference point for the new setting.
In case the symbol isn't a boolean or a radio
Then you get a completion session similar to M-xread-expression, but with the initial contents already filled
in. For example:
The read-expression part combines well with this setting in my config:
Here's a series of commands using
lispy-mode that I would typically
use for the screenshot above, to set ivy-re-builders-alist to a new
value:
C-f (forward-char) to get into special.
-e (lispy-ace-subword) to mark the plus part of the code.
C-d (lispy-delete) to delete the active region.
After the C-f -e C-d chain of bindings, the minibuffer contents become:
(setqivy-re-builders-alist'((t.ivy--regex-|)))
I press C-M-i (completion-at-point) to get completion for
all symbols that start with ivy--regex-. Since I have ivy-mode on,
C-M-i starts a recursive completion session. I highly
recommend adding these settings to your config:
;; Allow to read from minibuffer while in minibuffer.(setqenable-recursive-minibufferst);; Show the minibuffer depth (when larger than 1)(minibuffer-depth-indicate-mode1)
Finally, I would e.g. select e.g. ivy--regex-fuzzy and RET
RET to finalize the eval. The first RET exits from
completion-at-point and the second RET exits from
counsel-set-variable.
Outro
I think this command, especially the newly added read-expression
part, is quite a bit faster than what I did before. That is switching
to *scratch* and typing in the setq manually and evaluating with
C-j. Here's my binding for the new command:
fill-paragraph is probably among the most underappreciated Emacs commands. I
use it dozens of times a day, and never stop to think of just how awesome and
practical it is. Still, we can make it a little bit better. Every once in a
while I need to “unfill” (or “unwrap”) a paragraph that’s broken over many
lines.
By being clever enough, we can make this into a free feature. There’s never any
reason to hit M-q twice on the same paragraph, so we can use that as our
keybind for the “unfill” command.
(defunendless/fill-or-unfill()"Like `fill-paragraph', but unfill if used twice."(interactive)(let((fill-column(if(eqlast-command'endless/fill-or-unfill)(progn(setqthis-commandnil)(point-max))fill-column)))(call-interactively#'fill-paragraph)))(global-set-key[remapfill-paragraph]#'endless/fill-or-unfill)
With this, M-q will act as a toggle. Hitting it once will do its usual thing
(even if the paragraph is already filled), but hitting it twice will completely
unwrap the current paragraph into a single line.
I wish I had reviewed this book when it first came out, over one year ago. Alas,
those were busier times and this piece of work deserved more than a short post
hastily written between seminars and group meetings. Fortunately, Mickey has
unknowingly gifted me a second opportunity, by making it half-off for its 1 year
anniversary, and you have until tomorrow to grab the discount.
When it comes to technical reading, it’s rare to find a book that is both packed
with content and high in quality. Anyone who’s ever written a long document will
tell you how difficult it is to maintain a consistent level of quality
throughout the pages. Somewhere around the 100th or so page the complexity
spirals out of control and you spend more time organizing the chapters (to avoid
repetition or maintain the ideal order) than actually writing anything.
And yet, somehow, Mastering Emacs masterfully delivers on both areas, clearly
showing itself as a labor of love, and a whole lotta sweat and hard work. The
book’s beautiful cover smoothly gives way to an interior that is consistently
easy on the eyes and a pleasure to read. Most importantly, all of this eye-candy
is made all the sweeter by the healthy dose of knowledge that lies underneath.
If you’re a relative newcomer to Emacs (and I say “relative newcomer” in the
broadest sense), this book will be your initiation into the cult. It starts with
an entire chapter on “The Way of Emacs”, before it ever mentions keybinds or
commands, which is exactly how I’d start a book as well. Once that is done, it
goes on to teach you hundreds of pages of Emacs fundamentals.
If I had to be nit-picky, it’s hard to say whether these first few chapters are
too long or just about right. When it comes to introducing new concepts, the
ideal speed varies largely from reader to reader anyway. I also disagree with
recommending starter kits, but that’s about all the criticism I can come up
with for a couple hundred of pages.
Given all of the above, it’s very easy to recommend Mastering Emacs to Emacs
beginners. They are, after all, the target audience of this book, and they’ll
probably still be digesting its lessons 6 months from now.
But what about intermediate to advanced users? Despite not being the target
audience, it’s far from a worthless experience for them. The thing to understand
is that whenever a guru like Mickey puts down so many thoughts into words,
everyone is bound to learn something.
In particular, chapters 4, 5, and 6, contain a large collection of fairly
advanced tips. These are actually quite short, to the point that I was
frequently left wishing for more (which is not necessarily a bad thing), but
they’re usually just enough to get you started with the feature. Any
intermediate user is sure to not know many of them, and even advanced users
might benefit from a few. For instance, chapter 4 finally convinced me to try
Helm, and chapter 6 made me wonder why I never use Emacs to read log files.
Whether or not a few new ideas wrapped in a beautiful package is worth the price
being asked is entirely up to you (and, I suppose, up to how the price
translates in your currency). All I can say is that I’m happy for this book, and
I’m secretly wishing for a second volume.
show-paren-mode is a minor-mode that highlights the bracket at point (be it
round, square, or curly) as well as its corresponding open/close counterpart. I
find it a must-have for Elisp and Clojure programming. On the other hand, when
editing Ruby code, it also highlights whole block delimiters, like def, do,
if, and end, and all that must-haviness quickly turns into
in-your-faceviness.
The catch here is that show-paren-mode is a global minor-mode. So you can’t
just enable it locally in lisp-mode-hook, and if you try to do
(show-paren-mode -1) in ruby-mode-hook you’re going to disable it globally
every time you visit a ruby file.
Fortunately, show-paren-function checks the value of the variable
show-paren-mode, so we can pseudo-disable it by setting this variable locally.
With this, the mode will still be active in ruby-mode buffers, but it won’t
actually do anything.
Alternatively, you could reset show-paren-data-function to its original value
(also inside ruby-mode-hook). This will keep only the basic bracket
highlighting.
Emacs’ customizable variables (a.k.a., defcustom) are allowed to specify a
:type parameter for setting its custom-type. The customize interface uses this
information to produce a sophisticated menu for the user to customize that
variable. However, a large fraction of users use setq to directly edit custom
variables, and even some packages programmatically change the value of other
package’s custom variables. Ultimately, there are no guarantees that the value
in question matches the :type specified in the variable.
validate.el tries to address that by offering a small set of functions and
macros to validate that a value matches a :type schema, throwing an error when
it doesn’t. Most importantly, in case of error it provides very informative
messages about what part of the value failed to validate. So, instead of getting
some obscure wrong-type-argument error deep down in the code, you’ll get a
message like the following as soon as the variable is used:
Looking for ‘(repeat (choice string number))’ in ‘("aa" 90 la)’ failed because:
Looking for ‘(choice string number)’ in ‘la’ failed because:
all of the options failed
Looking for ‘string’ in ‘la’ failed because:
not a string
Looking for ‘number’ in ‘la’ failed because:
not a number
There are 3 main use-cases for this:
As an end-user of a package, you can use validate-setq instead of setq
for editing variables. This will ensure the configuration you provide matches
the schema specified by the developer, and thus prevents you from
misconfiguring stuff.
Also note that this works on any defcustom defined with a :type. That is,
it doesn’t matter if the package itself uses validate.el.
As a developer, when using a variable, use (validate-variable 'var-name).
This will be identical to just using var-name if the value passes
validation, but will immediately throw an error if it doesn’t.
Also as a developer, you can call (validate-mark-safe-local 'var-name)
which will create a safe-local predicate for the variable whenever the local
value satisfies its schema.
validate.el is available on GNU Elpa, so you can install it from the
package-menu or add it as a dependency in your package.
Here is what tried to build a TAG for multiple R packages. It enable
me to jump to a location where the function/variable is defined and
modify if I want to.
Useful variable and functions
ess-r-package-library-path
default path to find packages, should
be a list
ess-r-package-root-file
if the folder has DESCRIPTION file, then
the folder is a R package.
(ess-build-tags-for-directory DIR TAGFILE)
build tag on DIR to TARGET.
tags-table-list
List of file names of tags tables to search.
(visit-tags-table FILE &optional LOCAL)
Tell tags commands to use
tags table file.
;; new variable
(defvar ess-r-package-library-tags nil
"A TAG file for multiple R packages.")
(setq ess-r-package-library-path '("~/tmp/feather/R" "~/tmp/RPostgres/"))
(setq ess-r-package-library-tags "~/tmp/all_tags")
(dolist (pkg-path ess-r-package-library-path)
(let ((pkg-name (ess-r-package--find-package-name pkg-path)))
(unless (and pkg-name pkg-path
(file-exists-p (expand-file-name ess-r-package-root-file pkg-path)))
(error "Not a valid package. No '%s' found in `%s'." ess-r-package-root-file pkg-path))
(ess-build-tags-for-directory pkg-path ess-r-package-library-tags)
))
Note the workhorse is ess-build-tags-for-directory which does what
it means. The core of this function use find and etags program.
The find program will find files with extension .cpp, R, nw etc, and
then feed to (using pipe) to the etags program which generate a TAG
table. These two steps are demonstrated in the following snippet,
which is grabbed from the source code of
ess-build-tags-for-directory.
Note when they are used in Emacs, the tags-table-list variable is
appended with the path to the new TAG table. So that the user can use
xref-find-definitions (M-.) to jump (if the point is under a word) or
select which function/variable to jump to. The users then check the
function/variable definition, or modify it if it is necessary. Then
call xref-pop-marker-stack (M-,) to jump back.
As laptop touchpads seem to be steadily increasing in size, one unfortunate
consequence is that it becomes increasingly harder to avoid touching them by
accident while you type. Most systems have safeguards in place that disable the
touchpad as you’re typing, but they always seem to fall short for me when it
comes to Emacs. While in Emacs, my hands are permanently resting on the keyboard
(and over the touchpad), so even if I stop typing for several seconds I don’t
want the touchpad to reactivate.
There are ways to permanently deactivate the touchpad with a hotkey, but, so
far, the solution that best fits my use-style is to disable it only inside
Emacs.
(define-minor-modedisable-mouse-mode"A minor-mode that disables all mouse keybinds.":globalt:lighter" 🐭":keymap(make-sparse-keymap))(dolist(type'(mousedown-mousedrag-mousedouble-mousetriple-mouse))(dolist(prefix'(""C-M-S-M-S-C-M-C-S-C-M-S-));; Yes, I actually HAD to go up to 7 here.(dotimes(n7)(let((k(format"%s%s-%s"prefixtypen)))(define-keydisable-mouse-mode-map(vector(internk))#'ignore)))))
All we do here is define a minor-mode that binds all mouse-related keys to
ignore. This is a slight improvement over the code on this StackOverflow
answer. Of course, let’s not forget to enable this.
(disable-mouse-mode1)
Two relevant limitations:
This doesn’t distinguish between a touchpad an an actual mouse.
It is still possible for modes to define specific buttons that interact with
the mouse, but that’s not a huge problem because these buttons take a small
portion of the screen so it’s fairly difficult to touch them by accident.
Update 29 Jun 2016
By mere coincidence, it looks like Steve Purcell implemented something extremely
similar (comment below). Unlike me, he actually had the decency of wrapping the
minor-mode in a Melpa package, and I know some might prefer that over
lengthening their init file.
Countless build tools and shell scripts use ANSI escape codes to colorize their
output. This provides impressive improvements to readability when running from a
terminal that supports them, but tends to cause a catastrophic mess anywhere
else. Emacs’ compilation buffer is one such place. It doesn’t support ANSI
colors by default, but that’s very easy to fix.
Emacs already has a library for interpreting ANSI escape. All we need is to hook
it onto compilation-mode.
(require'ansi-color)(defunendless/colorize-compilation()"Colorize from `compilation-filter-start' to `point'."(let((inhibit-read-onlyt))(ansi-color-apply-on-regioncompilation-filter-start(point))))(add-hook'compilation-filter-hook#'endless/colorize-compilation)
Ivy is a completion method that's
similar to Ido, but with emphasis on simplicity and customizability.
New package names
Changes on MELPA
Due to multiple requests, in an attempt to simplify things a new
package ivy has been released on MELPA. The old package swiper,
which used to provide most of the ivy features, now only provides
swiper.el and depends on ivy. The third package counsel, which
provides most of the goodies using ivy hasn't been changed and is
still on MELPA. All three packages have the version 0.8.0 currently.
To reiterate the dependencies:
ivy depends on Emacs version larger than 24.1, preferably at least 24.3 (the most common one bundled currently with Linux distributions).
swiper depends on ivy and provides basically 3 commands: swiper, swiper-multi and swiper-all.
counsel depends on swiper and provides around 50 commands for all kinds of stuff. My favorites are counsel-M-x, counsel-git-grep, counsel-rhythmbox and counsel-grep-or-swiper.
Changes on GNU ELPA
On GNU ELPA, a single package
ivy-0.8.0 has replaced the previous stable version swiper-0.7.0.
This package provides all the files combined of the three separate
MELPA packages.
Release summary
The release consists of 282 commits over 5 months by 15 authors. The
detailed Changelog is available
here, thanks to the
ever useful Org mode export. The raw Org file is in
doc/Changelog.org
in the main repository.
The detailed documentation is available as an (ivy) Info node and
also in HTML form here. If anyone wants
to document something that's missing there, I'd appreciate the help:
simply edit
doc/ivy.org
and send me a PR.
Release highlights
Below, I'll highlight some of the new features.
Allow to compose collections
For example, to stack the top 10 elements of recentf on top of counsel-locate, use this code:
Here, (original-source) represents the async candidates of
counsel-locate. All extra sources are static - each function is called
once to generate a list of strings, which will be filtered later.
If you're not yet familiar with Ivy, you can get a quick reference
card by pressing C-h m during any completion session. It
will pop up an Org-mode buffer that describes most of the minibuffer
key bindings.
Additionally, C-o (hydra-ivy/body), which serves a quick
reference as well, received a small restructuring and a new
binding. Press D to go to the definition of this
hydra. This is useful to see what each key does, you might even want
to customize some of it.
Completion in region
From now on, ivy-mode will also set
completion-in-region-function. This means that functions like:
C-M-icomplete-symbol in many major modes,
TAB while in the M-: (eval-expression) minibuffer,
TAB in a shell buffer,
will use ivy for completion.
Many improvements to ivy-occur-mode
You can "permanently" save any completion session by pressing C-c
C-o (ivy-occur). This will generate a new buffer in
ivy-occur-mode with all your current candidates inserted there.
Clicking or pressing f on any of the candidates in that
buffer will result in the appropriate action being called with that
candidate.
The *ivy-occur ...* buffers can actually be customized per
collection type. Specifically for swiper, counsel-git-grep,
counsel-grep and counsel-ag, the customizations are already in
place that allow you to:
Edit the buffer with wgrep by pressing C-x C-q
(ivy-wgrep-change-to-wgrep-mode).
Refresh the buffer due to the original files being changed by pressing g (ivy-occur-revert-buffer).
The second feature is often useful to me when I want to somehow change
a symbol throughout the project. First I make a list of all
occurrences via e.g. swiper and ivy-occur. After I went through
some of the occurrences, I can press g to refresh the
search for the same symbol and see how many I still have left.
Yet another cool feature is to press c
(ivy-occur-toggle-calling) to toggle calling the action after each
line movement and cycle through candidates by holding either
j (ivy-occur-next-line) or k
(ivy-occur-previous-line).
This allows me to press M-o I to insert the current
candidate into the buffer. For instance, if I want to quote an Emacs
command, I can M-x (counsel-M-x), select the command I
want and press M-o I to insert it instead of calling it.
Virtual views in ivy-switch-buffer
Here, "virtual" buffer means something that's close to a buffer but
not an actual buffer. If you were using the setting
ivy-use-virtual-buffers, you'd have your bookmarks and recentf
items available to you as virtual buffers.
The new feature allows to select a whole window configuration with
many buffers inside nested horizontal and vertical splits from
ivy-switch-buffer.
To use it, set ivy-views, since it's nil by default. For instance,
here's what I have in my config:
For a more elaborate and detailed use, see
this post
by Manuel Uberti.
Magic slash in file name completion
From now on, if you want to enter a directory, simply select a
candidate which is a directory and press /. You can still
use the old binding C-j (ivy-alt-done), but /
is shorter and easier to get used to if you're switching from Ido.
Note that this does not prevent the use of old functionality like:
// to enter the root directory,
/ssh:RET to connect via TRAMP.
A better way to search with counsel-grep-or-swiper
If you've ever been annoyed with the long start-up time of swiper in
huge buffers, switch to this setting:
(global-set-key"\C-s"'counsel-grep-or-swiper)
This command will use swiper for small buffers, and counsel-grep for large buffers.
Something very similar to this command was highlighted in
this post
by Karl Voit.
Just to give you an idea of how fast counsel-grep is:
It has 0s start-up time, since it's async.
For a two million line file weighing 50MB produced by copying
org.el a few times, it takes 0.2s to find the 17,664 occurrences
of the word hook. It still takes 7.5s to search for org in that
file, simply because there are 500,000 candidates and it takes time
for Emacs to simply receive that input.
A list of all new commands
The new commands since 0.7.0 are: counsel-tmm, counsel-imenu,
counsel-decbinds, counsel-list-processes,
ivy-switch-buffer-other-window, counsel-git-stash,
counsel-git-log, counsel-pt, counsel-linux-app,
counsel-ace-link, counsel-esh-history, counsel-shell-history,
counsel-grep-or-swiper.
If any of those sound interesting, go ahead and try them out.
Are you a Chromebook user or thinking of becoming one? Are you a die-hard
Emacser who needs to see it run even in your browser for no good reason (no
judgement)? Either way, Emacs has you covered. Thanks to the efforts of Pete
Williamson (and friends), there is now an Emacs port for Chromebook and Chrome.
If that spikes your interest, you should read his post about it on Google Plus,
which provides installation instructions, a list of known issues, and some tips
on how to make the most of it. In short:
Launch the app and wait for it finish setting everything up. YMMV, but I had
to restart the app at this point.
Run emacs at the terminal you’re given (and celebrate!).
Again, his post contains a lot more information, including his email so you can
report bugs or file suggestions. Although I did run into a couple of small bumps
with the DevEnv app, the Emacs executable ran just fine and performed admirably.
Pete has been doing this as his 20% project — in which Google employees take
20% of their time to work on something else. If you follow the steps above, you
may notice that it uses Emacs 24.3, already two minor versions behind the latest
stable. That’s because 24.3 was the latest release back when he started this
endeavour.
If you’re wondering how hard it was, he talked about it on FOSDEM last year and
you can have a glance at the slides for this talk. It was harder than I would
have expected and clearly took some patience. Of course, all of the code is
publicly available under the webports project. It’s basically a shell script and
a patch file, so it’s worth a look if you’re curious.
R is a great language for R&D. It's fast to write prototypes, and has great
visualisation tools. One of constraints of R is it stores the data in
system memory. When the data becomes too big to fit in the memory, we
asked the user has to manually split the dataset and then aggregate
the output later. This process is inefficient and error prone for a
non-technical user.
I started an R development project to automate this split-aggregate
process. A viable solution is to store the whole data in PostgreSQL,
and let R to fetch one small chunk of the data at a time, do the
calculation, and then save the output to PostgreSQL. This solution
requires frequently data transferring between these two systems,
which could be a bottleneck in performance. So I did a comparison of
two R packages that interface R and PostgreSQL.
is a new package which provides similar functionality
to RPostgreSQL but rewrite using C++ and Rcpp. The development is
led by Kirill Müller.
Based on my testing, the RPostgres package is about 30% faster than
RPostgreSQL.
The testing set-up is quite simple: I write an R script to send data to
and get data out from a remote PostgreSQL database. It logs how long
each task takes to complete in R. To avoid other factors that can
affect the speed, it repeats this process 20 times and use the
minimal run-time as the final score. The dataset transferred between
R and PostgreSQL is a flat table with three columns and the number of
rows varies from ten thousand to one million.
The run-time in seconds are plotted against number for rows for each
package and operation.
Here is a summary of what I observed:
RPostgreSQL is slower than RPostgres. For getting data out, it's 75%
slower, which is massive! For writing, difference is closer, it's
about 20%. When combine both scores together, it is about 33% slower.
Particularly, it's slower to read than to write for RPostgreSQL
package, the ratio is about 1.5. While as it's quicker to read than
to write for RPostgres, the ratio is about 0.8. This is an interesting
observation.
Both package has a nice feature - the reading/writing time
linearly depends on the number of rows. This makes the time
estimation reliable. I would be confident to say that for 2
millions rows, it takes RPostgres package about 6 seconds to
read.
I don't why which part of implementation makes the RPostgres faster.
I guess its the usage of C++ and the magical Rcpp package.
Here is the script just in case you want to your own tests.
library(data.table)library(ggplot2)library(microbenchmark)library(RPostgreSQL)library(DBI)# config for PostgreSQL databasehost.name<-NULLdatabase.name<-NULLpostgres.user<-NULLpostgres.passwd<-NULLpostgres.port<-NULLtemporary.table.name<-NULL# config for testingnrows<-seq(10*1e3,1*1e6,length=10)repeats<-20# open PostgreSQL connectionpg.RPostgreSQL<-dbConnect(dbDriver("PostgreSQL"),host=host.name,dbname=database.name,user=postgres.user,password=postgres.passwd,port=postgres.port)pg.RPostgres<-dbConnect(RPostgres::Postgres(),host=host.name,dbname=database.name,user=postgres.user,password=postgres.passwd,port=postgres.port)ReadWriteWarpper<-function(pg.connection){# helper function write<-function()dbWriteTable(pg.connection,temporary.table.name,dt,overwrite=TRUE)read<-function()dbReadTable(pg.connection,temporary.table.name)var<-list()for(ninnrows){# create a datasetdt<-data.table(x=sample(LETTERS,n,T),# charactery=rnorm(n),# doublez=sample.int(n,replace=))# integer# read and write once first.write()read()# run and log run-timeres<-microbenchmark(write(),read(),times=repeats)# parse var[[as.character(n)]]<-data.table(num_row=n,operation=res$expr,time=res$time)}# aggregate and returnrbindlist(var)}# rundf0<-ReadWrite(pg.RPostgres);df1<-ReadWrite(pg.RPostgreSQL)df0$pacakge<-"RPostgres";df1$package<-"RPostgreSQL"df<-rbind(df0,df1)plot.df<-df[,min(time)/1e9,.(num_row,operation,package)]## generate plotplot.df[,operation:=gsub("\\(|\\)","",operation)]ggplot(plot.df,aes(x=num_row,y=V1,col=package))+geom_path()+geom_point()+facet_wrap(~operation)+theme_bw()+labs(x="Number of rows",y="Run time (sec)")
Admittedly, I’m a very late passenger in this boat — only after 4 years of
using Emacs did I decide to try a project manager. Nowadays I can’t even
remember my daily workflow without Projectile. This package mostly stays out of
your way, and provides a series of useful commands for dealing with a project
(which are aware of a lot of languages out-of-the-box). As usual, you can find
details in the readme, and we’ll jump straight into useful configurations.
(setqprojectile-keymap-prefix(kbd"C-x p"))
This places all Projectile keybinds under C-x p and requires no explanation.
Mnemonickeymapsarethebest. Most used
commands are C-x p f to find a file, and C-x p p to switch-project and
bring up the commander menu (see below)
By default, Projectile brings up the file-finder when you switch project with
C-x p p. That’s a reasonable default, but I find a lot of times I’m also looking for
magit-status or a shell buffer. Using projectile-commander means I have to
hit an extra key, but it always gets me where I want.
Furthermore, the menu of alternatives presented by projectile-commander is
very customizable, so we can add anything we want in there.
(require'projectile)(def-projectile-commander-method?s"Open a *shell* buffer for the project.";; This requires a snapshot version of Projectile.(projectile-run-shell))(def-projectile-commander-method?c"Run `compile' in the project."(projectile-compile-projectnil))
The first of those brings up a shell buffer in the project root and the second
runs M-xcompile. Both are super duper convenient for quickly running builds
or custom commands, and which one you use is entirely up to situational
preference.
(def-projectile-commander-method?\C-?"Go back to project selection."(projectile-switch-project))
The s key would normally be bound to project-switching. Since we’ve changed that
above, it’s useful to make Backspace take that role. This makes sense to me.
It’s like I’m “backing out” of the commander menu.
(def-projectile-commander-method?d"Open project root in dired."(projectile-dired))
By default d would be bound to projectile-find-dir, but that’s something I
never use. projectile-dired takes you to the root directory instead, which I
find more useful.
These two are more situational, but I’ve found I use them a lot. Whenever I sit
down to work, there’s a good chance I’m either going to start a REPL (j) or
fetch git remotes (F).
And last but not nearly least.
(projectile-global-mode)
Update 14 Apr 2016
Used Projectile’s built-in shell and compile commands.
As Android phones rise in power, bluetooth keyboards become cheaper, and your
addiction to Emacs grows, it’s only natural that you start thinking of combining
the three. Fortunately for you, it’s not as hard as it used to be. In fact, it’s
perfectly possible to reproduce (most of) your desktop config, if you know how
to get past a few obstacles.
I was going to precede these instructions with a short tale, but I got a little
carried a way and it grew a little too long. Because of that, I’ll give you
the instructions first and let you decide whether to read my self-indulgent
delusions below.
Copy your desktop’s init.el file to your phone (read below for a way to
keep it synchronized).
Run emacs (celebrate a little!).
If Emacs spits a “void-function …” error at you, comment out the relevant
part of your init file and try again. See below for the explanation.
Celebrate a lot!
I was surprised at how well this worked in the end. Even Beacon, which is a
largely graphical package, worked out of the box (if a bit laggy).
Update 11 Apr 2016
Fredrik Fornwall has patched the Emacs package on Termux to no longer require
the tmp dir workarounds. So now it’s easy as pie to get Emacs on your Android.
A tale of two thumbs
Last night, in a fit of boredom, far away from my laptop and from any physical
keyboard, I did what any reasonable person would have done: decided to install
Emacs on my phone. My last attempt at crossing this bridge was over a year ago,
and resulted in nothing but absolute failure. Still… Something felt different
this time. Something felt right.
My first stop was an obvious one. There’s an Emacs app on the play store
claiming to do exactly what I needed. On my previous attempts, this app had
given inconsistent results, crashing more often than running. This time,
however, it wouldn’t even install. The Play Store just greeted me with a
“Failure to install” error instead.
Not all was lost, though. A generous comment (with an less-than-generous rating)
points me in a new direction. Termux, my beacon of hope, is a robust app that
claims to offer “powerful terminal emulation with an extensive Linux package
collection”.
After installing it, I am indeed greeted with a terminal emulator, but I’m not
impressed. I’ve used terminal apps before. Tentatively, I run the commands (as
instructed by the man-from-the-review).
apt update
apt install emacs
Surprisingly, it played out even better than expected. Installing Emacs would
have been fine, but installing version 25 is exceptional! My init file is sure
to be incompatible with Emacs 24, but now that we have 25… Dare I even hope?
Could I reproduce my entire setup on this minute device?
Before proceeding, I hopped back over to the play store and installed the
Hacker’s keyboard, to get access to modifier keys. Then, starting emacs from the
terminal worked as expected, if a bit slowly, but my streak of good fortune was
about to end.
The first command I issued (M-xlist-packages) failed with a cryptic message
about nonexistent file in /tmp/asdlij198h1 (/this should not happen to you
thanks to Fredrik’s patch). The package menu displayed, but refused to refresh.
So I C-z out of Emacs, and try mkdir /tmp/.
Permission denied
Ok, I should have expected that. sudo doesn’t work either, so I issue fg to
go back to Emacs and try a different approach.
(setqtemporary-file-directory"~/tmp/")
Now list-packages again, and it works! I try to install company from the
gnu repository, and it works too! In fact, everything works. I run M-xglobal-company-mode and it’s all there, popup menu and everything.
At this point I’m probably way more satisfied with myself than I should for such
a senseless pursuit, but I’m too high to realize.
Next step is to port over my real init file. Fortunately, I keep it sync’d via
Dropbox, so it’s easy to make available on the phone. Just open the Dropbox app,
find my init.el, and mark it to be “Available offline”. Same thing for my
init.org. Dropbox saves the files somewhere deep inside its own data
directory, and automatically syncs them up when you make local changes (though
you have to manually ask it to pull down remote changes).
Back on Termux, I send Emacs to the background again with C-z, and
cd ~/.emacs.d
ln -s /sdcard/Android/data/com.dropbox.android/files/SOME_GIBBERISH/.emacs.d/init.el
ln -s /sdcard/Android/data/com.dropbox.android/files/SOME_GIBBERISH/.emacs.d/init.org
Actually finding the file took a bit of trial and error. After getting to the
files directory I had to tab-complete my way through several gibberish
directory names before finding the right one.
fg and we’re back in Emacs.
The next obstacle are the packages. I know my init file is not going to load
before I install the necessary packages. Sadly, I never had the foresight
of rewriting my configuration in something like use-package, but there’s
something almost as good. Like I’ve mentioned before, Emacs 25
keeps track of user-selected packages, so I was able to go into my
custom-set-variables and find a sizeable list under
package-selected-packages. With no further ceremony, I just evaluate the whole
thing with C-M-x, and issue M-xpackage-install-selected-packages.
This takes a while…
This is a good moment to find out if Termux works well in the background. Switch
to the Youtube app, watch a couple of videos, switch back… And it worked!
This is it now — the moment of truth. I’m still feeling high and mighty after
all this success, but I have to respect the odds. My init file has 4 thousand
lines of Elisp and involves over 100 packages. That’s a whole lot of
could-go-wrong potential.
I take a deep breath and mentally try to lower my expectations. I C-x C-c back
to the terminal, and then carefully type emacs followed by ↵, only to plunge
head-first into an error.
Something about the tmp directory again. Haven’t I solved this one? A bit more
digging reveals it’s being triggered by (server-start), because
server-socket-dir is bound to /tmp/emacs1000. The comments above the
variable explain why.
;; We do not use `temporary-file-directory' here,;; because emacsclient does not read the init file.(defvarserver-socket-dir...)
That’s understandable, but it doesn’t help me. Impatiently, I just wrap the call
to server-start in ignore-errors and try again…
Another error. This time it’s complaining that set-fringe-mode is a void
function. That’s a lot easier to understand. The Emacs binary provided in Termux
is compiled for terminal (why wouldn’t it be?), so many graphical functions
aren’t defined at all. It’s also easy to solve. Just add a conditional around
the function call.
A third time. C-x C-cemacs↵… And it works! No more errors. Not even a
warning! My beloved darktooth theme is unrecognizable, but all the other
indicative signs of my Emacs setup are there. Smart-mode-line discretely smiles
at me from the bottom of the screen. Beacon eagerly blinks at me as soon as I
start scrolling. And the unmistakable coziness of my personal keybinds reach me
even through the limited interface of a touchscreen-keyboard.
As we get to this point, a lesser person might feel an anticlimax — a hint of
a “what for?”, perhaps. But not me. I just feel safe and reassured, knowing
that, wherever I am and whenever I need it, Emacs will always be sitting in my
pocket.
Now I just need to get myself one of those bluetooth keyboards I mentioned…
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
We’re starting at 6:30pm, as always we will have pizza and beer!
Aldric website will be giving a talk about creating sqlup, creating minor modes and distributing them.
This talk will take you through the creation of a minor mode. It will
cover the fundamentals of the creation of a minor mode with all its
options and examples of how to use them, then take you through how
I’ve approached the creation of sqlup.el and some of the choices I’ve
made.
I've been getting more and more organized in tracking my tasks and
time with Org-mode. Still using the usual suspects, of course: GTD and
Pomodoro, I'm just getting more diligent with them than in the
previous years.
So today I wanted to prettify the good old
org-agenda-view-mode-dispatch, which is bound to v in
org-agenda-mode. Currently, it's just a boring static message and
read-char combination. Why not do it with a
hydra instead?
Here's the current full code, that uses the newly extended doc syntax:
And here's how it looks like in action, I simply pressed v while in the agenda:
Since many functions that org-agenda-view-mode-dispatch calls are
toggles, it makes sense for hydra-org-agenda-view to display the
status of these toggles.
And it's actually convenient to toggle a whole lot of things at once,
and the default red hydra keys really come in handy here.
Quick explanation of the syntax
Each head of a hydra looks like:
(key cmd &optional doc &rest plist)
The fairly new bit that I'm using here is the ability to use a sexp
instead of a plain string in the doc part. This sexp will be
evaluated each time the doc is re-displayed. This means that it can
represent a changing variable, for instance the state of a minor mode
or a variable.
And here's the best part: the doc of each head can be quoted in the
hydra's docstring by using the corresponding key, e.g. ?g?. This
allows to have very complex docstrings while keeping them easily
aligned in a tabular format.
Here is only the hydra's docstring, copied from the above code:
One of the things I like most in CIDER is how evaluation results are displayed
by inline overlays. And yet, for some reason, it’s taken me almost a year to
transfer that to Elisp.
It’s the tiniest of changes — you’re just taking something that would be
displayed in the minibuffer and moving it up a dozen or so lines — but the
difference is palpable. By displaying the result inline, you display it where
the user is looking, instead of forcing them to shift focus. The quick-feedback
loop we all love in our lisps becomes even faster (something I would’ve thought
impossible one year ago).
Assuming you already have CIDER installed, porting this feature to Elisp is
almost trivial. We just define a small wrapper around the function that creates
the overlay, and then advise the relevant elisp commands to call it.
(autoload'cider--make-result-overlay"cider-overlays")(defunendless/eval-overlay(valuepoint)(cider--make-result-overlay(format"%S"value):wherepoint:duration'command);; Preserve the return value.value)(advice-add'eval-region:around(lambda(fbegend&restr)(endless/eval-overlay(applyfbegendr)end)))(advice-add'eval-last-sexp:filter-return(lambda(r)(endless/eval-overlayr(point))))(advice-add'eval-defun:filter-return(lambda(r)(endless/eval-overlayr(save-excursion(end-of-defun)(point)))))
If I like this enough, I might implement it more properly and propose its
addition to Emacs core. For now the advices are more than enough.
If you don’t want to install CIDER, you can just copy that function to your
configs (you’ll also have to copy the functions above it in the same file, and
the when-let definition from cider-compat.el).
Have you ever stopped to think about why isearch leaves point at the end of the
match? It does make some intuitive sense to leave you after the characters you
have just typed, but that doesn’t make it the most practical behaviour.
This is something I’d never even given half a thought in the past, but it’s one
of the first things a friend questioned me as he was diving into Emacs (yes, the
same one as last week). Why not leave the cursor at the start of the match,
instead of the end?
It’s great when someone’s fresh perspective shakes new ideas into my
gradually-calcifying init file, so I immediately started looking for a way to do
this. Fortunately, Google was quick to find me an answer, in the dotfiles of one
Sylvain Rousseau. I should go through the entire file later to look for more
snippets, but for the current purpose, the following is all we need.
(add-hook'isearch-mode-end-hook#'endless/goto-match-beginning)(defunendless/goto-match-beginning()"Go to the start of current isearch match.
Use in `isearch-mode-end-hook'."(when(andisearch-forward(number-or-marker-pisearch-other-end)(notmark-active)(notisearch-mode-end-hook-quit))(goto-charisearch-other-end)))
After using this for a few days, I’ve already ran into a couple of cases where I
would have preferred the original behaviour. Still, the net balance has been
positive so far so it’s probably going to stay.
Update 23 Mar 2016
One alternative to the approach above is cutejumper's suggestion below, straight
from the Emacs wiki. This will only leave point at the start of search if you
exit the search with C-↵ instead of ↵.
(define-keyisearch-mode-map[(controlreturn)]#'isearch-exit-other-end)(defunisearch-exit-other-end()"Exit isearch, at the opposite end of the string."(interactive)(isearch-exit)(goto-charisearch-other-end))
I'd like to highlight the new command counsel-linux-app that I
recently added to the counsel
package:
This command looks through your /usr/share/applications/*.desktop
and offers to launch them, just like the launcher that you normally
get when you press the super key. My current system is Ubuntu, and the
launcher here is really horrible, so I'm glad to have a much faster
and reliable replacement.
Here's another screenshot, I open the keyboard settings quite often:
I’ve said before that clj-refactor is a magical package, and you wouldn’t catch
me bad-mouthing it in a million release cycles, but it’s impossible to please
everybody.
One consistent annoyance I have with it, is that I always have to type Tab
twice after invoking cljr-add-require-to-ns. That’s because I only ever use
requires of the form [x.y :as y], so I always have to skip the first two
“modes” offered by the snippet.
As I was teaching a friend about this command last week, he immediately asked me
if you couldn’t just write some Elisp to do that for you. Of course, the answer
is yes. And that’s exactly what I did — once I was done feeling embarrassed
that I hadn’t thought of that before.
Note that this uses the new advice interface, which relies on Emacs 24.5. While
this API in its entirety can be quite intimidating, you don’t need to learn it
all in one go. You can go a long way (and make your life a lot easier) by just
writing yourself a few :before and :after advices.
Just remember to be responsible. Advices let you run code in places where
packages don’t expect code to be run. So don’t be surprised if something else
breaks.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
This month we are doing lightning talks! The theme is how you use emacs in your day-to-day. You are welcome to give another lightning talk if you feel so inspired.
You can give a talk that is no more than 5 minutes in length.
(Contact us)[mailto:admin@emacsnyc.org] if you’d like to give a talk!
As usual, we’ll be starting at 6:30 with pizza and beer!
If you would like to give a lightning talk please feel free to come on up and speak. We will have everything set up for you when you get here.
If you would like to speak then or on any other occasion, take a look at this guide.
CIDER 0.11.0 has been out for less than week and already the snapshots are
getting new features. This one comes from a gentleman called Chris Perkins. It
provides an easy way to automatically skip some breakpoints during evaluation,
and it even comes with 300 brand new lines of tests.
The feature is very straightforward, and is already available on the snapshots
of 0.12.0. You’ve always been able to debug functions by adding a #dbg in
before it and then reevaluating its definition.
However, if you’re trying to debug a problem that only happens somewhere around
i = 1000, then you’re going to have to step through the body of that for one
thousand times before getting to where you need.
This new feature lets you specify a condition, and the debugger will
automatically skip all breakpoints as long as that condition is false. You
specify it via the :break/when metadata. In the above example, that would be
something like this.
Thus, everything inside the (some ...) form would be skipped as long as i ≤ 1000.
The debugger’s inner workings are not what I’d call simple, so it makes me happy
to know I’ve made the code legible enough for people to understand without my
help. Of course, it certainly helped that Chris is no average fellow. Did I
mention he even wrote a DSL for the new tests?
As you may remember, one of the commands I like the most from the clj-refactor
package are the ones that thread and unwind Clojure code forms for you. Now that
Emacs is also getting built-in threading macros, I figured the best way to give
them a fair chance in life is to also make them pretty convenient to use.
The point here is that you can place point before a paren, invoke a command, and
a regular code form gets transformed into a threading macro, or vice-versa. See
the linked post for what that means. Instead of writing whole new commands for
that, I had a fun time just hacking clj-refactor commands to work on
thread-first/last.
It should go without saying, you need clj-refactor installed for this to work.
(define-keyemacs-lisp-mode-map"\C-ctf"#'endless/elisp-thread-first)(define-keyemacs-lisp-mode-map"\C-ctl"#'endless/elisp-thread-last)(define-keyemacs-lisp-mode-map"\C-ctu"#'endless/elisp-unwind)(define-keyemacs-lisp-mode-map"\C-cta"#'endless/elisp-unwind-all)(defunendless/elisp-thread-last()"Turn the form at point into a `thread-last' form."(interactive)(cljr-thread-last-allnil)(save-excursion(when(search-backward"->>"nil'noerror)(replace-match"thread-last"))))(defunendless/elisp-thread-first()"Turn the form at point into a `thread-first' form."(interactive)(cljr-thread-first-allnil)(save-excursion(when(search-backward"->"nil'noerror)(replace-match"thread-first"))))(defunendless/elisp-unwind()"Unwind thread at point or above point by one level.
Return nil if there are no more levels to unwind."(interactive)(let((p(point)));; Find a thread above.(when(save-excursion(forward-sexp1)(and(search-backward-regexp"\\_<thread-\\(first\\|last\\)\\_>"nil'noerror);; Ensure that it contains the original point.(save-match-data(forward-char-1)(forward-sexp1)(>(point)p))))(replace-match(if(string=(match-string1)"first")"->""->>"))(let((thread-beginnig(match-beginning0)))(prog1(cljr-unwind)(save-excursion(goto-charthread-beginnig)(when(looking-at"\\_<->>?\\_>")(replace-match(if(string=(match-string0)"->")"thread-first""thread-last")))))))))(defunendless/elisp-unwind-all()"Fully unwind thread at point or above point."(interactive)(while(endless/elisp-unwind)))
Maybe just writing it from scratch would have made for shorter code (it would
certainly be more robust). But such is life. I try not to dwell too much on
quick hacks.
Here's a code snippet I've found long ago on the internet (the source
seems to be no longer accessible), that has proven valuable time and
time again:
;;;###autoload(defunora-dired-rsync(dest)(interactive(list(expand-file-name(read-file-name"Rsync to:"(dired-dwim-target-directory)))));; store all selected files into "files" list(let((files(dired-get-marked-filesnilcurrent-prefix-arg));; the rsync command(tmtxt/rsync-command"rsync -arvz --progress "));; add all selected file names as arguments;; to the rsync command(dolist(filefiles)(setqtmtxt/rsync-command(concattmtxt/rsync-command(shell-quote-argumentfile)" ")));; append the destination(setqtmtxt/rsync-command(concattmtxt/rsync-command(shell-quote-argumentdest)));; run the async shell command(async-shell-commandtmtxt/rsync-command"*rsync*");; finally, switch to that window(other-window1)))(define-keydired-mode-map"Y"'ora-dired-rsync)
Lets you copy huge files and directories without Emacs freezing up and
with convenient progress bar updates. That is all.
Thanks to tmtxt, the mysterious hacker-person from whom the snippet
likely originated . Good luck with getting your blog back up.
One of my personal favorite new additions to Emacs 25 is, in fact, completely
invisible to most users. The new macros if-let and when-let, although simple
in purpose, are a delight to use and are frequently finding their way into my
code. The other two additions, thread-first and thread-last, are a bit more
specific, and take a bit getting-used-to if you’ve never seen them before.
The two *-let macros are easier to explain by just expanding them. All they do
is summarize a very common situation for lisp programmers. Instead of writing
this:
(let((x(some-func)))(ifxdo-thendo-elsedo-else-2))
you can write this:
(if-let((x(some-func)))do-thendo-elsedo-else-2)
Seeing this example it shouldn’t be hard to understand when-let as well.
As you can see, it’s not exactly a revolution — you barely even save any
typing. But that extra level of nesting you save, along with an added bit of
clarity, for some reason makes me smile every time I get to use them.
The threading macros are something you see all the time in Clojure (under a
shorter name) but are probably not going to be as common in Emacs-Lisp — your
average Elisp code is far less functional than Clojure code. Still, they’re good
to know as they can be a huge help in some situations. Explaining again by
example:
;; First(thread-firstx(concaty)(formatz));; Expands to this:(format(concatxy)z);; Last(thread-lastx(concaty)(formatz));; Expands to this:(formatz(concatyx))
If the examples above don’t look too useful to you, that’s because they’re not.
thread-first and thread-last are relatively short names by Elisp standards,
but they’re still long enough that you’re almost always going to be typing
more, instead of less, when you use these macros. Instead of saving space,
their value lies in improving readability of some rather extreme scenarios
(which are all too common in Elisp). For instance, which one of the forms below
would you find easier to read?
;; This?(thread-lastsome-string(replace-regexp-in-string"regexp-1""replace-1")(replace-regexp-in-string"regexp-2""replace-2")(replace-regexp-in-string"regexp-3""replace-3")(replace-regexp-in-string"regexp-4""replace-4")(replace-regexp-in-string"regexp-5""replace-5"));; Or this?(replace-regexp-in-string"regexp-5""replace-5"(replace-regexp-in-string"regexp-4""replace-4"(replace-regexp-in-string"regexp-3""replace-3"(replace-regexp-in-string"regexp-2""replace-2"(replace-regexp-in-string"regexp-1""replace-1"some-string)))))
Finally, for those of you who use my speed-of-thought-lisp package, it already
has abbrevs for these macros under il, wl, tf, and tl.
Another library by the productive Nicolas Petton. map.el is a cousin to
seq.el (remember?), but instead of manipulating plain sequences, it
manipulates map-like collections (also known as dictionaries).
The range of functions should include everything you expect, from map-values
and map-keys to get a list of the values and keys (duh), to general getters
and setters like map-elt and map-put. Inside it, you’ll even find some
fancier stuff like map-let, map-values-apply, or map-nested-elt. For those
who know what that means, it also comes with a pcase macro!
Note that this library does not introduce a new data structure. It simply
provides a consistent and unified API for dealing with any map-like structure
you need. This applies to alists, hash-maps, and even vectors (it treats the
vector as a dictionary whose keys are all integers).
Many experienced Emacs users are aware of ffap command:
Find FILENAME, guessing a default from text around point.
If ffap-url-regexp is not nil, the FILENAME may also be an URL.
It's a great way to open an link, if you plan things in advance. But
for me it was usually C-x C-f (annoyed grunt) C-g M-x
ffap RET.
Now, thanks to counsel-find-file,
it's C-x C-f (anticipated annoyance, followed by a sigh of
relief) M-n.
With Ivy completion, M-n calls ivy-next-history-element,
which tries to
predict the history element
in case you've reached history's edge. The prediction usually simply
inserts thing-at-point into the minibuffer. My favorite applications
of this are:
C-s M-n - swiper thing-at-point, to get the occurrences
of the current symbol in the current file.
C-c j M-n - counsel-git-grep thing-at-point, to get the
mentions within the current project.
C-c g M-n - counsel-git thing-at-point to open a file
to which the current symbol links.
One thing I've recently added is the \_<...\_> wrapper for when
major-mode derives from prog-mode. Since the \_< regex matches
the symbol start, and \_> matches the symbol end, there's no chance
of getting partial matches. You can call undo or press
M-n again in case the symbol bounds aren't useful.
Finally, C-x C-f M-n can be used to open URLs. Recently,
I've added functionality to counsel-find-file that allows me to also
visit Github issues by simply pointing at the plain issue number,
e.g. #123 in either a version-controlled file or in a Magit buffer.
The command will query:
$ git remote get-url origin
and fill in all the details. So I no longer bother with
bug-reference-url-format and bug-reference-mode - now it's all
automatic.
It's also possible to make it work for places other than Github, for
instance this code (already included in
counsel) makes it work for the Emacs Git
repository:
(defuncounsel-emacs-url-p()"Return a Debbugs issue URL at point."(when(and(looking-at"#[0-9]+")(or(eq(vc-backend(buffer-file-name))'Git)(memqmajor-mode'(magit-commit-mode))))(let((url(match-string-no-properties0))(origin(shell-command-to-string"git remote get-url origin")))(when(string-match"git.sv.gnu.org:/srv/git/emacs.git"origin)(format"http://debbugs.gnu.org/cgi/bugreport.cgi?bug=%s"(substringurl1))))))(add-to-list'ivy-ffap-url-functions'counsel-emacs-url-p)
In the upcoming version, EWW is getting a number of small improvements. This web
browser, written by Lars Ingebrigtsen, is something of a new kid on the block,
as it just came to life at the very end of the Emacs 24 cycle. Although it’s
hard, if not impossible, to reliably render HTML inside an editor that’s 100%
line-based, EWW tends to find a reasonable compromise and deserves at least a
short post to cherish new features.
HTML can now be rendered using variable-width fonts.
A new command F (eww-toggle-fonts) can be used to toggle whether to use
variable-pitch fonts or not.
It goes without saying that this is very good, monospace fonts are not the most
appropriate for reading webpages. It might sound like something simple, but
this was probably the hardest feature to implement. Filling paragraphs in
variable-width fonts is not something Emacs does by default.
A new command R (eww-readable) will try do identify the main textual parts
of a web page and display only that, leaving menus and the like off the page.
I just tested this on the round quotes post and it worked well—the buffer
correctly hid everything but the post content. Unfortunately, it also hid images
that were inside the post.
https pages with valid certificates have headers marked in green, while invalid
certificates are marked in red.
Awareness about security is always a good thing.
You can now use several eww buffers in parallel by renaming eww buffers you
want to keep separate.
Partial state of the eww buffers (the URIs and the titles of the pages visited)
is now preserved in the desktop file.
The new S command will list all eww buffers, and allow managing them.
All of these contribute to giving EWW a more complete browser experience. While
we don’t have actual tabs yet, you can have multiple open pages, manage them
all with S, and even “remember open tabs” by enabling desktop-mode.
Since last week’s post was about Unicode characters, it makes sense to continue
that trend today. This feature might go unnoticed by a lot of people who live in
an ASCII world, but it will probably jump out at everyone else at one point or
another. The name, if a bit odd, is “character-folding search”.
In Emacs 25, Isearch can find a wide range of Unicode characters (like á, ⓐ, or
𝒶) when you search for ASCII characters (a in this example). To enable this
feature, set the variable search-default-mode to char-fold-to-regexp.
(setqsearch-default-mode#'char-fold-to-regexp)
This first came up due to a new feature of a recent version of Texinfo, where
some markup styles are exported in “round quotes” when generating the info
manual. This is a nice improvement in readability, but when you’re trying to
search for something with C-s, things can get a little difficult if your
keyboard can’t type some of the characters.
With char-folding, when you hit C-s and search for ", Emacs will also search
for a variety of double quotes, from the aforementioned “” to many others like
«» and ❝❞. The same goes for the single quote, and for pretty much any other
ASCII character.
You could always type any Unicode character by name with C-x 8 RET, and many
of them even have their own shortcuts under C-x 8, but not having to type
them can be a significant convenience when it comes up. As any Brazilian, I am a
daily user of diacritical marks (ó, ã, ê, and the likes), and even though my
keyboard can type these characters, I still enjoy the simplicity of not having
to.
Finally, you can extend this feature to the query-replace command, which is
also as easy as setting a variable.
Don’t be fooled by the apparent simplicity of this feature. Its implementation
has been the most controversial addition to the upcoming Emacs release — to a
comical degree. This post, however, is not about arguments or implementation,
it’s about Emacs 25. And this little nugget is all set for the next release.
The feature is simple, next time you’re reading a docstring (maybe via C-h f
or C-h v), instead of looking like this
it’ll look like this
It’s a simple, but noticeable, improvement for user experience.
If you don’t like the change, it’s configurable via the new option:
text-quoting-style.
Note that these quotes are introduced directly in the *Help* buffer. The
function that does this conversion is substitute-command-keys, so you can use
it if you’d like the same effect for other purposes. You should still use
`grave-and-straight' quotes when writing docstrings for your defun.
On the other hand, if you need to write a straight quote that shouldn’t be
converted to a round quote when it’s printed in the *Help* buffer, you’ll need
to write \\= before it.
This release consists of 77 commits done over the course of the last 7
months by me and many
contributors. Similarly
to the 0.3.0 release, the release notes are in
Changelog.org.
You can read them either at
github or
inside Emacs.
Big thanks to all contributors.
Highlights
A lot of new code is just straight upgrades, you don't need to do
anything extra to use them. Below, I'll describe the other part of the
new code, which is new commands and custom vars.
avy-goto-char-timer
This command now allows as many characters as you like, which makes it
similar to a isearch + avy-isearch combination. As you type, you
get an isearch-like highlight, then after a short delay you
automatically get the avy selection.
Switch the action midway from goto to kill/mark/copy
This is similar to the cool feature of
ace-window that
allows you to switch the action after you get the avy prompt.
For example, suppose you have:
(global-set-key(kbd"M-t")'avy-goto-word-1)
Here's what you can do now to a word that starts with a "w" and is
select-able with "a":
To jump there: M-t w a.
To copy the word instead of jumping to it: M-t w na
To mark the word after jumping to it: M-t w ma.
To kill the word after jumping to it: M-t w xa.
You can customize avy-dispatch-alist to modify these actions, and
also ensure that it plays nicely with your avy-keys, if you
customized them. By default, it works fine, since avy-keys is '(?a
?s ?d ?f ?g ?h ?j ?k ?l) and the keys on avy-dispatch-alist are
'(?x ?m ?n).
avy-pop-mark
This command reverses avy-push-mark which most of avy commands call.
It has its own history and works across multiple windows and frames.
I'm using it currently as an upgrade to my old (set-mark-command 4)
lambda:
(global-set-key(kbd"M-p")'avy-pop-mark)
Here's a line to make avy-pop-mark work also for swiper:
Isearch is one of Emacs’ most useful (and probably most used) features. Getting
in the habit of quickly hitting C-s followed by 2–4 letters will forever
change the way you navigate buffers, and adding it to your repertoire is a
tremendous productivity improvement. What, then, could we possibly improve on
such a phenomenal command?
One of the search actions I find myself doing a lot is to search forward for
other occurrences of the symbol at point. You can do that in Isearch by typing
C-w a few times (which copies the next word at point into the prompt) until
you’ve got the entire symbol, and then you type M-s _ to toggle symbol-search.
However, this is a lot of keys, and it only works if the cursor is at the start
of the symbol. Fortunately, it’s very easy to improve this because the action we
want is already implemented as the isearch-forward-symbol-at-point command.
(defunendless/isearch-symbol-with-prefix(p)"Like isearch, unless prefix argument is provided.
With a prefix argument P, isearch for the symbol at point."(interactive"P")(let((current-prefix-argnil))(call-interactively(ifp#'isearch-forward-symbol-at-point#'isearch-forward))))(global-set-key[remapisearch-forward]#'endless/isearch-symbol-with-prefix)
The result here is that C-s starts Isearch as usual, but if I type C-u C-s
instead it’s going to search for other occurrences of the symbol at point—a
huge improvement over the 6+ keys of our previous option.
Normally, calling Isearch with a prefix would start it in regexp-mode. That’s
not something I’ve ever used, but you should be aware of it before you override
it.
In an
earlier post, I
described how I've been managing Rhythmbox from Emacs. I've bound the
entry point to C-S-o:
(global-set-key(kbd"C-S-o")'counsel-rhythmbox)
Obviously, this entry point won't work while outside Emacs. Today,
I'll describe how I've made it work everywhere. Everywhere on Ubuntu
14.04, that is, although a similar approach should work for other
distributions.
Using emacsclient is essential to avoiding the extra startup time:
even a startup time of one second feels sluggish when all I need is to
open a menu with a song playlist.
Step 2: Install the relevant X window tool
Initially, I only wrote a call to emacsclient, which resulted in the
Emacs window gaining focus in the end. Then I thought it would be
nice to give the focus back the original window after the end of
selection, and raise it as well.
I wanted to do something with wmctrl, but I found that
xdotool
can do what I want in a simple way.
Here, (x-focus-frame nil) will raise the Emacs window and give the
keyboard input focus. emacsclient will return as soon as I select
something or press C-g. At that point the keyboard focus
will be returned to whatever window had it when the script was
invoked.
By the way, here's a cool configuration that automatically makes a
file executable if it starts with #!.
Open this (possibly using gnome-control-center instead, if applicable):
unity-control-center keyboard
And add a new shortcut in Shortcuts/Custom Shortcuts. I've bound
that one to C-S-o as well.
The final result
It's pretty convenient: as I'm scrolling something I'm reading in
Firefox with j (via
Firemacs),
I can seamlessly press C-S-o moo RET to play "Sisters of
the Moon", and continue scrolling the web page with j.
What's more, Emacs has very nice support for input methods with
C-\ (toggle-input-method), so I can also quickly select
Ukrainian-titled songs, while still keeping shortcuts like
C-n and C-m (without having to switch the
input method back).
The whole experience is similar to gnome-do/synapse, which I was
using a few years back, except better because now it's in Emacs.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
This month we are doing lightning talks! The theme is how you use emacs in your day-to-day. You are welcome to give another lightning talk if you feel so inspired.
You can give a talk that is no more than 5 minutes in length.
(Contact us)[mailto:admin@emacsnyc.org] if you’d like to give a talk!
As usual, we’ll be starting at 6:30 with pizza and beer!
If you would like to give a lightning talk please feel free to come on up and speak. We will have everything set up for you when you get here.
If you would like to speak then or on any other occasion, take a look at this guide.
My Emacs auto-correct is one of the oldest posts on this blog, and I still see
it pop up here and there on occasion. Last week, Norman Ramsey asked about an
improvement to that command and I figured it’s worth an update post.
As of today, the code in the post will search backwards for a typo, instead of
just correcting the current word. So you can invoke it without trouble even if
the typo happened several words ago.
Recently, I wrote some code to add better highlighting for
Ivy's fuzzy matcher. Here's a
quick step-by-step to get an equivalent of
flx-ido-mode working with Ivy.
Step 1: install the packages
M-xpackage-install - counsel and flx (MELPA should be configured).
Step 2: configure ivy-re-builders-alist
Here's the default setting:
(setqivy-re-builders-alist'((t.ivy--regex-plus)))
The default matcher will use a .* regex wild card in place of each
single space in the input. If you want to use the fuzzy matcher,
which instead uses a .* regex wild card between each input letter,
write this in your config:
The t key is used for all fall-through cases, otherwise the key is
the command or collection name.
The fuzzy matcher often results in substantially more matching
candidates than the regular one for similar input. That's why some
kind of sorting is important to bring the more relevant matching
candidates to the start of the list. Luckily, that's already been
figured out in flx, so to have it working just make sure that the
flx package is installed.
The ivy-initial-inputs-alist variable is pretty useful in
conjunction with the default matcher. It's usually used to insert ^
into the input area for certain commands.
If you're going fuzzy all the way, you can do without the initial ^,
and simply let flx (hopefully) sort the matches in a nice way:
What do you do if you want to override a key only in a certain context? Take
this Quotation Marks post as an example. We want to change the " key in
general, but retain the regular behaviour if we’re inside a code-block. In this
case the solution was to just call the old behaviour manually, but what if
you’re writing a more general command and you don’t know what this “old
behaviour” is?
As it turns out, it’s a little-known feature of Emacs that you can specify
filter functions to determine whether a keybind should be active. We use this
quite a bit in SX.el. While the syntax is far from simple, it’s very easy to
copy-paste and just fill in your predicate.
First, we redefine endless/round-quotes without that ugly part that
conditionally calls self-insert-command (previous version here).
(defunendless/round-quotes(italicize)"Insert “” and leave point in the middle.
With prefix argument ITALICIZE, insert /“”/ instead (meant
for org-mode)."(interactive"P")(if(looking-at"”[/=_\\*]?")(goto-char(match-end0))(whenitalicize(if(derived-mode-p'markdown-mode)(insert"__")(insert"//"))(forward-char-1))(insert"“”")(forward-char-1)))
Then, we define the same key as before, but instead of just passing the command
we use a menu-item bound to nil. The fact that it’s a menu-item is irrelevant
here, it behaves exactly like a key bound to nil (i.e., an empty keybind).
However, this allows us to setup a filter that changes the keybind to
endless/round-quotes if we’re not inside an org-src-block.
That maybe-round-quotes is just a useless name for the menu-item, and you can
learn more about all of this on this manual page. For now, it suffices to say we
deserve a more convenient way to use this feature.
Of course, that’s nothing a good macro can’t fix.
(defmacroendless/define-conditional-key(keymapkeydef&restbody)"In KEYMAP, define key sequence KEY as DEF conditionally.
This is like `define-key', except the definition
\"disappears\" whenever BODY evaluates to nil."(declare(indent3)(debug(formformform&restsexp)))`(define-key,keymap,key'(menu-item,(format"maybe-%s"(or(car(cdr-safedef))def))nil:filter(lambda(&optional_)(when,(macroexp-prognbody),def)))))
I’ve written before about what prettify-symbols-mode can do for your buffers,
ranging from pure eye-candy to signficant readability improvements. Simply put,
this minor-mode “disguises” some strings in your buffer to look like something
else. For instance, in emacs-lisp-mode it makes lambda be displayed as λ,
and (for the next release) it’ll apply to a wide range of symbols in
(la)tex-mode too.
The only problem is that this (obviously) hides the symbol itself, and there’s
no way of revealing it other than disabling the mode. That’s usually not a big
deal, but it can get just a little annoying sometimes—specially when a mode
adds over 400 elements to prettify-symbols-alist.
In Emacs 25.1, thanks to Tassilo Horn, there’s a new variable called
prettify-symbols-unprettify-at-point. If you set it to t, prettify-symbols
will “unprettify” a symbol as long as the cursor is inside it. So you can easily
(and temporarily) reveal a symbol by just moving over to it.
My preference is to set it to right-edge which also reveals the symbol if the
point is immediately after it.
Today’s tip is one I learned from Magnar. A lot of Emacsers don’t know this, but
most commands that move point large distances (like isearch or
end-of-buffer) push the old position to the mark-ring. The advantage is that
you can easily jump back through this history of positions by hitting C-u
C-SPC.
This is a hugely convenient take-me-back-to-that-last-place command. The only
problem is that sometimes the ring gets filled with repeated entries, so you
find yourself hitting C-u C-SPC 2 to 4 times in the same place. Of course,
this is Emacs, so all it takes to solve our problem is one simple advice.
;; When popping the mark, continue popping until the cursor;; actually moves(defadvicepop-to-mark-command(aroundensure-new-positionactivate)(let((p(point)))(dotimes(i10)(when(=p(point))ad-do-it))))
Finally, a simple setq ensures we can quickly pop the mark several times by
typing C-u C-SPC C-SPC, instead of having to type C-u C-SPC C-u C-SPC.
(setqset-mark-command-repeat-popt)
Update 20 Jan 2016
Kaushal Modi provides us with how this advice would look using the new
advice-add interface.
(defunmodi/multi-pop-to-mark(orig-fun&restargs)"Call ORIG-FUN until the cursor moves.
Try the repeated popping up to 10 times."(let((p(point)))(dotimes(i10)(when(=p(point))(applyorig-funargs)))))(advice-add'pop-to-mark-command:around#'modi/multi-pop-to-mark)
Although there’s a surprising number of packages offering alternative minibuffer
selection systems, the default minibuffer completion in Emacs is nothing to be
scoffed at. Hitting Tab in the minibuffer gives you a slightly beefed up
version of the bash completion, and after all these years that is still my
preferred method for completing file-names (though I do have some custom-written
alternatives).
Still, there are two small changes you can make that significantly improve the
convenience of this feature. The simplest one is to make it case-insensitive.
The second, and more fine-tuned, is to exclude extensions which you never really
want. These are usually auto-generated files whose names are similar to files
you care about. My list is useful for elisp and LaTeX, so you may need to figure
out yours.
I’m an occasional user of some of the Emacs chat clients. Erc and jabber are
both powerful packages, and it’s great to be able to use Slack, Gitter, and
Google Chat from the cosy comfort of my Emacs frame. If I have one complain,
though, it’s that neither of them has a functionality for keeping track of what
part of the conversation I’ve already read.
Of course, this is Emacs. And the solution is but a hack away.
(defunendless/mark-read()"Mark buffer as read up to current line."(let((inhibit-read-onlyt))(put-text-property(point-min)(line-beginning-position)'face'font-lock-comment-face)))(defunendless/bury-buffer()"Bury buffer and maybe close its window."(interactive)(endless/mark-read)(bury-buffer)(when(cdr(window-listnil'nomini))(delete-window)))(eval-after-load'jabber'(define-keyjabber-chat-mode-map(kbd"<escape>")#'endless/bury-buffer))(eval-after-load'erc'(define-keyerc-mode-map(kbd"<escape>")#'endless/bury-buffer))
Whenever you’re done reading a conversation, just hit Esc. The buffer will be
buried, and next time you open it again everything you had read before will be
marked in grey.
Ivy-mode is a completion method
that's similar to Ido, but with emphasis on simplicity and
customizability. Currently, there are two related packages on MELPA:
swiper and counsel:
swiper provides an isearch replacement, using ivy-read for
completion, as well as the basic ivy-mode.
counsel provides some extra commands that use ivy-read, like -M-x, -ag,
-load-theme etc.
Release Notes
The release notes are available at the
homepage as usual.
There are 220 commits since the last release, which was on Aug 5, roughly
4 months ago. Slowly but surely, the
contributors list
has grown to 20 people, besides me. A few people even got their Emacs
Copyright Assignment just to make large contributions. Statistically,
Org-mode is probably the prime package that leads to the most CA, but
I'm glad that Ivy is there as well, contributing in a small way.
The release notes are made in
Org-mode,
each new version is a level 1 heading. With time, hopefully, their
parts will make their way into the
manual,
which is also in Org-mode.
In addition, I also export each new release notes to
Markdown using
pandoc, since that's what Github prefers.
Release Process
Since the release notes for 0.7.0 are huge, I don't embed them into
the post, they're listed separately. You can go through them at your
own pace, or wait until I make some highlights for each piece either
or the blog or in the manual. The manual, by the way, is a work in
progress but is already distributed in MELPA. Use C-h i
followed by g(ivy) to read it. If you're new to reading
the info pages, there's info on info in info format: use dg(info) to access it.
Today, I'll describe some cool stuff that I used to generate the
Markdown notes from Changelog.org.
Org-mode's org-narrow-to-subtree
Since Changelog.org already has a 0.6.0 branch that I didn't want
to see, I've narrowed the buffer to only the 0.7.0 branch. This is
possible to do thanks to Emacs' narrowing feature and
org-narrow-to-subtree. After this command, the buffer behaves as if
the 0.6.0 branch isn't there and the only content is 0.7.0 branch.
But if I perform any edits and save the file, everything that was
hidden is still there.
As a shortcut, I'm using my worf to narrow faster.
Pressing [ while worf-mode is active goes back to the current heading start.
While at heading start, pressing alphanumeric keys calls commands instead of self-inserting:
N calls org-narrow-to-subtree,
W calls widen, which turns narrowing off.
There are, of course, many more commands and bindings in worf. Check
it out if you like Org's Speed Keys feature, but feel like it could
use more structure.
pandoc-mode
pandoc-mode is an Emacs
interface to pandoc - a tool that allows to
export documents from one format to another.
The Elisp package is available in MELPA.
And I installed pandoc-1.15.2-1-amd64.deb from
its homepage with:
sudo dpkg -i pandoc-1.15.2-1-amd64.deb
After that, M-xpandoc-mode and I'm on easy street: C-c / calls pandoc-main-hydra/body:
Set the output format to Github-flavored Markdown with OG.
Set the input format to Org-mode with bIo.
Export with C-c / r.
View the resulting buffer with C-c / V.
Add table of contents to Markdown
I used M-xmarkdown-toc/generate-toc for this. The MELPA
package markdown-toc
provides this function. The resulting table of contents is a list
with a bunch of links, which turned out to be dead, because of the way
Github renders Markdown for releases.
Use swiper to replace each Markdown link with its title
Since swiper works with regexps, here's what I input to match each link:
\[\(.*?\)\](.*?)
This matches anything in brackets (non-greedy), followed by anything
in parens; the bracket's content is captured in a group.
Then I press M-q (swiper-query-replace) and enter \1 as
replacement - the first captured group. After this, I confirm each
replacement with y or confirm them all at once with
!.
Use rectangle-mark-mode to promote TOC one level
Since all entries in the TOC were children to a single 0.7.0 entry,
I wanted to remove that entry and promote its children one level.
This can be done with rectangle-mark-mode, bound by default to C-x SPC.
Where, hydra-rectangle/body is provided by hydra-examples.el and
is also described in
an earlier post. I
really liked the hydra-rectangle/body idea and use it all the time.
Here's a key sequence I used to delete a 4x95 rectangle in order to
promote the list items: C-x SPC 4l95jdo.
Use basic Elisp to turn each bug reference into an Org-mode link
When copy-pasting from the commit log into Changelog.org, I quickly
tired of putting the each issue link as
e.g. [[https://github.com/abo-abo/swiper/issues/244][#244]]. So I
wrote e.g. #244 instead, and used this code in the end to make the
transformation:
If anyone reading the blog wants to start with some basic Elisp, the
above function is a nice intro to a lot of useful functions. And I
expect that many people face this sort of automation scenario pretty
often. I'm pretty sure that M-%query-replace-regexp
could work here as well, but it's easier for me to just write out the
code and save it for later.
Outro
Thanks to everyone who contributed issues, code and
documentation. Enjoy the new release.
It’s been two years since I last wrote about Elfeed, my
Atom/RSS feed reader for Emacs. I’ve used it every single
day since, and I continue to maintain it with help from the community.
So far 18 people besides me have contributed commits. Over the last
couple of years it’s accumulated some new features, some more obvious
than others.
Every time I mark a new release, I update the ChangeLog at the top of
elfeed.el which lists what’s new. Since it’s easy to overlook many of
the newer useful features, I thought I’d list the more important ones
here.
Custom Entry Colors
You can now customize entry faces through elfeed-search-face-alist.
This variable maps tags to faces. An entry inherits the face of any
tag it carries. Previously “unread” was a special tag that got a bold
face, but this is now implemented as nothing more than an initial
entry in the alist.
I’ve been using it to mark different kinds of content (videos,
podcasts, comics) with different colors.
Autotagging
You can specify the starting tags for entries from particular feeds
directly in the feed listing. This has been a feature for awhile now,
but it’s not something you’d want to miss. It started out as a feature
in my personal configuration that eventually migrated into Elfeed
proper.
For example, your elfeed-feeds may initially look like this,
especially if you imported from OPML.
If you wanted certain tags applied to entries from each, you would
need to putz around with elfeed-make-tagger. For the most common
case — apply certain tags to all entries from a URL — it’s much
simpler to specify the information as part of the listing itself,
Today I only use custom tagger functions in my own configuration to
filter within a couple of particularly noisy feeds.
Arbitrary Metadata
Metadata is more for Elfeed extensions (i.e. elfeed-org)
than regular users. You can attach arbitrary, readable
metadata to any Elfeed object (entry, feed). This metadata is
automatically stored in the database. It’s a plist.
Metadata is accessed entirely through one setf-able function:
elfeed-meta. For example, you might want to track when you’ve read
something, not just that you’ve read it. You could use this to
selectively update certain feeds or just to evaluate your own habits.
Two things motivated this feature. First, without a plist, if I added
more properties in the future, I would need to change the database
format to support them. I modified the database format to add
metadata, requiring an upgrade function to quietly upgrade older
databases as they were loaded. I’d really like to avoid this in the
future.
Second, I wanted to make it easy for extension authors to store their
own data. I still imagine an extension someday to update feeds
intelligently based on their history. For example, the database
doesn’t track when the feed was last fetched, just the date of the
most recent entry (if any). A smart-update extension could use
metadata to tag feeds with this information.
Elfeed itself already uses two metadata keys: :failures on feeds and
:title on both. :failures counts the total number of times
fetching that feed resulted in an error. You could use this get a
listing of troublesome feeds like so,
The :title property allows for a custom title for both feeds and
entries in the search buffer listing, assuming you’re using the
default function (see below). It overrides the title provided by the
feed itself. This is different than elfeed-entry-title and
elfeed-feed-title, which is kept in sync with feed content. Metadata
is not kept in sync with the feed itself.
Filter Inversion
You can invert filter components by prefixing them with !. For
example, say you’re looking at all my posts from the past 6 months:
@6-months nullprogram.com
But say you’re tired of me and decide you want to see every entry from
the past 6 months excluding my posts.
@6-months !nullprogram.com
Filter Limiter
Normally you limit the number of results by date, but you can now
limit the result by count using #n. For example, to see my most
recent 12 posts regardless of date,
nullprogram.com #12
This is used internally in the live filter to limit the number of
results to the height of the screen. If you noticed that live
filtering has been much more responsive in the last few months, this is
probably why.
Bookmark Support
Elfeed properly integrates with Emacs’ bookmarks (thanks to
groks). You can bookmark the current filter with M-x
bookmark-set (C-x r m). By default, Emacs will persist bookmarks
between sessions. To revisit a filter in the future, M-x
bookmark-jump (C-x r b).
Since this requires no configuration, this may serve as an easy
replacement for manually building “view” toggles — filters bound to
certain keys — which I know many users have done, including me.
New Header
If you’ve updated very recently, you probably noticed Elfeed got a
brand new header. Previously it faked a header by writing to the first
line of the buffer. This is because somehow I had no idea Emacs had
official support for buffer headers (despite notmuch using them all
this time).
The new header includes additional information, such as the current
filter, the number of unread entries, the total number of entries, and
the number of unique feeds currently in view. You’ll see this as
<unread>/<total>:<feeds> in the middle of the header.
As of this writing, the new header has not been made part of a formal
release. So if you’re only tracking stable releases, you won’t see
this for awhile longer.
You can supply your own header via elfeed-search-header-function
(thanks to Gergely Nagy).
Scoped Updates
As you already know, in the search buffer listing you can press G to
update your feeds. But did you know you it takes a prefix argument?
Run as C-u G, it only updates feeds with entries currently listed in
the buffer.
As of this writing, this is another feature not yet in a formal
release. I’d been wanting something like this for awhile but couldn’t
think of a reasonable interface. Directly prompting the user for feeds
is neither elegant nor composable. However, groks suggested the
prefix argument, which composes perfectly with Elfeed’s
existing idioms.
Listing Customizations
In addition to custom faces, there are a number of ways to customize
the listing.
Choose the sort order with elfeed-sort-order.
Set a custom date format with elfeed-search-date-format.
Adjust field widths with elfeed-search-*-width.
Or override everything with elfeed-search-print-entry-function.
Gergely Nagy has been throwing lots of commits at me over the last
couple of weeks to open up lots of Elfeed’s behavior to customization,
so there are more to come.
Thank You, Emacs Community
Apologies about any features I missed or anyone I forgot to mention
who’s made contributions. The above comes from my ChangeLogs, the
commit log, the GitHub issue listing, and my own memory, so I’m likely
to have forgotten some things. A couple of these features I had
forgotten about myself!
A few weeks ago I noticed a new package on Melpa called github-notifier by
Chunyang, which displays a count of your Github notifications on the mode-line.
Instead of just installing the package like a normal person, I had an urge to
try and see how hard it would be to write from scratch. Paradox already has a
function for interacting with the Github API, so it’s just a matter of putting
it to work.
The first thing you need to do is visit your Github tokens page, and edit the
Paradox token to allow access to your notifications.
Next, define a command to visit your notifications, and a function to display a
button in the mode-line.
For convenience, I’ve also added visit-notifications to my launcher-map under
n.
Next, write a function to query the Github API and pass the vector of
results to count-for-mode-line.
;; If already installed, this does nothing.(package-install'paradox)(autoload'paradox--github-action"paradox-github")(defunendless/check-gh-notifications()"Check for github notifications and update the mode-line."(paradox--github-action"notifications":reader(lambda()(let((json-falsenil)(json-array-type'list))(json-read))):callback#'endless/count-for-mode-line))
Finally, just set the timer to some convenient interval. I run it at every 30
seconds of idle time.
And that’s all! Roughly 30 lines of code and you have a convenient notification
system. Whenever you see that GH-2 show up at the corner of your mode-line,
just click on it (or type C-x l n). That will remove the button and take you
to Github. On Github you can navigate with j, k, and RET, and “mark as
read” with m.
Some people find lispy too weird
and/or complex to try, quite possibly because of it's sort-of-modal
key binding structure. Which is a shame, since out of 7k lines of
lispy's code, only 500 lines do key bindings, the rest do all sorts of
useful stuff, like sexp navigation/modification, outlines and REPL
interaction.
In this post, I'll show a short example of using lispy's functions
outside of lispy-mode, and compare it with the default approach.
Moving s-expressions while the point is anywhere
Using this simple hydra, and the
key sequence C-c m sss www I got the following GIF:
Doing the same lispy-way
To do it the lispy-way, I move the point before the list that I wish
to operate on (with
[ key), and press sss www to get the following GIF:
The difference here is that no hydra is necessary in the second case,
the key bindings come only from lispy-mode. But the first case will
also work if lispy-mode is off.
Which way is better?
I think the second way is better, since it's faster and more clear,
but you can decide for yourself. By the way, you can also move
symbols, comments and sub-words (if that makes sense) with
w and s, provided you mark them with a region
first.
I can’t write a whole new post this week due to being buried under some
once-in-a-lifetime stuff. Still, the Monday post is one I refuse to miss, so I
leave you today with an update on last weeks post.
Running tests on file save really didn’t work out for me, because I’ve configured
CIDER to save files before loading them when I use cider-load-file. So the
tests were always being run against outdated definitions.
The fix is simple, though. Just run the tests after load-file. CIDER even had
a hook for that already. I also made the minor-mode global (even though it only
applies to clojure buffers) so that it’s easier to disable everywhere with a
single command.
I recently released an Emacs package called x86-lookup.
Given a mnemonic, Emacs will open up a local copy of an Intel’s
software developer manual PDF at the page documenting the
instruction. It complements nasm-mode, released earlier this
year.
To use it, you’ll need Poppler’s pdftotext command line
program — used to build an index of the PDF — and a copy of the
complete Volume 2 of Intel’s instruction set manual. There’s only one
command to worry about: M-x x86-lookup.
Minimize documentation friction
This package should be familiar to anyone who’s used
javadoc-lookup, one of my older packages. It has a common
underlying itch: the context switch to read API documentation while
coding should have as little friction as possible, otherwise I’m
discouraged from doing it. In an ideal world I wouldn’t ever need to
check documentation because it’s already in my head. By visiting
documentation frequently with ease, it’s going to become familiar that
much faster and I’ll be reaching for it less and less, approaching the
ideal.
I picked up x86 assembly [about a year ago][x86] and for the first few
months I struggled to find a good online reference for the instruction
set. There are little scraps here and there, but not much of
substance. The big exception is Félix Cloutier’s reference,
which is an amazingly well-done HTML conversion of Intel’s PDF
manuals. Unfortunately I could never get it working locally to
generate my own. There’s also the X86 Opcode and Instruction
Reference, but it’s more for machines than humans.
Besides, I often work without an Internet connection, so offline
documentation is absolutely essential. (You hear that Microsoft? Not
only do I avoid coding against Win32 because it’s badly designed, but
even more so because you don’t offer offline documentation anymore!
The friction to API reference your documentation is enormous.)
I avoided the official x86 documentation for awhile, thinking it would
be too opaque, at least until I became more accustomed to the
instruction set. But really, it’s not bad! With a handle on the
basics, I would encourage anyone to dive into either Intel’s or AMD’s
manuals. The reason there’s not much online in HTML form is
because these manuals are nearly everything you need.
I chose Intel’s manuals for x86-lookup because I’m more familiar with
it, it’s more popular, it’s (slightly) easier to parse, it’s offered
as a single PDF, and it’s more complete. The regular expression for
finding instructions is tuned for Intel’s manual and it won’t work
with AMD’s manuals.
For a couple months prior to writing x86-lookup, I had a couple of
scratch functions to very roughly accomplish the same thing. The
tipping point for formalizing it was that last month I wrote my own
x86 assembler. A single mnemonic often has a dozen or more different
opcodes depending on the instruction’s operands, and there are often
several ways to encode the same operation. I was frequently looking up
opcodes, and navigating the PDF quickly became a real chore. I only
needed about 80 different opcodes, so I was just adding them to the
assembler’s internal table manually as needed.
How does it work?
Say you want to look up the instruction RDRAND.
Initially Emacs has no idea what page this is on, so the first step is
to build an index mapping mnemonics to pages. x86-lookup runs the
pdftotext command line program on the PDF and loads the result into a
temporary buffer.
The killer feature of pdftotext is that it emits FORM FEED (U+0012)
characters between pages. Think of these as page breaks. By counting
form feed characters, x86-lookup can track the page for any part of
the document. In fact, Emacs is already set up to do this with its
forward-page and backward-page commands. So to build the index,
x86-lookup steps forward page-by-page looking for mnemonics, keeping
note of the page. Since this process typically takes about 10 seconds,
the index is cached in a file (see x86-lookup-cache-directory) for
future use. It only needs to happen once for a particular manual on a
particular computer.
The mnemonic listing is slightly incomplete, so x86-lookup expands
certain mnemonics into the familiar set. For example, all the
conditional jumps are listed under “Jcc,” but this is probably not
what you’d expect to look up. I compared x86-lookup’s mnemonic listing
against NASM/nasm-mode’s mnemonics to ensure everything was accounted
for. Both packages benefited from this process.
Once the index is built, pdftotext is no longer needed. If you’re
desperate and don’t have this program available, you can borrow the
index file from another computer. But you’re on your own for figuring
that out!
So to look up RDRAND, x86-lookup checks the index for the page number
and invokes a PDF reader on that page. This is where not all PDF
readers are created equal. There’s no convention for opening a PDF to
a particular page and each PDF reader differs. Some don’t even support
it. To deal with this, x86-lookup has a function specialized for
different PDF readers. Similar to browse-url-browser-function,
x86-lookup has x86-lookup-browse-pdf-function.
By default it tries to open the PDF for viewing within Emacs (did you
know Emacs is a PDF viewer?), falling back to on options if the
feature is unavailable. I welcome pull requests for any PDF readers
not yet supported by x86-lookup. Perhaps this functionality deserves
its own package.
That’s it! It’s a simple feature that has already saved me a lot of
time. If you’re ever programming in x86 assembly, give x86-lookup a
spin.
As I was catching up on a few Parens of the Dead episodes this weekend, I was
amused at how Magnar set up his Emacs to run tests whenever the file is saved.
At first I thought it wasn’t for me (I’m one of those who obsessively saves
every few seconds), but I’ve been trying it out lately and it’s starting to grow
on me.
At its core, all you really need is an hook. But I walked the extra yard and
wrote a minor mode for it, so I can easily call M-xtdd-mode to disable it
if it ever gets on my nerves.
(defuntdd-test()"Thin wrapper around `cider-test-run-tests'."(when(cider-connected-p)(let((cider-auto-select-test-report-buffernil)(cider-test-show-report-on-successnil))(cider-test-run-ns-testsnil'soft))))(define-minor-modetdd-mode"Run all tests whenever a file is loaded."nilnilnil:globalt(iftdd-mode(add-hook'cider-file-loaded-hook#'tdd-test)(remove-hook'cider-file-loaded-hook#'tdd-test)))
I also had to change the cider-test-success-face to something a little less
“screamy”.
The concept of a lazy-map might sounds odd at first. How do you know if a map
contains an entry without resolving the whole map? But it’s not the entries that
are lazy, it’s the values they hold. See this example from the Readme:
Lazy-map is on Clojars, so you can just add it as a dep and play around:
How would this ever be useful?
One of my side projects a few months ago (when I was playing around with
Clojure-on-Android) was an Android client for a desktop Java application.
Because of this, a lot of the code was about interfacing with Java objects
defined by this package. These objects were from many different classes, but
the one thing they had in common is that they were full of get methods.
I wanted so much to be able to use these objects as maps that I wrote a protocol
for converting general objects to Clojure maps. Here’s an example of how it
worked on a string.
user>(to-map"My own String!"){:to-char-array#object["[C"0xdf4ddb3"[C@df4ddb3"],:empty?false,:to-string"My own String!",:chars#object[java.util.stream.IntPipeline$Head0xad35343"java.util.stream.IntPipeline$Head@ad35343"],:classjava.lang.String,:length14,:trim"My own String!",:bytes#object["[B"0x75ef7d8f"[B@75ef7d8f"],:hash-code1673659170,:object"My own String!",:to-upper-case"MY OWN STRING!"}
For comparison, here’s how bean works (a similar function from
clojure.core).
user>(bean"My own String!"){:bytes#object["[B"0x1ad60072"[B@1ad60072"],:classjava.lang.String,:emptyfalse}
The protocol is actually quite smart. It uses a number of heuristics to only
convert methods that look like they’re side-effect free. Of course, it’s not
foolproof (this is Java we’re talking about), but the macro used to extend the
protocol lets you specify methods to exclude.
The only problem was the performance cost. Some of these methods were very
expensive to run, and eagerly calling all methods of all objects just so I could
later access some of these was obviously a bad deal. The solution, clearly, was
to only call these methods when the map entries were actually accessed. And so
lazy-map was born.
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
As usual, we’ll be starting at 6:30 with pizza and beer!
Diego Berrocal websitetwittergithub is a Recurser at the Recurse Center and he will be teaching us how to order salad using Emacs
I have been eating exclusively salads the past few weeks, and I have been using emacs for programming that whole time. It was about time I merged those interests together. I’ll show how I used the request.el library and dash.el to have a more functional paradigm.
prettify-symbols-mode is a very nice minor-mode that is a little too modest
for its own good. You can turn it on right now if you’re using a recent Emacs,
but it’ll do nothing more than turn lambda into λ in emacs-lisp-mode.
Still, it’s powerful and versatile and deserves that you give it a try. To
extend its feature-set you can install packages or customize it yourself, and
that’s what we’re here to do today, specifically in clojure-mode.
First of all, let’s make sure it’s turned on.
(global-prettify-symbols-mode1);; We're going to play with this below.(defvarendless/clojure-prettify-alist'())
If you read the docstring for this mode, it’ll explain that any symbol can be
displayed as any character, so the first thing that comes to my mind is
displaying <= or >= as ≤ or ≥. But that comes with a drawback. Suddenly
that symbol is 1 character shorter, so Emacs is going to indent sexps
accordingly, and people reading your code will see bad indentation.
The solution is to configure prettify-symbols-mode to compose these symbols in
a special way. This feature is somewhat accidental, and wasn’t even documented
in the last release. The composition rules are slightly complicated to write,
and for that I’ll point to the docstrings of compose-region (see the third
argument) and reference-point-alist. Below are several examples you can toy
around with.
One way to fix the width, is to join two spaces together, and then stick the
inequality on top of them.
The option I prefer is to just add a small dot before the symbol. This makes it
clear that the symbol occupies the space of two characters, while still looking
nicer than a plain <=.
Then there are the -> and ->> macros, which are in dire need of a makeover
if you ask me. The solution I currently use is a spaced-out version of 🠊
(you could also use ⇨, 🡒, or →).
Here’s what they look like with this, compared to what they usually look like.
And finally, none of this will work if we don’t set it up. Note that
clojure-mode already defines fn to display as λ, so we don’t need to
configure this one.
I've had an idea with this feature for quite some time, and only
recently got to finally implement it. So here it goes: with
ivy-occur, bound to C-c C-o while in the minibuffer, you
can store almost Ivy any
completion session in progress, and you can have as many of them as
you like.
Example 1
This is where the command name originates from: swiper + ivy-occur
= occur. You can store all of swiper's matching lines into a
separate buffer. This doesn't give too much advantage over the
good-old occur, the only thing is that you can use ivy-style regexps
with wild spaces, and have an interactive preview beforehand. Having
said that, it actually sounds pretty good!
Example 2
I press C-S-o, bound to counsel-rhythmbox, and enter
u2. After pressing C-c C-o, bound to ivy-occur, the
completion session is closed (effectively C-g), and a new
buffer *ivy-occur counsel-rhythmbox "u2"* is generated with all
songs that match u2.
As a reminder, counsel-rhythmbox has two actions: play (the
default) and enqueue. In this new buffer, pressing RET or
mouse-1 will call the play action (since it was active at the
moment ivy-occur was called) for the current candidate. So I've
effectively added a playlist functionality to counsel-rhythmbox
through a generic command.
Note that it's very easy to identify a completion session by the
command name and input. So I can distinguish e.g. *ivy-occur
counsel-rhythmbox "u2"* and *ivy-occur counsel-rhythmbox "скря"*,
and quickly select them with ivy-switch-buffer: just input rhy,
usually only these two and similar buffers will match.
Example 3
Suppose I want to go through the documentation of all variables that
end in -function. Easy:
<f1> v (counsel-describe-variable) with input
function$.
C-c C-o (ivy-occur).
I get a new buffer named *ivy-occur cousnel-describe-variable
"function$*" with 346 candidates. I can go through them at my own
pace, possibly doing other completion stuff in between without
disturbing my process. It's also convenient to navigate these buffers
with swiper.
Example 4
Let's tweak the previous one. After inputting function$ I can press
C-M-a (ivy-read-action) followed by d to
select the definition action. Then again C-c C-o
(ivy-occur). Pressing RET in the resulting buffer will
take me to that variable's definition, instead of describing it as
before. A similar thing could be done for counsel-rhythmbox to get
the enqueue action in *ivy-occur* buffer, instead of play.
Example 5
This is an improvement to my workflow for quickly looking at a new
package's features. This can be done with oge
(lispy-goto-elisp-commands) from
lispy, which scans the source for
(interactive) tags and allows to jump to their definition.
It's a pretty cool optimization, compared to looking at all tags. For
example, projectile currently
has 375 top-level tags (functions and variables). But with
lispy-goto-elisp-commands I only get 48 tags. And now I can have
them in a convenient permanent list too.
Alternatively, if projectile is already loaded, I can use
counsel-M-x with input projectile-, followed by C-M-a d
to select the definition action, followed by C-c C-o.
As a reminder of how it works, counsel-M-x is defined with a single
action that calls the selected command. But then, I've also added this
statement at top-level of counsel.el:
This means that you can add as many actions as you like to ivy-read
commands. And of course customize the binding and the hint, which are
in this case d and definition respectively.
Limitations
Unfortunately, since the *ivy-occur* buffer needs to know the action
to execute, it only works for commands that explicitly pass :action
to ivy-read. For instance, it won't work for package-install with
ivy-mode on: the buffer will be properly generated, but pressing
RET won't install a package.
Fortunately, it's not hard to write a version that works:
Here's a buffer with a list of packages matching "ga"; pressing
RET will install the selected package:
Small note on key bindings
In the initial post, I wanted to bind ivy-occur to C-c o
instead of C-c C-o. But I was reminded that C-c LETTER are
reserved.
I still think it's a better binding. If you agree, you can add it to your config:
Additionally, ivy-occur is also available on C-o u,
through the C-o hydra.
Outro
I think ivy-occur is a very powerful command that shouldn't be
overlooked. Just as ivy-resume implements a sort of DEL
or C-u key for completion, ivy-occur implements a
convenient way to switch and store the completion context, a sort of
C-x o or C-x b for completion.
When I first started learning Clojure, I was charmed by how well integrated
CIDER was with Emacs. In many ways, it felt just like writing Emacs-lisp.
Nowadays, that feeling has gone slightly past the goal mark, and there are
actually features I miss when I’m writing elisp. Clj-refactor is one of them.
This package is a Swiss knife of refactoring utilities for Clojure. From simple
operations like turning a form into a -> thread, to the more complex
situations like renaming entire namespaces, it’s hard to find something
clj-refactor can’t do.
I won’t list all of them here, they already have a nice wiki page for that. The
point here is to expose a few features (the ones I find most useful) and
hopefully that’ll get you started on the road to enlightenment. To see some of
the features in play (at a pretty quick pace), you can check out Magnar’s Parens
of the dead videos. I also plan on doing a proper and thorough video review of
this package in the future.
First, and most importantly, are the threading commands. These allow you to
instantly turn something like (filter odd? (map inc (range n))) into
(->>(rangen)(mapinc)(filterodd?))
or back again. Clj-refactor default keybinds are pretty well thought out, but I
use these commands so frequently that I felt the need to shorten them to just 2
or 3 keys.
Next is another feature that comes up a lot, but this one doesn’t even take up a
keybind. When you type /, clj-refactor will look at the alias you’ve just
written, and try to add a require for it in the ns form. The current snapshot
release is very smart about it, and actually learns what alias you use for each
namespace by looking at other files in the same project. You can also define
your preferences with the cljr-magic-require-namespaces variable.
Another command you should definitely memorize is cljr-create-fn-from-example.
Instead of specifying a custom key for this one, we’ll just let the package
define all other keys for us.
The nice thing about the default keys is that they’re mnemonic, and you might
remember I have an entire series on this topic. The point being: it’s very easy
to remember that cljr-create-fn-from-example is bound to C-c r f e when you
can actually sing “clojure refactor from example” as you’re typing.
A few others I use a lot (and the keys they’re bound to) are introduce-let (i
l), move-to-let (m l), and promote-function (p f).
Lastly, some miscellaneous configurations that simply reflect my personal
preferences, but which you should be aware of nonetheless.
The dot-dot-dot ellipsis that org-mode uses to indicate hidden content is
usually just fine. It’s only when you’re staring at a document where every line
is a folded headline, that you start to feel like they’re a little too much “in
your face”. I have a few org files with thousands of lines and hundreds of
headlines, and changing that ... to something shorter greatly reduces visual
clutter.
The more straightforward option is to use a proper ellipsis character (the same
effect with a third the length).
(setqorg-ellipsis"…")
The one I’m currently using is a cornered arrow.
(setqorg-ellipsis"⤵")
Other interesting characters are ▼, ↴, ⬎, ⤷, and ⋱.
Emacs comes with a wonderful arbitrary-precision computer algebra
system called calc. I’ve discussed it previously and
continue to use it on a daily basis. That’s right, people, Emacs can
do calculus. Like everything Emacs, it’s programmable and extensible
from Emacs Lisp. In this article, I’m going to implement the RSA
public-key cryptosystem in Emacs Lisp using calc.
If you want to dive right in first, here’s the repository:
This is only a toy implementation and not really intended for serious
cryptographic work. It’s also far too slow when using keys of
reasonable length.
Evaluation with calc
The calc package is particularly useful when considering Emacs’
limited integer type. Emacs uses a tagged integer scheme where
integers are embedded within pointers. It’s a lot faster than the
alternative (individually-allocated integer objects), but it means
they’re always a few bits short of the platform’s native integer type.
calc has a large API, but the user-friendly porcelain for it is the
under-documented calc-eval function. It evaluates an expression
string with format-like argument substitutions ($n).
Notice it returns strings, which is one of the ways calc represents
arbitrary precision numbers. For arguments, it accepts regular Elisp
numbers and strings just like this function returns. The implicit
radix is 10. To explicitly set the radix, prefix the number with the
radix and #. This is the same as in the user interface of calc. For
example:
(calc-eval"16#deadbeef");; => "3735928559"
The second argument (optional) to calc-eval adjusts its behavior.
Given nil, it simply evaluates the string and returns the result.
The manual documents the different options, but the only other
relevant option for RSA is the symbol pred, which asks it to return
a boolean “predicate” result.
(calc-eval"$1 < $2"'pred"4000""5000");; => t
Generating primes
RSA is founded on the difficulty of factoring large composites with
large factors. Generating an RSA keypair starts with generating two
prime numbers, p and q, and using these primes to compute two
mathematically related composite numbers.
calc has a function calc-next-prime for finding the next prime
number following any arbitrary number. It uses a probabilistic
primarily test — the Fermat Miller-Rabin primality test
— to efficiently test large integers. It increments the input until
it finds a result that passes enough iterations of the primality test.
Unfortunately calc’s random function is based on Emacs’ random
function, which is entirely unsuitable for cryptography. In the real
implementation I read n bits from /dev/urandom to generate an n-bit
number.
From here the code just follows along from the Wikipedia article.
After generating the primes p and q, two composites are computed,
n = p * q and i = (p - 1) * (q - 1). Lacking any reason to do
otherwise, I chose 65,537 for the public exponent e.
The function rsa--inverse is just a straight Emacs Lisp + calc
implementation of the extended Euclidean algorithm from the Wikipedia
article pseudocode, computing d ≡ e^-1 (mod i). It’s not much
use sharing it here, so take a look at the repository if you’re
curious.
(defunrsa-generate-keypair(bits)"Generate a fresh RSA keypair plist of BITS length."(let*((p(rsa-generate-prime(+1(/bits2))))(q(rsa-generate-prime(+1(/bits2))))(n(calc-eval"$1 * $2"nilpq))(i(calc-eval"($1 - 1) * ($2 - 1)"nilpq))(e(calc-eval"2^16+1"))(d(rsa--inverseei)))`(:public(:n,n:e,e):private(:n,n:d,d))))
The public key is n and e and the private key is n and d. From
here we can compute and verify cryptographic signatures.
Signatures
To compute signature s of an integer m (where m < n), compute
s ≡ m^d (mod n). I chose the right-to-left binary method, again
straight from the Wikipedia pseudocode (lazy!). I’ll share this
one since it’s short. The backslash denotes integer division.
Verifying the signature is the same process, but with the public key’s
e: m ≡ s^e (mod n). If the signature is valid, m will be
recovered. In theory, only someone who knows d can feasibly compute
s from m. If n is small enough to factor, revealing
p and q, then d can be feasibly recomputed from the public key.
So mind your Ps and Qs.
So that leaves one problem: generally users want to sign strings and
files and such, not integers. A hash function is used to reduce an
arbitrary quantity of data into an integer suitable for signing. Emacs
comes with a bunch of them, accessible through secure-hash. It
hashes strings and buffers.
Since the result is hexadecimal, just prefix 16# to turn it into a
calc integer.
Here’s the signature and verification functions. Any string or buffer
can be signed.
(defunrsa-sign(private-keyobject)(let((n(plist-getprivate-key:n))(d(plist-getprivate-key:d))(hash(concat"16#"(secure-hash'sha384object))));; truncate hash such that hash < n(while(calc-eval"$1 > $2"'predhashn)(setfhash(calc-eval"$1 \\ 2"nilhash)))(rsa--mod-powhashdn)))(defunrsa-verify(public-keyobjectsig)(let((n(plist-getpublic-key:n))(e(plist-getpublic-key:e))(hash(concat"16#"(secure-hash'sha384object))));; truncate hash such that hash < n(while(calc-eval"$1 > $2"'predhashn)(setfhash(calc-eval"$1 \\ 2"nilhash)))(let*((result(rsa--mod-powsigen)))(calc-eval"$1 == $2"'predresulthash))))
Note the hash truncation step. If this is actually necessary, then
your n is very easy to factor! It’s in there since this is just a
toy and I want it to work with small keys.
Putting it all together
Here’s the whole thing in action with an extremely small, 128-bit key.
Each of these operations took less than a second. For larger,
secure-length keys, this implementation is painfully slow. For
example, generating a 2048-bit key takes my laptop about half an hour,
and computing a signature with that key (any size message) takes about
a minute. That’s probably a little too slow for, say, signing ELPA
packages.
What started out as a cute idea I was playing around with, eventually turned to
be one of my favorite packages. Beacon won’t help you type faster, code
better, or cure cancer like some of the other packages. Its effect is mostly
cosmetic, but with practical benefits. Put simply, if you turn on this minor
mode, whenever the window scrolls up or down a light will blink on your cursor.
That’s it.
At first, I thought this would be purely cosmetic. But my super scientific
experience has been that there’s actually a practical benefit to it. The amount
of brain energy you save and the level of eye strain you avoid by having this
simple “eye-guide” is noticeable, specially during those late-night writing
marathons.
Just install from Gelpa with the usual M-xlist-packages and turn it on.
It has a lot of configuration options for you to specify situations where it
shouldn’t blink, as well as different details like duration, delay, size, and
color. Either have a look at the Readme or just issue M-xcustomize-group to
see them all.
Today I'll describe my past and current workflows for compressing
files and directories in Dired. In my opinion, Dired is one of the
best features of Emacs - a great abstraction of cd+ls that lets
you get things done much faster. The default binding for dired is
C-x d (I bind it to C-;-d though, see my
xmodmap post).
My previous workflow
To compress several files in Dired, first I compile a list of files to
work on by marking them with m (dired-mark), navigating
to ones that you want with n (dired-next-line) and
p (dired-previous-line).
Once a list is ready, I press & (dired-do-async-shell-command), which allows me to run an arbitrary shell command.
I enter:
tar -czf ~/tmp/path/foo.tar.gz *
into the minibuffer, press RET and I'm done. It's a
standard shell command with 2 advantages:
I get TAB completion for the directory of the created archive.
* means the selected file list, so I don't enter them by hand.
My current workflow
Since I often forget the command for tar, and I really don't want to
remember the commands for zip etc, I've implemented a more convenient approach:
This new command dired-do-compress-to, bound to c, will
prompt me for a file name of the output archive. So now it's easier to
type in that file name, if you use ido-mode / ivy-mode /
helm-mode. What's more, the command will be automatically determined
from the archive extension and executed. Here's the corresponding
customization variable:
(defvardired-compress-files-alist'(("\\.tar\\.gz\\'"."tar -c %i | gzip -c9 > %o")("\\.zip\\'"."zip %o -r --filesync %i"))"Control the compression shell command for `dired-do-compress-to'.Each element is (REGEXP . CMD), where REGEXP is the name of thearchive to which you want to compress, and CMD the thecorresponding command.Within CMD, %i denotes the input file(s), and %o denotes theoutput file. %i path(s) are relative, while %o is absolute.")
This thing is pretty self-explanatory. It currently only supports
*.tar.gz and *.zip, use M-xreport-emacs-bug if you
want to add more options.
Other improvements in compression/decompression
I improved the good-old Z (dired-do-compress) as
well. Now it can compress directories to *.tar.gz, as well as
decompress *.tar.gz and *.zip to directories. It's all automatic,
you only have to press Z, either on an archive to
decompress, or on a directory to compress.
Easy way to have a bleeding edge Emacs
If you want to get these improvements right now and are on a Debian
type system (I use Ubuntu), you can use:
Then just use emacs-snapshot executable. It updates every few days.
Myself, I track the Emacs master from
git. But emacs-snapshot is
very useful for CI tests, and can also be used by people who don't
want to build the executable from git.
How to get your improvements into Emacs
If you'd like to contribute code to the Emacs core yourself, it takes
two steps:
Sign the Copyright Assignment
(CA).
This gives the Copyright for your changes to Emacs to the Free
Software Foundation. This is useful so that the FSF can defend your
rights in court, in case the GPL regarding your contribution is broken
by someone.
M-xreport-emacs-bug and attach a patch.
It seems that some people could have a moral objection to the CA. I
see no reason for it whatsoever. Since Elisp libraries make calls to
GPL code and are GPL themselves, you have to publish Elisp code under
GPL, there's no choice there. GPL means basically that, while you're
still acknowledged as the author, everyone else has about as much
right to your code as you do. No one can hide or restrict access to
it, but neither can you. The CA is a further step, which makes the FSF
own the Copyright to your changes to Emacs. You submit these changes
yourself, and the Copyright over these changes isn't really that
useful: they're only changes - they do nothing on their own, outside
of Emacs. Besides, with the FSF owning the Copyright, the range of
things that you can do to your code stays exactly the same as if you
owned the Copyright, as long as you publish under GPL.
To illustrate, the FSF CA is very polite and benign, compared to the
CA that I have to give to Elsevier when I publish an article. That CA
means that a multi-billion dollar corporation owns the right to my
paper until the end of time, and can charge people to view it
(including myself and my peers) as much as they see fit (currently,
$30). Of course, I don't receive a dime of that money, but hey - I get
published, and at least I can share my article for free for the
first 50 days of the publish date. And if that one doesn't seem
draconian enough for you, just think of the NDAs in commercial
companies: out there, you can't even publish or mention anything
related to your work. So please, if you're hating on the FSF, do
reconsider, they're the good guys in my book.
Eventually, after a few submitted patches, you'll be granted commit
access. This is a great way to implement small improvements faster.
However, the larger issues that aren't simple improvements still need
to be discussed with other developers, either on debbugs or on
emacs-devel. For example, initially I went with the tar -czf
command. But it turns out that it can't be used on Solaris 10 or AIX
7.1. Whatever those things are, people still use Emacs on them, and
it's important that the core features are usable on all supported
systems.
It’s difficult for me to spend much time interacting with a major-mode and not
find something to tweak. Even when that mode is my oldest companion in the world
of Emacs, something will surely pop up. So it’s only fitting that in the final
week of my thesis submission deadline I start tinkering with latex-mode again.
I wrote the following snippet after compiling a recent Emacs snapshot which
defines a myriad of rules for prettify-symbols-mode in tex-mode. Equations
become much easier to read when you replace the verbose math symbols like
\alpha with α, but that only makes all the LaTeX styling commands stand out
even more. Suddenly, all those \left, \right, \! and \;, are sticking
out like that mole on a hag’s nose.
What do we do about that? We hide them away. Not completely, of course. We still
need to see those commands. We just tuck them away a little, where we can see
them without having to see them.
(deffaceendless/unimportant-latex-face'((t:height0.7:inheritfont-lock-comment-face))"Face used on less relevant math commands.")(font-lock-add-keywords'latex-mode`((,(rx(or(and"\\"(or(any",.!;")(and(or"left""right""big""Big")symbol-end)))(any"_^")))0'endless/unimportant-latex-faceprepend))'end)
Here’s a sample of what it might look like for you.
Quick update to say that I’m quite pleased with the Hydra package. Turns out
it’s not just eye-candy on top of keymaps, it also offers convenient
functionality that is rather dull to implement on plain keymaps.
If you use Paradox, next time you upgrade you’ll find that a sneaky hydra has
made its way into it. It’s a very simple one under the f key, but it’s a nice
improvement over the plain keymap.
Also, if you happen to follow the Emacs master branch, you’ll see that this new
hydra is actually not that simple and includes some filtering options that the
previous keymap didn’t even have. You have Kaushal Modi to thank for this.
If you’re the kind of person who hates mythical creatures, you can revert to the
plain keymap with this snippet.
One of the great advantages of dependency analysis is parallelization.
Modern processors reorder instructions whose results don’t affect each
other. Compilers reorder expressions and statements to improve
throughput. Build systems know which outputs are inputs for other
targets and can choose any arbitrary build order within that
constraint. This article involves the last case.
The build system I use most often is GNU Make, either directly or
indirectly (Autoconf, CMake). It’s far from perfect, but it does what
I need. I almost always invoke it from within Emacs rather than in a
terminal. In fact, I do it so often that I’ve wrapped Emacs’ compile
command for rapid invocation.
I recently helped a co-worker set this set up for himself, so it had
me thinking about the problem again. The situation in my
config is much more complicated than it needs to be, so I’ll
share a simplified version instead.
First bring in the usual goodies (we’re going to be making closures):
Then a couple of interactive functions to set these on the fly. It’s
not strictly necessary, but I like giving each a key binding. I also
like having a history available via read-string, so I can switch
between a couple of different options with ease.
Now finally to the good part. Below, quick-compile is a
non-interactive function that returns an interactive closure ready to
be bound to any key I desire. It takes an optional target. This means
I don’t use the above quick-compile-set-command to choose a target,
only for setting other options. That will make more sense in a moment.
(cl-defunquick-compile(&optional(target""))"Return an interaction function that runs `compile' for TARGET."(lambda()(interactive)(save-buffer); so I don't get asked(let((default-directory(locate-dominating-filedefault-directoryquick-compile-build-file)))(ifdefault-directory(compile(concatquick-compile-command" "target))(error"Cannot find %s"quick-compile-build-file)))))
It traverses up (down?) the directory hierarchy towards root looking
for a Makefile — or whatever is set for quick-compile-build-file
— then invokes the build system there. I don’t believe in recursive
make.
So how do I put this to use? I clobber some key bindings I don’t
otherwise care about. A better choice might be the F-keys, but my
muscle memory is already committed elsewhere.
Each of those invokes a different target without second guessing me.
Let me tell you, having “clean” at the tip of my fingers is wonderful.
Parallel Builds
An extension common to many different make programs is -j, which
asks make to build targets in parallel where possible. These days
where multi-core machines are the norm, you nearly always want to use
this option, ideally set to the number of logical processor cores on
your system. It’s a huge time-saver.
My recent revelation was that my default build command could be
better: make -k is minimal. It should at least include -j, but
choosing an argument (number of processor cores) is a problem. Today I
use different machines with 2, 4, or 8 cores, so most of the time any
given number will be wrong. I could use a per-system configuration,
but I’d rather not. Unfortunately GNU Make will not automatically
detect the number of cores. That leaves the matter up to Emacs Lisp.
Emacs doesn’t currently have a built-in function that returns the
number of processor cores. I’ll need to reach into the operating
system to figure it out. My usual development environments are Linux,
Windows, and OpenBSD, so my solution should work on each. I’ve ranked
them by order of importance.
Number of cores on Linux
Linux has the /proc virtual filesystem in the fashion of Plan 9,
allowing different aspects of the system to be explored through the
standard filesystem API. The relevant file here is /proc/cpuinfo,
listing useful information about each of the system’s processors. To
get the number of processors, count the number of processor entries in
this file. I’ve wrapped it in if-file-exists so that it returns
nil on other operating systems instead of throwing an error.
When I was first researching how to do this on Windows, I thought I
would need to invoke the wmic command line program and hope the
output could be parsed the same way on different versions of the
operating system and tool. However, it turns out the solution for
Windows is trivial. The environment variable NUMBER_OF_PROCESSORS
gives every process the answer for free. Being an environment
variable, it will need to be parsed.
This seems to work the same across all the BSDs, including OS X,
though I haven’t yet tested it exhaustively. Invoke sysctl, which
returns an undecorated number to be parsed.
When using Emacs, it happens sometimes that I accumulate too many
buffers. The usual next action is to close the buffers for the project
that I'm not currently working on. Previously, I marked those buffers
in *Buffer List* by hand with d and
C-n. Today, I'll show a faster way.
It's very easy to select these buffers in *Buffer List* with
swiper: just type a part the
shared directory name. Another example is to select all dired
buffers: a "dired by" input will usually match all of them, since they
all have Dired by name in their line.
Afterwards, I want to open
multiple-cursors for
each matched line. I've bound this action to C-7. Here's
the very simple code:
After C-7, here's the sequence to delete the selected buffers:
d (Buffer-menu-delete) to mark each item for deletion.
C-g (keyboard-quit) to exit multiple-cursors.
x (Buffer-menu-execute) to execute the deletions.
And that's it: all selected buffers are now deleted. To summarize,
C-sdired byC-7 d C-g x will close all
dired buffers from your *Buffer List*.
Other interesting applications are also possible, like mass renames in
wdired, or un-commenting everything in a function. For example,
suppose that you have this code:
If it's a part of a larger buffer, you can narrow with C-x nd (narrow-to-defun)
or C-x nn (narrow-to-region). Then C-s ;; C-7 DEL DEL C-d C-g to
remove all comments.
I have to say that the functionality intersects a bit with
mc/mark-all-like-this. Except you know the result ahead of time
(number and position of matches), and you can use a regex instead of a
literal string.
The Multiple Cursors package has been given much praise throughout the
Emacsphere. It has a smaller use-case than keyboard macros, but it is usually
quicker and just plain looks awesome. To make full use of its commands, I
combine two of the concepts I've explained here before, rebinding M-number and
intuitive keymaps.
I won’t go into what this package is, as Magnar already has a whole video on
that. Instead, I’ll just explain how I use it.
Firstly, the following keys make the most sense to me.
(require'multiple-cursors-core);; This is globally useful, so it goes under `C-x', and `m';; for "multiple-cursors" is easy to remember.(define-keyctl-x-map"\C-m"#'mc/mark-all-dwim);; Usually, both `C-x C-m' and `C-x RET' invoke the;; `mule-keymap', but that's a waste of keys. Here we put it;; _just_ under `C-x RET'.(define-keyctl-x-map(kbd"<return>")mule-keymap);; Remember `er/expand-region' is bound to M-2!(global-set-key(kbd"M-3")#'mc/mark-next-like-this)(global-set-key(kbd"M-4")#'mc/mark-previous-like-this)
These three commands do most of the hard work. I'll never forget about them, so
their keys don't need to be clever, they just need to be quick. Having
mc/mark-next-like-thisright next to er/expand-region is really the best
place, although now-a-days I use mc/mark-all-dwim almost exclusively.
Something that took me a long time to figure out is that you can unmark stuff
you just marked. Previously, whenever I marked-next-like-this once too many I’d
just abort and start again. It makes sense to bind this to the same keys as
above with the Shift modifier.
;; These vary between keyboards. They're supposed to be;; Shifted versions of the two above.(global-set-key(kbd"M-£")#'mc/unmark-next-like-this)(global-set-key(kbd"M-$")#'mc/unmark-previous-like-this)
On the other hand, this package has a myriad of commands which are extremely
useful on a less-than-daily basis, and this is where we invoke the power of
intuitive key-maps.
(define-prefix-command'endless/mc-map);; C-x m is usually `compose-mail'. Bind it to something;; else if you use this command.(define-keyctl-x-map"m"'endless/mc-map);;; Really really nice!(define-keyendless/mc-map"i"#'mc/insert-numbers)(define-keyendless/mc-map"h"#'mc-hide-unmatched-lines-mode)(define-keyendless/mc-map"a"#'mc/mark-all-like-this);;; Occasionally useful(define-keyendless/mc-map"d"#'mc/mark-all-symbols-like-this-in-defun)(define-keyendless/mc-map"r"#'mc/reverse-regions)(define-keyendless/mc-map"s"#'mc/sort-regions)(define-keyendless/mc-map"l"#'mc/edit-lines)(define-keyendless/mc-map"\C-a"#'mc/edit-beginnings-of-lines)(define-keyendless/mc-map"\C-e"#'mc/edit-ends-of-lines)
Note how easy these keys are to remember. I use mc/insert-numbers barely once
a week, but I never forget it's bound to C-x m i (and when I do use it, it’s a
godsend). Other commands that I rarely use but save me a lot of trouble when I
do are sort/reverse-regions and mc-hide-unmatched-lines-mode.
org-mode has a very useful command, org-time-stamp, which helps you insert
dates from a calendar. So you can quickly type C-c . RET to insert
<2015-10-05 Mon>, for instance. These time-stamps are used by Org in a variety
of ways, so they are wrapped in <> to make them easy to parse. The downside
being that they look less than optimal when exported.
I was bit by this again today while updating the post on donations, and I
finally decided to look for a way to fix it. Of course, org-mode is nothing if
not configurable, so the answer wasn’t very far away.
(add-to-list'org-export-filter-timestamp-functions#'endless/filter-timestamp)(defunendless/filter-timestamp(transback_comm)"Remove <> around time-stamps."(pcaseback((or`jekyll`html)(replace-regexp-in-string"&[lg]t;"""trans))(`latex(replace-regexp-in-string"[<>]"""trans))))
The loyal readers might notice how similar this is to the second lambda we
used for exporting Youtube links. Org is quite consistent in its use of export
filters.
The above is enough to remove the surrounding <>, but we can still make it
better. The YYYY-MM-DD weekday format isn’t commonly used in prose, so let’s
switch that as well.
(setq-defaultorg-display-custom-timest);;; Before you ask: No, removing the <> here doesn't work.(setqorg-time-stamp-custom-formats'("<%d %b %Y>"."<%d/%m/%y %a %H:%M>"))
As a bonus, this format will also be used to display time-stamps in your
org-mode buffers. If don’t want that, you can let-bind the
org-display-custom-times variable when calling the export function, instead of
setting it globally.
If any of this doesn’t work for you, you might need to update your Org
package. Fortunately, Org is on GElpa, so anyone can do that with M-xlist-packages.
In case you encounter files with very long lines, the built-in
M-xvisual-line-mode might come in handy. Although 1MB
long one-line files are a disaster for Emacs, there's no problem
handling smaller files with long lines that result from bad markdown
renderers (notably Github). They way visual-line-mode works, is it
inserts virtual newlines in appropriate positions, so that no line is
truncated. The advantage is that these newlines aren't written to the
file when the buffer is saved (otherwise, a simple M-q
(fill-paragraph) would work too).
Today I've added visual-line-mode support to
swiper, so handling these files
should be even easier now. The change was surprisingly simple: just
two checks for visual-line-mode variable and these basic
replacements:
By the way, visual-line-mode is one of the small number of commands
that I call by name instead of by key chord. But thanks to
counsel-M-x and smex calling this command by name is as short as
C-t v RET. To explain a bit more, here's my setup:
;; make sure to install smex(global-set-key(kbd"C-t")'counsel-M-x)
The default command bound to C-t is transpose-chars - a
highly situational and ultimately counter-productive command
(M-DEL and re-typing the word is faster than surgically
navigating to the transpose point and calling transpose-chars, plus
it trains the hands to type the word correctly next time). Anyway,
C-t is a prime binding for launching interactive commands.
And since smex is installed it remembers that my most used command
that starts with "v" is visual-line-mode - so it's the first one
that gets selected after C-t v. This means that I can just
press RET and be done with it.
Finally, related to visual-line-mode, here's a command I use to
negate M-q (fill-paragraph) (source):
(global-set-key(kbd"C-M-q")'ora-unfill-paragraph)(defunora-unfill-paragraph()"Transform a paragraph into a single line."(interactive)(let((fill-column(point-max)))(fill-paragraphnilt)))
The recent business with is a great example of how quickly things can change (). Other things, like for example, don't change too terribly much over very long periods of time.
This is a continuation of the thought that I started in the
last post a few
weeks ago. I don't fancy myself a graphical designer, but I kind of
have a feeling of what I like - I certainly know it when I see it.
I wasn't quite happy with the color of the current selection in the
minibuffer. Then it hit me that I like the way the Firefox address
bar does it: invert the selected text foreground to white and add a
flashy background. After that, it was only a matter of going though a
few random color permutations in the GIMP color selector to arrive to
a nice state:
I did add something similar for dark background themes, although I
rarely use them myself, since they contrast too much with Firefox,
with most websites having a light background:
(deffaceivy-current-match'((((classcolor)(backgroundlight)):background"#1a4b77":foreground"white")(((classcolor)(backgrounddark)):background"#65a7e2":foreground"black"))"Face used by Ivy for highlighting first match.")
Of course, the standard recommendation follows to use
rainbow-mode in
buffers that deal with colors - it's great.
Finally, counsel-git-grep and counsel-ag now also add the proper
minibuffer coloring if you have it turned on:
(setqivy-display-style'fancy)
If you're interested in how the face backgrounds are combined, you can
read my older article on
color blending in Elisp.
This is one of those small functionalities that makes your life considerably
easier, and yet a surprising number of people don’t know about it. When Emacs
prompts you for something in the minibuffer, you might be aware that you can
navigate back and forth in the prompt’s history with M-p and M-n, but did you
know you can even step into the future?
Well, sort of… At least that’s the spirit. If you type M-p at the prompt,
Emacs fills the prompt with previous items from the history. On the other hand,
if you type M-n, Emacs will try to guess from context what your next input
is going to be. The meaning of this depends on the command (some of these
require Emacs 24.5).
In file prompts, hitting M-n fills in the name of the current file. This is
very useful with write-file or with dired commands like copy or move.
M-x tries to find a command name under point.
In query-replace-regexp it fills in the symbol at point, which is usually a
pretty good guess of what you want to replace. And you can hit M-n a second
time to get \_<symbol-delimiters\_> around the symbol.
As Marco points out in the comments, if you have two dired windows open, file
or directory prompts in one window fill in the directory name of the other
window.
These are the ones I use most often, but I’m sure there are more commands that
support this feature, and it’s always worth trying it out with your favorite
ones.
I've always wanted to create a GIF using Emacs to demonstrate some
features, it just looks so cool. I finally got a chance after
attending the Leeds Code Dojo. The final exercise is bit unusual; we
have to write a basic expression evaluation program without using the
eval function in whatever language we choose. The first problem we
had was to figure out the order of sub-expression to evaluate. For
example, in (5 * (2 + 1) ) expression, we know we firstly add 2 to 1
to get the 3, and then multiply 3 by 5. It sounds trivial but it is
actually hard to write a program to do that.
I used regular expression1 to locate the most inner
expression to evaluate, then replaced the expression with its
evaluating result, and continued these two steps until there was no
expression2.
The above GIF shows each step in a expression evaluation program
written in Emacs Lisp.
This post show how to make GIF in Emacs on Ubuntu system.
Dependencies
There are three packages to install first. We need recordmydesktop
to capture the motion of the screen, mplayer to view the video, and
imagemagic to convert the recorded video into GIF file. They can be
installed easily using the apt-get command, as in the following bash
shell script:
On Emacs side, I use camcorder package to control the
workflow. It is hosted in MELPA repository, and can be installed by
(package-install 'camcorder)
Then everything should work nicely together.
Workflow
After these packages are installed, creating a GIF is simply, only
requiring three steps.
1. Initiate the recording
In Emacs,
Switch to the buffer we want to record, let's call this buffer the
recording buffer,
Initiate the recording by M-x camcorder-record command,
Choose where to save the video file, then
A new frame with the recording buffer will pop up. It is wrapped inside
a white rectangular box. Everything inside the box will be recorded and
saved in the video file. Note, if we move the window or overlay it
with other windows, we probably get undesired results.
2. Record
Choose the recording buffer/frame,
Press F-11 to pause/resume,
Show some cool things,
Press F-12 to stop,
Note the demonstration must have an effect on the recording buffer, and
we can use with-current-buffer function to dump the output for a
particular buffer, for example,
(with-current-buffer "Demo_Buffer"
(insert "Start to demo: "))
will insert "Start to demo: " into the Demo_Buffer.
I found it is useful to wrap the demonstration into a function and
bind to a key because I will probably run it many times.
(defun yt/camcorder-show-off ()
(interactive)
(goto-char (point-min))
(insert "going to show you something cool, don't blink your eyes.")
(sleep-for 2)
;;;; apply some functions
(insert "\nExciting isn't?"))
(define-key camcorder-mode-map [f5] 'yt/camcorder-show-off)
There are two functions that are helpful control the flow. Use
sleep-for function to let the program wait few seconds, and use
y-or-n-p to let us choose whether to proceed or switch flow.
3. Make gif
After the demo is finished,
Type M-x camcorder-convert to convert a video file to a GIF file,
Choose a file name for the GIF file,
Select convert method, and choose use mplay with imagicstick.
We probably repeat the step 1-3 multiple times until we are happy
with the GIF.
This weekend I found myself doing some heavy-weight refactoring in CIDER. This
is the kind of situation where Flycheck helps a lot, but I still needed it to do
a bit more. Every time I made a significant change to a file, I had to visit 3
or 5 other files and trigger Flycheck on each one of them. It wasn’t long before
I decide there had to be a way to just Flycheck a whole directory.
Enter endless/flycheck-dir. This command runs Flycheck on all files in the
current directory and reports the result to the *Compile-Log* buffer. You can
then navigate through all issues for the entire directory by TAB-ing through
the buffer or using next-error.
(define-keyflycheck-command-map"d"#'endless/flycheck-dir)(defunendless/flycheck-dir(dir)"Run flycheck for each file in current directory.
Results are reported in a compilation buffer."(interactive"DDirectory: ")(displaying-byte-compile-warnings(let((pnil))(with-current-buffer(get-buffer-createbyte-compile-log-buffer)(setqdefault-directorydir)(unless(eqmajor-mode'compilation-mode)(compilation-mode))(goto-char(point-max))(let((inhibit-read-onlyt))(insert"\n\xc\n\n"))(setqp(point)))(dolist(file(directory-files"./"nil"\\`[^\\.].*\\'"))(endless/-flycheck-filefile))(with-selected-window(display-bufferbyte-compile-log-buffer)(goto-charp)(recenter1)))))(defunendless/-report-error(fmt&restargs)"Print an error on `byte-compile-log-buffer'."(let((inhibit-read-onlyt)(fill-prefix" "))(with-current-bufferbyte-compile-log-buffer(let((l(point)))(insert"\n"(apply#'formatfmtargs))(fill-region(1+l)(point))))))(defunendless/-flycheck-file(file)"Check FILE and report to `byte-compile-log-buffer'."(let((was-visited(find-buffer-visitingfile)))(with-current-buffer(orwas-visited(progn(find-filefile)(current-buffer)))(when(ignore-errors(flycheck-buffer))(while(flycheck-running-p)(accept-process-outputnil0.1))(pcaseflycheck-last-status-change((or`errored`suspicious)(endless/-report-error"%s: Something wrong here!"(file-name-nondirectory(buffer-file-name))))(`finished(dolist(eflycheck-current-errors)(endless/-report-error"%s:%s:%s:%s: %s"(file-name-nondirectory(buffer-file-name))(flycheck-error-linee)(flycheck-error-columne)(flycheck-error-levele)(flycheck-error-messagee))))))(ifwas-visited(bury-bufferwas-visited)(kill-buffer(current-buffer))))))
If you’ve taken the time to browse some Elisp source files, you’ve no doubt run
into that odd little ^L, a.k.a. the form feed character. Emacs uses these
white space characters as page delimiters. This makes for a very convenient way
to split a file into sections, and quickly navigate between them. I won’t go too
deep into them, as Eric James has already written a great crash course on pages
that you should go check out.
What I wanted to write about is the way that I do page navigation in Emacs.
Firstly, I find the default keys to be nothing short of abhorrent. Take a prefix
key with the Control modifier, attach to it a non-modified key, and then make that key
be something not-so-easy to hit, like ], and you have the recipe for painful
fingers.
These keys would normally be bound to set-goal-column and mark-page, which
I’ve never ever ever used (in fact, the former is disabled by default).
Then there’s a minor peeve. In some corner cases Emacs might leave the cursor at
the bottom of the screen after moving. Here we make sure that never happens.
(defunendless/-recenter-advice(&rest_)"Recenter to page start."(when(called-interactively-p'any)(recenter5)));; Requires Emacs 24.5(advice-add#'backward-page:after#'endless/-recenter-advice)(advice-add#'forward-page:after#'endless/-recenter-advice)
And then there’s the best part. It turns out you don’t need the form-feed
character to delimit pages. That’s important because some languages aren’t that
nice about them, and some dev teams might prefer you don’t stick those
^L all over the place. In Clojure, for instance, cljfmt confuses it for a
blank line and freaks out a little.
Fortunately, Elisp style already recommends using ;;; to indicate comment
sections, and the form feed character is most commonly placed right above these
sections. So why not use that instead?
The regexp above is a bit special. We’re setting the page delimiter to be a
;;; at the start of a line, plus any number of empty lines or comment lines
that follow it (that # part is to exclude ;;;###autoload cookies).
Consequently, when we hit C-x C-n or C-x C-p, the point is left right at the
start of the first code-line after the ;;;. That’s usually where I want to be,
and it works even on buffers without ^L, Clojure and Elisp. No doubt you can
extended that to your programming language of choice by replacing the semicolons
with the appropriate comment character.
Even better, why not write up a general solution based on the comment-start
variable?
thoughtbot NYC
20th floor
1384 Broadway
New York, NY
As usual, we’ll be starting at 6:30 with pizza and beer!
Jay Dixit website will be talking about being a writer using emacs.
Jay Dixit is a science writer whose work has appeared in The New York Times, Rolling Stone, and Psychology Today. Jay will discuss how he uses Emacs as a non-programmer, and how Emacs has made him a more productive writer, editor, and researcher.
Every computer user, to some extend, is a user of Open Source Software (even if
most of them are oblivious to that). This is only possible because the
developers of these pieces of software have donated their time to us, who are
nothing short of complete strangers to them. These are regular people, with just
as much free time as you or I—sometimes a bit more, sometimes even less.
There are many ways we can thank them for this. Just saying “thank you” is one
way, anyone likes to be appreciated and would be happy to hear that they did
something that helped you. Another way is to spread the word, share the
software, and thus help them help more people. A third option is to help back,
send a patch or a PR, write documentation, or even write your own software to
help your own complete strangers, adding one more link to this selfless chain.
And lastly, you can also donate.
A few weeks ago, I setup a Gratipay page for Endless Parentheses. If you’d like
to help, you can donate to the blog or to my packages by clicking on the button
below.
But that was weeks ago, it was something else that prompted me to write this
post today. Over the last four days, two of Emacs most popular packages have
setup new donation channels, so I saw this as a chance to group all these links
together into a post.
CIDER now has a page on Salt (Bountysource’s new monthly donations scheme), and
Magit has got its Gratipay page up and running again after some down time. Both
of these are true feats of Emacs development, way beyond your usual weekend
programming exercise, so consider donating if they make your life easier. They
also offer several other channels, so see these links to their respective
manuals.
Bastien Guerry points out that Org-mode now has Gratipay page as well. You can
also donate to it via Paypal on the website (top-right corner). I shouldn’t have
to tell you that this blog probably wouldn’t exist without org-mode!
Two more huge and important Emacs projects I forgot to mention before are
Flycheck and Spacemacs, and you can find their Paypal links on their respective
Readmes.
Nameless is an Emacs package for hiding namespace prefixes in elisp code. It
is a short and simple minor-mode that changes the display, without changing the
contents of the buffer. Using it is as simple as turning it on, there’s no need
to change your package in any way.
Nine months ago I introduced Names, a package that allows you to write elisp
code inside a namespace. Since then, I’ve used it on two of my packages without
issues (as have several other Emacs hackers) and it’s been delightful. I’ve been
hacking elisp for several years now, but stripping away all those namespace
prefixes really makes it more pleasurable to write and read code.
Still, there’s no denying that it’s a bit heavy-handed. It’s quite a large
library that your package has to depend on, and it changes the way you write the
file itself, which means it messes with anything from grep to find-function.
Nameless, on the other hand, hooks onto font-lock-mode and changes the way
symbols are displayed. This means it doesn’t interfere with any other tools you
might use for code management or navigation. At the end of the day, it tries a
lot less and therefore does a lot more.
Nameless is available from GNU Elpa. To give it a try, just M-xpackage-install it, and turn on the minor-mode wherever you want.
My MacBookPro's hard drive stooped working last week and I managed to
recover most of the data from a Time Machine back-up 6 months ago. But
I couldn't get the mu4e and mu working. I feed up with googling,
trying, and decide to immigrate to Ubuntu. It would save me from a
lot of frustrations and time in making my Mac and office PC work the same
way.
Ideally, I will built a Ubuntu on Mac which is exactly the same as the
one on my office PC, by just copy over everything 1. As a minimalist, I
decided to build the system from scratch and install software one by
one so that I can have an better understanding of what are the
necessities for me.
In the last few days, I become extra mindful about the what and how I
used the Ubuntu system in the office, and realise the things I need
can be grouped into three categories:
Configuration,
the .ssh folder for the ssh-agent,
the .fonts folder for new fonts,
the .mbsynrc file for sync emails,
the .ledgerrc.
Software for
Development: like git, gcc, Emacs, and R.
Writing: org-mode, LaTeX,
Email: mu, mu4e, and mbsync.
Finance: ledger.
Personal git repositories
public reposity on GitHub,
private reposities on BitBucket
For 1), since they are small, I can zip up and copy over, or even
better, create a git repository so that sync on two machines becomes better
easier.
For 2), I need to find the software's package name in the Ubuntu's
software repository, and then install all of them by a script. The
dependencies should be resolved automatically.
For 3), I need to create a shared folder between the host system and the
Ubuntu system, and then copy over the ~/git/ folder.
It really sounds like a plan! I am going to download the Ubuntu
installation file now and hopefully the transition will be very smooth.
If you haven't used it before, counsel-M-x is part the counsel
package on MELPA, or part of swiper package if you're installing from
GNU ELPA. Basically, it's a M-x replacement (that I like to
bind to C-t for efficiency reasons), that doubles as
find-function (just press C-. instead of RET
to find the function instead of calling it). If you're using
counsel-M-x I highly recommend to also install smex, since then
counsel-M-x will use smex for sorting the matched commands.
The update will propertize the minibuffer contents with a new set of faces:
(defcustomswiper-minibuffer-faces'(swiper-minibuffer-match-face-1swiper-minibuffer-match-face-2swiper-minibuffer-match-face-3swiper-minibuffer-match-face-4)"List of `swiper' faces for minibuffer group matches.")
Initially, when responding to
#212, I used the
original swiper faces in the minibuffer as well. But after some use,
their brightness became a bit annoying. So I introduced a new set of
faces that can be customized separately.
It's similar to what the Firefox address bar uses: gray background and
bold font. Additionally, the matching parts are highlighted with pale
gray, blue and magenta colors. If you like the color scheme, it's part
of eclipse-theme. Another
choice, for even less distraction, could be to remove the background
from these faces, only leaving the bold part on the matches.
Thanks to @Wilfred for nudging me to
finally implement this feature. It's been on my list for a long time,
but I've been putting it off.
As you grow accustomed to fine-tuning your Emacs experience, it’s not unusual to
start using local variables in your files. These are specified as comment lines
at the end of the file, and are extremely practical in a number of scenarios. Here’s
a very simple org file.
* Some headline
Ramblings no one cares about.
* Another headline
Thoughtfully rescinded.
# Local Variables:
# fill-column: 666
# End:
The only problem is that org-mode thinks that everything after a headline is
part of its contents, which is clearly not the case here. One example where this
is bad is when you want to shuffle the headlines around with M-↓ or M-↑, and
you end up with something like this.
* Another headline
Thoughtfully rescinded.
# Local Variables:
# fill-column: 666
# End:
* Some headline
Ramblings no one cares about.
I asked about that on Emacs.StackExchange and got a very simple solution: just
add a “dummy” level-1 headline before the local variable specification.
* Some headline
Ramblings no one cares about.
* Another headline
Thoughtfully rescinded.
* COMMENT Footer
# Local Variables:
# fill-column: 42
# End:
Now you can move, archive, and refile all your headlines as you wish, without
fear of destroying your precious variables. You can even fine-tune the
visibility to folded, so that the footer is always folded, and you won’t have
to see those variables at all.
But this wouldn’t be our weekly Endless Parentheses if I didn’t give you some
code to practice your elisp. The following command is just like end-of-buffer,
except it places point before the footer. If invoked again, it will move to the
real end of buffer.
(defunendless/org-eob()"Move to end of content, then end of buffer."(interactive)(unless(use-region-p)(push-mark))(if(looking-at-p"\n\\* COMMENT Footer")(goto-char(point-max))(goto-char(point-min))(if(search-forward"\n* COMMENT Footer"nil'noerror)(goto-char(match-beginning0))(goto-char(point-max)))))(define-keyorg-mode-map[remapend-of-buffer]#'endless/org-eob)
I had a fascination with Python at one point, until I got too annoyed
with the indentation rules. I still have a few scripts left over, so
I'm sympathetic to Python support in Emacs. The reason for
today's post comes from
this question
on Emacs StackExchange. Essentially, the user wants to insert parens
automatically when a function name is completed.
The completion candidates come from
jedi plugin. To quickly see
where the list of strings is coming from, I've examined ac-sources
variable and saw that it contains ac-source-jedi-direct, which evaluates to:
This means that jedi:complete-request has to be called at point,
followed by jedi:ac-direct-matches to obtain the list of strings. So
basically, this code:
Note the call to deferred:sync!. I had to add that one to make sure
that jedi:complete-request completes before jedi:ac-direct-matches
is called.
Testing with some Python code, where | is the point:
params={"foo":"bar"}params.i
The Elisp code above will return:
(#("items"05(summary"function: __builtin__.dict.items"symbol"f"document"items(self)D.items() -> list of D's (key, value) pairs, as 2-tuples"))#("iteritems"09(summary"function: __builtin__.dict.iteritems"symbol"f"document"iteritems(self)D.iteritems() -> an iterator over the (key, value) items of D"))#("iterkeys"08(summary"function: __builtin__.dict.iterkeys"symbol"f"document"iterkeys(self)D.iterkeys() -> an iterator over the keys of D"))#("itervalues"010(summary"function: __builtin__.dict.itervalues"symbol"f"document"itervalues(self)D.itervalues() -> an iterator over the values of D")))
So these strings come with the symbol documentation and symbol type
encoded as string properties. After this, the rest of the Elisp code
follows very easily, you can find it as part of
counsel
package on MELPA:
(defuncounsel-jedi()"Python completion at point."(interactive)(let((bnd(bounds-of-thing-at-point'symbol)))(ifbnd(progn(setqcounsel-completion-beg(carbnd))(setqcounsel-completion-end(cdrbnd)))(setqcounsel-completion-begnil)(setqcounsel-completion-endnil)))(deferred:sync!(jedi:complete-request))(ivy-read"Symbol name: "(jedi:ac-direct-matches):action#'counsel--py-action))(defuncounsel--py-action(symbol)"Insert SYMBOL, erasing the previous one."(when(stringpsymbol)(with-ivy-window(whencounsel-completion-beg(delete-regioncounsel-completion-begcounsel-completion-end))(setqcounsel-completion-beg(move-marker(make-marker)(point)))(insertsymbol)(setqcounsel-completion-end(move-marker(make-marker)(point)))(when(equal(get-text-property0'symbolsymbol)"f")(insert"()")(setqcounsel-completion-end(move-marker(make-marker)(point)))(backward-char1)))))
Essentially, the last interesting part is (equal (get-text-property 0
'symbol symbol) "f") which test if the string corresponds to a
function or not. The rest of the code just fiddles with symbol markers
with ensure that the previous symbol is erased before the new symbol
is inserted if you press C-M-n (ivy-next-line-and-call),
or use ivy-resume. Just to describe how this would be useful for the
Python code above: I call counsel-jedi, followed by C-M-n
to get:
params={"foo":"bar"}params.iteritems(|)
Pressing C-M-n again will result in:
params={"foo":"bar"}params.iterkeys(|)
Once I'm satisfied with my selected candidate, I can press either
C-m or C-j or C-g. Most of the above
code can be used almost verbatim if you're a Helm fan and want to
implement helm-jedi. Basically, the approach I described (going
through ac-sources) can be used to implement alternative completion
in a lot of cases where auto-complete-mode completion is already
available.
Finally, if you like the idea of auto-inserting parens with completion
and are using C/C++, have a look at
function-args - this
package does that too, with Ivy, Ido and Helm as available back ends.
If you’ve every tried to do some spell-checking in org-mode you know how
finicky that can be. Ispell is happy to check absolutely anything, even code
blocks and property drawers! When you’re blogging about code-snippets from an
org file this annoyance quickly turns into irritation. Here’s how you fix it.
(defunendless/org-ispell()"Configure `ispell-skip-region-alist' for `org-mode'."(make-local-variable'ispell-skip-region-alist)(add-to-list'ispell-skip-region-alist'(org-property-drawer-re))(add-to-list'ispell-skip-region-alist'("~""~"))(add-to-list'ispell-skip-region-alist'("=""="))(add-to-list'ispell-skip-region-alist'("^#\\+BEGIN_SRC"."^#\\+END_SRC")))(add-hook'org-mode-hook#'endless/org-ispell)
Commenting is a very frequent piece of a programmer’s workflow, and it’s
important to make it seamless and simple. For the more statemental languages,
that’s as easy as writing a custom comment-line command. However, when
you’re writing in Lisp languages, that just won’t do. Trying to comment out
lines in a sexp-oriented structure, feels a lot like trying to hit a nail with a
heavy screwdriver—it sometimes gets the job done, but it mostly just leads to
frustration.
That said, a comment-sexp command is considerably more complicated to write.
Not because commenting sexps is hard, but because it is quite difficult to
identify sexps when removing comments. Still, I’m nothing if not stubborn. So
after much hair pulling and teeth gritting, I have finally found a version I’m
happy with.
The gif above speaks for itself, so I’ll just give you the code and let you play
with it.
(defununcomment-sexp(&optionaln)"Uncomment a sexp around point."(interactive"P")(let*((initial-point(point-marker))(inhibit-field-text-motiont)(p)(end(save-excursion(when(elt(syntax-ppss)4)(re-search-backwardcomment-start-skip(line-beginning-position)t))(setqp(point-marker))(comment-forward(point-max))(point-marker)))(beg(save-excursion(forward-line0)(while(and(not(bobp))(=end(save-excursion(comment-forward(point-max))(point))))(forward-line-1))(goto-char(line-end-position))(re-search-backwardcomment-start-skip(line-beginning-position)t)(ignore-errors(while(looking-at-pcomment-start-skip)(forward-char-1)))(point-marker))))(unless(=begend)(uncomment-regionbegend)(goto-charp);; Indentify the "top-level" sexp inside the comment.(while(and(ignore-errors(backward-up-list)t)(>=(point)beg))(skip-chars-backward(rx(syntaxexpression-prefix)))(setqp(point-marker)));; Re-comment everything before it. (ignore-errors(comment-regionbegp));; And everything after it.(goto-charp)(forward-sexp(orn1))(skip-chars-forward"\r\n[:blank:]")(if(<(point)end)(ignore-errors(comment-region(point)end));; If this is a closing delimiter, pull it up.(goto-charend)(skip-chars-forward"\r\n[:blank:]")(when(eq5(car(syntax-after(point))))(delete-indentation))));; Without a prefix, it's more useful to leave point where;; it was.(unlessn(goto-charinitial-point))))(defuncomment-sexp--raw()"Comment the sexp at point or ahead of point."(pcase(or(bounds-of-thing-at-point'sexp)(save-excursion(skip-chars-forward"\r\n[:blank:]")(bounds-of-thing-at-point'sexp)))(`(,l.,r)(goto-charr)(skip-chars-forward"\r\n[:blank:]")(save-excursion(comment-regionlr))(skip-chars-forward"\r\n[:blank:]"))))(defuncomment-or-uncomment-sexp(&optionaln)"Comment the sexp at point and move past it.
If already inside (or before) a comment, uncomment instead.
With a prefix argument N, (un)comment that many sexps."(interactive"P")(if(or(elt(syntax-ppss)4)(<(save-excursion(skip-chars-forward"\r\n[:blank:]")(point))(save-excursion(comment-forward1)(point))))(uncomment-sexpn)(dotimes(_(orn1))(comment-sexp--raw))))
I'd like to highlight a new command added to
ivy-mode today:
ivy-kill-ring-save. It allows to store all current candidates to
the kill ring. This could have a number of uses.
Scenario 1
Suppose you want to learn some Elisp, specifically all functions that
start with forward-. Then call counsel-describe-function with
"forward-" as input and press M-w. Then just yank this into
some buffer, org-mode for instance, and go through the functions
one-by-one takings notes on the way:
Suppose you want to store a subset of projectile-find-file names
that match a pattern. Just press M-w and those file names
will be on the top of the kill ring. Of course, you need to have this setting on:
(setqprojectile-completion-system'ivy)
This scenario should apply to basically any place where you're
completing file names, like counsel-locate or
counsel-find-file. Also
find-file-in-project,
which uses ivy for completion if it's installed.
Outro
Note also that the command comes for free, since normally nothing is
done by M-w when the region isn't active. When the region
is active, this command will store the selected text instead of
matched candidates.
At times, I need to use the same link at multiple places in a long document. For
those cases it would be useful to have link IDs like in markdown.
This is [an example][some-id] reference-style link.
Then, anywhere in the document, you define your link label like this, on a line by itself:
[some-id]: http://example.com/
The solution I came up with defines three different functions, so I’ll let you
follow the link instead of clogging up this blog post. The result is that you
can define link IDs like this anywhere in your document (probably the top or the
bottom).
#+LINK-ID: wiki http://www.emacswiki.org
And then you can use that URL in links throughout the document by writing them
like this.
Here is a [[lid:wiki][wiki link]], and here
is [[lid:wiki][another one]].
For those who don't keep up,
ivy-mode is a completion method
that's similar to ido, but with emphasis on simplicity and
customizability. Currently, there are two related packages on MELPA:
swiper and counsel:
swiper provides an isearch replacement, using ivy-read for completion, as well as the basic ivy-mode.
counsel provides some extra commands, like -M-x, -ag, -load-theme etc.
The reasoning behind the split into two packages is that there's less
update overhead if only one of them is updated.
completing-read-function conundrum
This is to explain a bit the presence of the counsel
package. Initially, I hoped that ivy-mode could do most of the work,
via the Emacs' completing-read-function interface. How it works:
most built-in packages will call completing-read when they require
completion. That function will forward the completion to
completing-read-function if it's set. This is the way how things
like icomplete-mode or ivy-mode or helm-mode work.
Unfortunately, the interface rather limits what you can do in
completing-read-function. Essentially, you're given a list of
strings and you have to return a string. You have no idea which
function called you, so you can't do any fancy stuff depending on the
caller, short of examining this-command which isn't very reliable.
You have no idea what will be done with the one string that you
return, so you can't do something fancy like select two or three
strings and perform that action for each of them.
The result is that sometimes I have to replace the built-in functions,
instead of re-using them. So instead of re-using find-file, I wrote
my own counsel-find-file that looks like this:
(defuncounsel-find-file()"Forward to `find-file'."(interactive)(ivy-read"Find file: "'read-file-name-internal:matcher#'counsel--find-file-matcher:action(lambda(x)(with-ivy-window(find-file(expand-file-namexivy--directory)))):preselect(whencounsel-find-file-at-point(require'ffap)(ffap-guesser)):require-match'confirm-after-completion:history'file-name-history:keymapcounsel-find-file-map))
It's still possible to call the original find-file, and you'll get
ivy-read completion for it, but:
ivy-resume won't work, since ivy-read doesn't know what to do with the string that you select.
C-M-n (ivy-next-line-and-call) won't work to select another file within the same completion session, for the same reason.
M-o (ivy-dispatching-done) may work with some
customization, but since it dispatches on this-command to get the
extra actions, it can cause a problem if the command wasn't called
directly.
I think it would be cool to extend the built-in
completing-read-function interface in the future Emacs
versions. Both Ivy and Helm (which uses the same strategy of passing
the action to the completion) would benefit from this.
Release summary and highlights
It's been two months and 150 commits since the last version. The full
release notes can be found inside the repository in
Changelog.org
or at Github's
release tab.
I recommend to look through the whole list to see if anything catches your attention.
I'll highlight a few things that I think the users might be interested in most.
Fuzzy completion
This is off by default, since I think having less candidates is better than having more.
You can customize this for all commands or per-command. For example:
You can always use M-o o to access the default action. When there is
only one action, M-o does the same as C-m.
More descriptions with a built-in Hydra
Toggle C-o in any completion session to get an overview of
the things that you can do.
Many commands that interface with shell calls
The following commands will call the appropriate shell tool to give you a list of candidates:
counsel-git-grep
counsel-ag
counsel-locate
counsel-recoll
These commands will refresh after each new letter entered, and they
account for the fact that the shell call will usually take more time
than the time it takes to input a new letter. So the process will be
killed and restarted, resulting in virtually no keyboard delay and no
downtime.
Many commands that offer more options than the built-in counterparts
This list includes:
counsel-find-file,
counsel-M-x,
counsel-load-theme,
counsel-org-tag and counsel-org-tag-agenda.
Most extra things that you can do are:
Doing stuff with multiple candidates through C-M-n like in this video.
Using ivy-resume to resume the last completion session.
Make sure to try counsel-org-tag, this one is a bit tricky for
completion, since it requires to return multiple candidates at once.
So you can toggle each tag with C-M-m (ivy-call) and then
exit with C-m (ivy-done). The currently selected tags
will be displayed in the prompt.
Outro
A big thanks to all people who contributed code and issues towards
this release. I'm really happy that you're helping me to refine this
nice package.
Transposing is another of those features that I really miss when not in Emacs.
It took me several months of actively reminding myself in order to finally
incorporate it into my regular arsenal. Now, not a day goes by that I don’t
transpose a few lines, and usually some words and sexps as well, but the
usefulness of transpose-char still seems to elude me.
I hear this command is great at fixing some typos, so perhaps I don’t find it as
useful because auto-correct takes care of those for me. Meanwhile, the other
transpose commands have the very different role of refactoring code or moving
text around (transpose-lines is specially nice with one-line-per-sentence text).
Whatever the reason, the point is that C-t is too good a key for a command
that I’m not going to use and C-x C-t is too long for a command I use so
often. The answer, of course, is to swap the two around.
If you’re a frequent reader, no doubt you noticed an embedded Youtube video on a
post a couple of weeks ago. Youtube makes it pretty simple to embed videos, they
give you the entire iframe HTML code to use, but this wouldn’t really be Emacs
if we couldn’t make things just a little bit easier. Just add the snippet below
to your init file, and you’re good to go.
(defvaryt-iframe-format;; You may want to change your width and height.(concat"<iframe width=\"440\""" height=\"335\""" src=\"https://www.youtube.com/embed/%s\""" frameborder=\"0\""" allowfullscreen>%s</iframe>"))(org-add-link-type"yt"(lambda(handle)(browse-url(concat"https://www.youtube.com/embed/"handle)))(lambda(pathdescbackend)(cl-casebackend(html(formatyt-iframe-formatpath(ordesc"")))(latex(format"\href{%s}{%s}"path(ordesc"video"))))))
To use this, just write your org links in the following way (optionally adding a description).
[[yt:A3JAlWM8qRM]]
When you export to HTML, this will produce that same inlined snippet that
Youtube specifies. The advantage (over simply writing out the iframe) is that
this link can be clicked in org-mode, and can be exported to other formats as
well.
I know that most Emacs hackers love the simplicity and usability of
grep, but sometimes it just doesn't cut it. A specific use case is my
Org-mode directory, which includes a lot of org files and PDF
files. There are just too many files for grep to be efficient, plus
the structure of PDF doesn't lend itself to grep, so another tool is
required: a desktop database.
I got into the topic by reading John Kitchin's post on
swish-e,
however, I just couldn't get that software to work. But as a reply to
his post, another tool -
recoll was mentioned on
Org-mode's mailing list. In this post, I'll give step-by-step
instructions to make Recoll work with Emacs.
Building Recoll
I'm assuming that you're on a GNU/Linux system, since it's my
impression is that it's the easiest system for building (as in make
...) software. Also, it's the only system that I've got, so it would
be hard for me to explain other systems.
If you want to toy around with the graphical back end of Recoll, you can install it with:
sudo apt-get install recoll
Unfortunately, the shell tool recollq isn't bundled with that
package, so we need to download
the sources.
The current version is 1.20.6.
Extract the archive
After downloading the archive, I open ~/Downloads in dired and
press & (dired-do-async-shell-command). It guesses from
the tar.gz extension that the command should be tar zxvf. By
pressing RET, I have the archive extracted to the current
directory. I've actually allocated ~/Software/ for installing stuff
from tarballs, since I don't want to put too much stuff in ~/Downloads.
Open ansi-term
I navigate to the recoll-1.20.6/ directory using dired, then press
` to open an *ansi-term* buffer for the current
directory.
(defunora-terminal()"Switch to terminal. Launch if nonexistent."(interactive)(if(get-buffer"*ansi-term*")(switch-to-buffer"*ansi-term*")(ansi-term"/bin/bash"))(get-buffer-process"*ansi-term*"))(defunora-dired-open-term()"Open an `ansi-term' that corresponds to current directory."(interactive)(let((current-dir(dired-current-directory)))(term-send-string(ora-terminal)(if(file-remote-pcurrent-dir)(let((v(tramp-dissect-file-namecurrent-dirt)))(format"ssh %s@%s\n"(arefv1)(arefv2)))(format"cd '%s'\n"current-dir)))(setqdefault-directorycurrent-dir)))(define-keydired-mode-map(kbd"`")'ora-dired-open-term)
Configure and make
Here's a typical sequence of shell commands.
./configure && make
sudo make install
cd query && make
which recoll
sudo cp recollq /usr/local/bin/
I was a total Linux newbie 5 years ago and had no idea about shell
commands. Using only the first two lines, you can build and install a
huge amount of software, so these are a great place to start if you
want to learn these tools. I actually got by using only those two
lines for a year a so.
After the first run or ./configure it turned out that I was missing
one library, so I had to do this one and redo the ./configure step.
sudo apt-get install libqt5webkit5-dev
I think this one would work as well:
sudo apt-get build-dep recoll
Configuring Recoll
I only launched the graphical interface to select the indexing
directory. It's the home directory by default, I didn't want that so I
chose ~/Dropbox/org/ instead. Apparently, there's a way to make the
indexing automatic via a cron job; you can even configure it via the
graphical interface: it's all good.
Using Recoll from Emacs
Emacs has great options for processing output from a shell command. So
first I had to figure out how the shell command should look like. This
should be good enough to produce a list of indexed files that contain the word "Haskell":
recollq -b 'haskell'
And there's how to adapt that command to the asynchronous ivy-read interface:
(defuncounsel-recoll-function(string&rest_unused)"Issue recallq for STRING."(if(<(lengthstring)3)(counsel-more-chars3)(counsel--async-command(format"recollq -b '%s'"string))nil))(defuncounsel-recoll(&optionalinitial-input)"Search for a string in the recoll database.You'll be given a list of files that match.Selecting a file will launch `swiper' for that file.INITIAL-INPUT can be given as the initial minibuffer input."(interactive)(ivy-read"recoll: "'counsel-recoll-function:initial-inputinitial-input:dynamic-collectiont:history'counsel-git-grep-history:action(lambda(x)(when(string-match"file://\\(.*\\)\\'"x)(let((file-name(match-string1x)))(find-filefile-name)(unless(string-match"pdf$"x)(swiperivy-text)))))))
The code here is pretty simple:
I don't start a search until at least 3 chars are entered, in order to not get too many results.
I mention :dynamic-collection t which means that recollq should be called after each new letter entered.
In :action, I specify to open the selected file and start a swiper with the current input in that file.
Outro
I hope you found this info useful. It's certainly pretty cool:
cd ~/Dropbox/org && du -hs
# 567M .
So there's half of a gigabyte of stuff, all of it indexed, and I'm
getting a file list update after each new key press in Emacs.
If you know of a better tool than recoll (I'm not too happy that match
context that it gives via the -A command option), please do share.
Also, I've just learned that there's
helm-recoll out there, so
you can use that if you like Helm.
So many Emacs commands and bindings, but little time to learn them
all. That's why many-in-one solutions like
hydra are often a good deal: you
only remember the base binding, and get more info along the way if you
get lost.
I've wanted to optimize this approach when it comes to completion as
well. In
last month's post
I described how you can get help / jump to definition / jump to info
for the current F1 v (counsel-describe-variable)
candidate with C-m / C-. / C-,.
While that approach can be viable, it has a few problems. The first
problem is that it's not discoverable: you may be using C-m
option for ages and not know that C-. and C-,
exist. The second problem is that it's not extensible: if you wanted
more actions, there are hardly any good bindings left in the
minibuffer to bind those actions.
In
a more recent post,
I described a solution that alleviates both problems. When you press
C-o, you see how many actions are
available, and you can cycle them with w and
s. However, while discoverable, this approach is cumbersome
if you already know what you want to do. Especially cycling can't be
great when there are a lot of actions.
So today I've added a new approach in addition to the previous one.
When you press M-o (ivy-dispatching-done), you get a hint
showing you which actions are available. It's like C-m
(ivy-done), only allows you to quickly select the
action. Conveniently, the default action is bound to o, so
M-o o is equivalent to C-m.
Here's how I add one more action to the default one of ivy-switch-buffer:
If you're familiar with hydra, the format is the same: (key cmd
hint). This approach is very extensible: you can add actions and
bindings to commands without changing the code of the original
command. It's really fast: you're typing just one extra key to select
an action. And you'll hardly run out of keys to bind, with 25 lower
case keys at your disposal (other bindings work as well, I just think
that lower case keys are the fastest to press).
In case you wonder what (ivy--reset-state ivy-last) does, it's used
to update the list of buffers, since one of them was deleted. This
way, you can delete e.g. 4 buffers with C-x b C-o s gggg,
and have the buffer list update after each g.
New functionality in action
Let's look at one package that has a multitude of actions available
for each candidate - projectile:
(defvarhelm-projectile-projects-map(let((map(make-sparse-keymap)))(set-keymap-parentmaphelm-map)(helm-projectile-define-keymap(kbd"C-d")#'dired(kbd"M-g")#'helm-projectile-vc(kbd"M-e")#'helm-projectile-switch-to-eshell(kbd"C-s")#'helm-find-files-grep(kbd"M-c")#'helm-projectile-compile-project(kbd"M-t")#'helm-projectile-test-project(kbd"M-r")#'helm-projectile-run-project(kbd"M-D")#'helm-projectile-remove-known-project)map)"Mapping for known projectile projects.")
Here's the basic Ivy function for selecting a project:
(defunivy-switch-project()(interactive)(ivy-read"Switch to project: "(if(projectile-project-p)(cons(abbreviate-file-name(projectile-project-root))(projectile-relevant-known-projects))projectile-known-projects):action#'projectile-switch-project-by-name))(global-set-key(kbd"C-c m")'ivy-switch-project)
And now let's add all those actions:
(ivy-set-actions'ivy-switch-project'(("d"dired"Open Dired in project's directory")("v"helm-projectile-vc"Open project root in vc-dir or magit")("e"helm-projectile-switch-to-eshell"Switch to Eshell")("g"(lambda(x)(helm-do-grep-1(listx)))"Grep in projects")("c"helm-projectile-compile-project"Compile project")("r"helm-projectile-remove-known-project"Remove project(s)")))
Here's what I get now after pressing M-o:
Looks pretty good, I think. I hope you find the new approach useful.
The multi-action exit is currently enabled by default for
ivy-switch-buffer, counsel-locate and counsel-rhythmbox, but you
can add actions yourself to almost any command that uses ivy-read.
In the rare case when ivy-read isn't in the tail position, you can
use ivy-quit-and-run
inside the added action functions.
This is something that’s bothered me for a very long time. My pinky is slow when
it comes to releasing the Shift key, and frequently leads to typos. MOst typos
(hitting letters in the wrong order) are already covered by auto-correction, but
there’s another common typo that it doesn’t fix. EVery now and then, I’ll start
a sentence with two uppercase letters.
After the billionth time that it happened to me, I finally took the time to
write up a question on Emacs.StackExchange. The answers were fast, and so high
quality that I don’t even have improvements to suggest. Instead, I’ll just point
you straight to Dan's answer, and show how I activate it.
Today I'll introduce a pretty cool new option that you can use to
quickly create hydra docstrings arranged in a matrix. This, of
course, was already possible before, but you had to format the matrix
by-hand like this (I cite just bit instead of the
full code, it was
cut to fit into the Jekyll code column restriction):
The syntax above is still very useful if you want to arrange every
detail perfectly or need Ruby-style quoting. But here's the new
syntax that allows to get almost the
same docstring
much easier:
(defhydrahydra-projectile(:colorblue:columns4)"Projectile"("a"projectile-ag"ag")("b"projectile-switch-to-buffer"switch to buffer")("c"projectile-invalidate-cache"cache clear")("d"projectile-find-dir"dir")("s-f"projectile-find-file"file")("ff"projectile-find-file-dwim"file dwim")("fd"projectile-find-file-in-directory"file curr dir")("g"ggtags-update-tags"update gtags")("i"projectile-ibuffer"Ibuffer")("K"projectile-kill-buffers"Kill all buffers")("o"projectile-multi-occur"multi-occur")("p"projectile-switch-project"switch")("r"projectile-recentf"recent file")("x"projectile-remove-known-project"remove known")("X"projectile-cleanup-known-projects"cleanup non-existing")("z"projectile-cache-current-file"cache current")("q"nil"cancel"))
It still looks pretty good, in my opinion:
Note how all redundant information has been removed. Another advantage
is the you can easily:
Add/remove heads.
Update head hints.
Change the number of columns.
A bit of customization
After each change, just re-eval the defhydra and you're done - no
need to manually re-arrange the docstring. Since there already is an
interface that allows you to customize the docstring very precisely, I
tried to make the new interface as automatic as possible: just specify
the number of columns and you're done. But if you want just a little
bit more customization, look at this code:
(defvarhydra-key-doc-function'hydra-key-doc-function-default"The function for formatting key-doc pairs.")(defunhydra-key-doc-function-default(keykey-widthdocdoc-width)(format(format"%%%ds: %%%ds"key-width(--1doc-width))keydoc))
So if you want to add a little space here and there, or e.g. wrap key
bindings in brackets, just clone hydra-key-doc-function-default,
customize it a bit and set hydra-key-doc-function to the new
function.
Outro
If you've been using just the plain style before and felt that you
can't fit your docstring on one line, try the new style: it's just one
line of change with respect to your old code. And if you've been
building the docstring by-hand, see if it's possible to transform it
to the new style: adding and removing new heads will become easier.
Thanks to @Fuco1 for the suggestion in
#140.
Install JDEE! Ok, maybe that’s an overstatement. JDEE is far from simple, and it
hasn’t been able to keep up very well since Java 1.4. However, thanks to Stephen
Leake & folks, that might be starting to change. JDEE is now on Github, and it
could definitely use your help.
Stephen has already been updating the code base for the recent Emacs versions,
and other people have shown interest in this as well. There are already some
productive discussions going on in the issues, and some of those items are
things anyone could do. So if you like Emacs and you use Java, now this is a
great chance to give something back.
However, if JDEE is not to your taste, don’t forget you have emacs-eclim and
malabar-mode too. I’ve never used malabar-mode myself, but last time I had to
deal with Java I gave eclim a try and it worked pretty well. You do need an
eclipse server running in the background, but once you do it facilitates a good
amount of features.
Someday I typed more than 80 thousand times just in Emacs. This is
pretty awesome at first sight but it can cause serious health problem.
Last month, I felt burning pain of my forearms. It is an symptom
of Repetitive strain injury (RSI). I realised that if continue typing
like that, one day I will never able to do programming, like the Emacs
celebrities in Xah Lee' article about RSI.
Since then I've deliberately tried to avoid aimless and unproductive
typing, take more typing breaks, think though things before trying,
write more on paper.
Conditions are getting better: I don't feel server pain any more, only
sometimes uncomfortable.
But I need to find a better way to improve it. Because sometimes I got
the idea, but can't touch the keyboard. This feeling really suck.
So I investigated the Hydra package and use it to group related
commands together so that use only two keys are needed to perform
frequent tasks.
For example, to search something in current project, instead of typing
M-x helm proj grep, that's 16 keystrokes, I only need F5 G with
Hydra. The implementation is listed in this post.
But calling functions/commands in Emacs counts only a small proportion
of my typing; most of the time, I write code and report.
This is where Yasnippets kicks in, it enable me to type less without
losing quality. For example, I use this snippet quite often when
writing R code,
res<-sapply(seq_len(n),function(i){## })
That's more than 40 keystrokes. Yasnippets can short it to only 6s!
After I type sapply and then hit TAB, it will expand to the region
above.
I will investigate the Yasnippet package next week. If you know any
good tutorials for Yasnippet or snippets for writing R code, please
share your resources.
Sometimes, when you are in callback-function, you might want to
abandon the function that called you, useful-command in this case,
and call a different function, with the current context.
Here's what I've come up with to do just that:
(defmacroivy-quit-and-run(&restbody)"Quit the minibuffer and run BODY afterwards."`(progn(put'quit'error-message"")(run-at-timenilnil(lambda()(put'quit'error-message"Quit"),@body))(minibuffer-keyboard-quit)))
To break it up into parts:
minibuffer-keyboard-quit will unwind the call stack all the way to
the command loop, preventing e.g. do-thing-3 from being
called. Note that the call stack can be as deep as you like, e.g.
useful-command might be called by utility-command etc.
run-at-time with the argument nil will run the code as soon as
possible, which is almost exactly after we're back into the command
loop.
The final trick is to prevent Quit from being displayed in the minibuffer.
Sample application
Suppose that I've called find-file when ivy-mode is
active. Typically, I'd select a file and press C-m to open
it. However, sometimes I just want to see the selected file in dired
instead of opening it. This code is from before
ivy multi-action interface,
it plainly binds a command in ivy-minibuffer-map:
So at the moment when C-: is pressed, the call stack is:
C-x C-f called find-file.
find-file called completion-read-function which is set to ivy-completing-read.
ivy-completing-read called ivy-read.
ivy-read called read-from-minibuffer.
Thanks to the new macro, I can unwind all of that stuff, making sure
nothing extra will be executed that was supposed to be executed after
read-from-minibuffer had returned. In other words, the file will not
be opened. Instead, a dired buffer will be opened, centered on the
selected candidate.
What I described above can actually be a pretty common scenario, you
could adapt it to helm or avy or projectile. Basically to
anything that includes some form of completion (i.e. commands that
wait for input) and offers you a customizable keymap. Small
disclaimer: the above code falls into the quick-and-dirty category, I
don't recommend it if you can do something smarter instead. But if you
can't do anything smarter due to being constrained by an interface you
don't control, this macro could help you out.
Over the last couple of weeks I had a few more days to work on the Cider
debugger, and it’s getting a slew of improvements on the next release
(0.10.0). This starts with a complete rewrite, so it now supports almost
everything, and ends with some small features and UI improvements. Without
further delay, here’s a video.
There’s a lot going on in there, so let’s start dissecting.
Maps are now supported. This may sound trivial, but the previous version
couldn’t debug inside maps (it’s trickier than it seems).
More specifically, prepost-maps are supported! If this sounds sweet, that’s
because it is.
Even some code-rewriting stuff is supported, like the beloved threading macros.
Function literals are also supported.
The n, c, i, and q keys were already available in the previous
version, so I’ll just link you to the previous post on them.
The o key, showcased halfway through the video, moves you *o*ut of a sexp,
without moving back in. It’s useful in while loops and in map-like
operations.
The l key presents an inspector buffer detailing local variables.
Unlike in the previous version, you are allowed to move around and do other
stuff while the debugger waits for input. You can even evaluate stuff in the
current lexical environment with e or the usual C-x C-e.
In addition to the command used the video (which simply debugs an entire
function) you can also debug specific forms with #dbg, and you can place a
single breakpoint anywhere with #break. If you use one of these, just
evaluate the form with your usual evaluation commands (like C-x C-e or C-c
C-c) and you’re good to go.
Functions which are currently instrumented are marked with a red box around
the name.
You can list all instrumented functions with M-xcider-browse-instrumented-defs.
The next planned step for the debugger is ClojureScript support, but that’s
likely going to take a while. In the meantime, why not try the Cider 0.10.0
snapshots and help us find any bugs?
I might have mentioned before that I'm using GNU/Linux on all of my
computers. The particular flavor is Ubuntu, although it shouldn't
matter much, since I can count the graphical applications that I use
at all, besides Emacs, on one hand. They are: Firefox, Evince,
Rhythmbox and VLC. Of course, as most true Emacs-ers, I strive to
reduce this number to zero. So I got quite excited when I saw
helm-rhythmbox show up in
my package list. It works great: you can play and enqueue tracks with
completion without leaving Emacs. Big thanks to
@mrBliss, and, of course, the authors
for dbus.el which makes interaction with D-Bus possible through
Elisp.
I'm not as big a fan of Helm as I used to be, so I quickly implemented an Ivy equivalent:
(defuncounsel-rhythmbox-enqueue-song(song)"Let Rhythmbox enqueue SONG."(let((service"org.gnome.Rhythmbox3")(path"/org/gnome/Rhythmbox3/PlayQueue")(interface"org.gnome.Rhythmbox3.PlayQueue"))(dbus-call-method:sessionservicepathinterface"AddToQueue"(rhythmbox-song-urisong))));;;###autoload(defuncounsel-rhythmbox()"Choose a song from the Rhythmbox library to play or enqueue."(interactive)(unless(require'helm-rhythmboxnilt)(error"Please install `helm-rhythmbox'"))(unlessrhythmbox-library(rhythmbox-load-library)(while(nullrhythmbox-library)(sit-for0.1)))(ivy-read"Rhythmbox: "(helm-rhythmbox-candidates):action'(1("Play song"helm-rhythmbox-play-song)("Enqueue song"counsel-rhythmbox-enqueue-song))))
I listed the whole code just to show how easy it is to interact with
D-Bus, and also to show-off the shiny new multi-action interface of
ivy-read. Besides being discoverable via C-o, the
multi-action interface is extensible as well. Here's how to add a
"Dequeue" action without touching the original code:
Very simple, counsel-rhythmbox-dequeue-song is a clone of
counsel-rhythmbox-enqueue-song with only the method change from
AddToQueue to RemoveFromQueue (I blind-guessed the name, but there
should be a reference somewhere). If you got tired of having a whole
three actions to choose from, you can revert to the initial two with:
(ivy-set-actions'counsel-rhythmboxnil)
And here's how the updated C-o option panel now looks like:
A bit of descriptions:
j moves to the next one of the 18 current candidates.
k moves to the previous candidate.
h moves to the first candidate.
l moves to the last candidate.
g calls the current action without exiting.
d calls the current action and exits.
s moves to the next action.
w moves to the previous action.
i and C-o close the options panel without exiting.
o closes the panel and exits.
Similarly to hydra, each action has a short docstring, like "Play
song" that should describe what it does. Here are a few usage
scenarios:
If I wanted to play the first song and enqueue the third and the
sixth and exit the minibuffer, I would press gsjjgjjjd.
If I wanted to play the third and enqueue all the following, I would
press jjgsjcjjjjjjjjjjjo. The many j is just
me holding j until the selection reaches the end, then I
simply exit without doing anything else by pressing o.
The way c works is that it toggles the "calling" state -
a state where the current action is called whenever a different
candidate is selected.
Almost forgot, if this little intro got you excited, the packages that you should install from MELPA are: helm-rhythmbox and counsel.
Normally, org-mode ignores your attempts to markup text that starts with
" or '. That’s probably a safe measure because "~/" is a very
common string to write but ~ is one of Org’s markup elements.
Fixing that is a simple matter, but it takes a bit of digging around. We just
need to remove those two characters from the 3rd element of
org-emphasis-regexp-components.
;; This HAS to come before (require 'org)(setqorg-emphasis-regexp-components'(" ('\"{“”""- .,!?;''“”\")}/\\“”"" \r\n,""."1))
Today I'd like to highlight a generic command recently added to
hydra. It's kind of cute, and
maybe something that you didn't even know you wanted. Basically, the
hydra-pause-resume command allows to pause and resume the current
hydra with one key. And it works for all hydras at once, without any
extra configuration. All you have to do is to bind it globally to
whatever you like, for example:
(global-set-key(kbd"C-M-k")'hydra-pause-resume)
What it does:
If a hydra is active, deactivate and push it on the stack.
If no hydra is active, pop one from the stack and call it.
If the stack is empty, call the last hydra.
Personally, I haven't found too much use from the stack - resuming the
last one is enough for me. But the stack comes at no extra cost, and
might be useful to someone.
If, for example, you have 2 hydras on the stack, how to resume the
older one?
The way stacks work, you simply have to go through all elements one by one. So that's:
C-M-k to resume the newer one.
Quit the current one with the appropriate shortcut, only not with C-M-k, since that would just put it back on the stack.
C-M-k to resume the only one remaining on the stack - the older one.
I'm sure many people know that Emacs comes with a locate
command. This command, if you're on a Linux system, will find all
files on your system that match a particular pattern. The advantage of
locate over find when searching the whole system, is that it is
much faster, since it uses a pre-computed database. This database is
updated periodically, you can force an update with:
sudo updatedb
Of course find is faster if you need to search only a specific
directory instead of the whole system, but sometimes you just don't
know that directory.
counsel-locate
Dynamic
The way locate works it that it asks you for a query, which is
glob-based instead of regex-based, and then prints the results to a
static buffer.
On the other hand, counsel-locate is dynamic: each time you input a
new character a new locate query is ran, and the old one is
terminated. On my system, it takes around 2 seconds for a query to
complete, so it requires a bit of patience.
Regex-based
I like regex way more than globs for some reason. Here's the command
called for the input mp3$:
locate -i --regex mp3$
Of course, the standard ivy-mode
method is used to build the regex from a list of space separated
words. So the input fleet mp3$ will result in:
locate -i --regex \\(fleet\\).*?\\(mp3$\\)
You could go your own way and update the regex matcher to be
ivy--regex-fuzzy, which results in:
locate -i --regex f.*l.*e.*e.*t.* .*m.*p.*3$
But I think less matches is usually better than more matches.
Multi-exit
This is just the coolest feature. Basically, for each file you locate, you can easily:
Open it in Emacs (default).
Open it with xdg-open, so that PDF files are forwarded to evince
and MP3 files are forwarded to rhythmbox etc.
Open it in dired.
Here's an example of how to do it. First I call counsel-locate,
which I like to bind to C-x l. Then I enter emacs pdf$
and wait around 2 seconds for the 248 results to come up. Then I
scroll to the 18th result and press C-o to open up the
hydra-based option panel:
The last column (Action) is newer than others. As you can see, it has
3 exit points, which I can scroll with w and s.
And currently I'm on the default exit point, which would open the file
in Emacs.
For the next screenshot:
I pressed s twice to change the action to dired.
I pressed c to make the current action execute each time a new candidate is selected.
So now, I could just scroll through all my directories on my system
that contain PDF files related to Emacs by just holding j.
A similar thing can be done for music tracks:
C-x ldire mp3$ to get the list of all Dire Straits tracks on my system.
C-o s to switch the action from "open in Emacs" to "xdg-open".
From here, I could open one track after another by pressing C-M-n repeatedly. Or I can press c and then j repeatedly.
I've been experimenting with opening EPS and PDF files in quick
succession. It's still a work in progress, since I need to use a special
wmctrl script to prevent the Emacs window from losing focus each
time a new Evince instance is opened.
Outro
You can check out the new feature by installing counsel from
MELPA. It will automatically fetch ivy-mode as well. When you enable
ivy-mode, besides doing all your completion, it will also remap
switch-to-buffer to ivy-switch-buffer. That command also has a
multi-exit: pressing C-o sd instead of C-m will
kill the selected buffer instead of switching to it. It's a very
minor optimization: instead of C-m C-x k you press C-o
sd, however you could e.g. use C-o scjjjj to kill
five buffers at once.
While the idea of multi-exits is powerful, it's hard to find places to
use it efficiently. I think counsel-locate is a nice place for it,
although it could work without it:
Find file in Emacs with C-m in the completion interface.
The search-forward-regexp, replace-match, and match-string
functions work together nicely, and makes my job much easier and enjoyable!
I am writing a release notes for the a software updates. Part of the
process is to associate the SVN Revision number that relates to
important changes, so that others can backtrack and review the code and
see what exactly has been implemented.
In Phabricator, the revision number will be render automatically.
Clicking them takes me to the exact revision, showing the difference
with previous version. But the documentation will be eventually built
by Sphinx and hosted on a remote server. So I have to manually add the
URL to all the SVN revision number. For example, to replace rS1234 to
There are 31 revision number in the whole document. I could do it
manually but for the long term benefits, it would be more efficient
write a function to process it automatically, maybe others can use it
as well.
Implementation
The first thing I noticed is each SVN revision numbers consist of two
letters (rS) and few digits. Because the four digits I don't know
beforehand, I have to use regular expression to do the pattern search.
The tricky bit here is to retrieve the values that matched the
pattern, because of it is needed to construct the URL that points to
the commits, and I also need to replace the it with differnet values.
The procedure can be summarised as:
Find the revision number that match the patterns described above. I
use search-forward-regexp() to search the pattern "rS[0-9]+", which
means a string that starts with rS with one or more digits.a
retrieve the values that matched the pattern. This is done by match-string().
replace the revision number with the constructed URL. This is done by
replace-match(), and I use concat() to combine the IP address with the
revision number.
The following is a workable implementation:
(defvar revision-pattern "rS[0-9]+"
"The RegExp pattern of the SVN revision number")
(defvar repo-url "http://10.0.0.11/"
"The IP address of the SVN repository")
(defun yt/add-link-to-SVN-revision-number ()
"add links to svn commits identifier"
(interactive)
(while (search-forward-regexp revision-pattern)
(let* ((commit (match-string 0))
(link (concat repo-url commit)))
(replace-match "")
(org-insert-link nil link commit))))
Note the last two lines of the function can be simplified as
(replace-match (concat "[[" link
"][" commit "]]"))
You can easily adopt the code and make it applicable to your case,
just modify the revision-pattern and repo-url variables. But
beware that you should not apply the function to the same buffer more than
once, otherwise you will get something crazy like this:
One way to make it better is to have a test before replacing: if the
revision number is already associated with a URL, then do nothing. If
you have figure out how to do it, please let me know and I've happy to
update this post.
My posts published last year showed my frustration with regular expression in
Emacs. But now I am looking forward doing more text processing with
it, because it will be fun!
The search-forward-regexp, replace-match, and match-string
functions work together nicely and make the my job much easier and
enjoyable!
What's your favourite functions in regular expression? Do you have
something to recommend?
In this post, I'll demonstrate a snippet that came up this week in a
conversation over John Kitchin's
post.
While the following snippet isn't inherently useful, it might help a creative
user to understand Hydra a bit more and accomplish more advanced
things.
This hydra demonstrates that the hint and even the key bindings aren't
set in stone: you can modify them depending on the current state of
your Emacs, in this case the current line number.
Just paste that code into *scratch*, and press C-j. If
you want pretty output (153 lines of code instead of 26), turn on
lispy-mode and press E
instead of C-j.
Anyway, among these defined variables is hydra-vi/hint which is
evaluated each time to display the hint. So now we can just redefine
hydra-vi/hint to make it so that on even lines n calls
next-line, while on odd lines it's j, with the
appropriate changes in the doc. The change in bindings, modifying
hydra-vi/keymap - also one of the defined variables, needs to be a
side-effect, since hydra-vi/hint is expected to evaluate to a
string.
Just to give you some idea of how it could be used: you can have a
context-aware "open" command that, for instance, delegates to
open-in-pdf-tools or open-in-firefox or open-in-emacs when it
detects that the point is on a link. And of course all these commands
would have their own key binding that works only if the command makes
sense.
This approach is described on
the wiki,
in case you read this post much later and want to see an up-to-date
code, or even update it yourself. In case something cool comes out of
this snippet, I can try to implement a more palatable API for
defhydra, most likely an option to supply a function name in the
docstring argument position.
Org is one of those packages that you can use for a lifetime and still not know
all of its features. One of the first things you learn is how to use the
#+STARTUP header to define the initial visibility of headlines when you first
open an org-mode file. But did you know you can also use that on a per headline
basis?
Besides the usual 4 attributes you can chose from for the header (overview,
content, showall and showeverything), you can also assign similar options
to the VISIBILITY property of any headline. The specific options are folded,
children, content, and all, and their names are fairly self-explanatory.
As an example, remember this screenshot of my init.org from the birthday post?
To achieve that, you just had to set the * init.el headline to have
:VISIBILITY: children, along with the usual startup header you see in the
image. I was almost implementing this myself, when I decided to Google first and
wasn’t surprised to find org-mode already had it.
When this blog was conceived, I decided that I wanted it to be entirely
contained in a single org file, and that this would also be my Emacs init file.
On the blog’s very first post I explained how to implement the latter, an init
file that also serves other purposes. Today, Endless Parentheses turns 1 year
old, and it’s time to explain the former, how to turn a file into a blog.
As is usually the case with first birthdays, the child has no clue of what’s
going on, and it’s really just an excuse to indulge the parents into praising
themselves for keeping the kid alive 12 whole months. As such, this will not be
a productive post. Expect no code snippets, no Emacs news, and no productivity
tips, as I release myself from my usual shackles and go off on a tangent for a
change. Still, just in case someone else likes the idea, I’ve pushed the code to
Github.
Before anything else, it should go without saying that the content of the posts
is written in org-mode. The engine I use for exporting is a large wrapper
around ox-jekyll, and the posts are all pushed to Github and rendered by their
built-in Jekyll support.
Why keep a blog in a single file?
First of all because org, once you learn its knobs and bobs, is just plain
powerful. This is all the more true because the post contents are also in org
format, so you’re effectively removing one layer of distance between you and the
content. For example, if I want to reschedule a post, I just find it with C-c
C-j and hit C-c C-s; whereas, if the posts were separate files, I’d have to
find it in dired, then visit it, and then hit C-c C-s.
This is a small difference, but it applies all around. If I want to link to a
previous post, I find it with C-c C-j and then move back with C-u C-SPC, all
without leaving the buffer. When I look at the posts list, the tags are listed
right beside the title, I don’t have to open a file to see them.
Now let me be honest with you. I didn’t predict these advantages before I
started the blog, so it’s not why I chose this approach. Rather, it was linked
to the fact that I wanted to blog about all these elisp snippets I had built
over years and accumulated in my init file.
You see, if posts were separate files I would have to copy the snippets to a
separate org file, and then write about them there, and then export them to
Jekyll. In this scenario, I just know I would eventually change some snippet (a
healthy init file is a fluid creature) and forget to update the corresponding
org file, and the thought of leaving out-dated code lying around sent a chill
through my spine. Not to mention, this whole flow of “init file → org post →
jekyll post” has one layer too many for my taste, and redundancy is an evil I
slay with a chainsaw.
How it works
First of all, here’s what I see when Emacs starts up.
If you read (and remember) the first post, you’ll know that the actual init file
is composed of code blocks inside that init.el headline. Everything else is
text. Also, notice that last SEQ_TODO header. It specifies that headlines in
this file have three possible TODO states, TODO, READY and DONE.
Every post is marked with one of these states, and that is what defines them.
TODO is something I plan on writing about; READY is, well, ready to publish;
and DONE is published.
This three-states setup has several uses, mainly:
I can review my entire history and my future schedule with a custom agenda
command, leveraging all the features of org-agenda.
Whenever I change a snippet, and the headline above it is marked DONE, I’m
immediately reminded to update the post (as simple as C-c b). So I’ll never
leave outdated code around.
When Monday arrives sooner than expected, and I didn’t write anything new, I
can issue C-c / T READY and get an org-sparse-tree of all READY posts.
See how practical that is? This file is not just the blog and the init file,
it’s also the future posts queue and the “vague ideas” list. All without having
to do manual maintenance. Of course, the headline is automatically marked DONE
when posted.
Then there are a few more advantages that arise simply from the fact we’re using
org-mode.
I can physically move posts around with M-up/down and M-S-right/left, to
whatever order makes more sense, instead of being constrained by alphabetical
or historical order of files.
By nesting several posts under a headline, any tags or properties applied to
the parent are inherited by the posts. (Killing redundancy, remember?)
The C-c C-j (org-goto) command makes it a breeze to jump around.
It feels more Emacsy than anything else you could possibly do.
Last, but not least
This part applies to any of the org blogging methods (not just my setup), but
still, the org-export engine is extensive and even more so if you know where to
hack it (org links, in particular, are extremely versatile).
With a single C-c b, Emacs will spell-check the contents, export to Jekyll,
clean up some links, move the file to the right directory, bring up
magit-status, and even save a commit message to the kill-ring. You could
even have it commit and push automatically too, if you’re a bit of a thrill
seeker.
Good night, and thanks for coming
Well, now that I’ve finished monologuing, the cake is completely gone, there’s
more candy on the floor than on the tables, and I think I hear a child crying
somewhere, so it’s clearly time to wind down the party. Thanks for reading, and
I mean that generally. Thanks for commenting too. Thanks for the emails and the
tweets. Thanks for the bug reports and the pull requests, and any other form of
positive interaction. This was a fun year.
Oh, and thanks for coming to the party. Little EP totally didn’t notice that
you didn’t bring a gift. (Though he might notice next year… just saying.)
The first step in data analysis is to get the data into the modelling
platform. But it may not be as straightforward as it used to be since
nowadays statistician are more likely face the data files that are not
in CSV or others format that can feed directly to the read.table()
function in R, in which cases, we need to understand the data files in
terms of the structure and apply pre-process first. My general
strategy is to discard the unnecessary information in the data files
and hopefully leave a regular data files.
In my last week's post, Why I Should Explore Regular Expression and
Why I Haven't, I expressed my interests in Regular Expression and lucky
I got a chance to use it for getting the data into R. It provides me a
different strategy: pick only what I am interested in.
The Irregular Data Files
The task is simple: I have about 1,800 .text data files downloaded
from British Oceanographic Data Centre (BODC). They are the historical
tidal data and are separated by year and by port. I need to combine all
the data into one giant table in R, and save it later for modelling.
One sample data file looks like this:
Port: P035
Site: Wick
Latitude: 58.44097
Longitude: -3.08631
Start Date: 01JAN1985-00.00.00
End Date: 03OCT1985-19.00.00
Contributor: National Oceanography Centre, Liverpool
Datum information: The data refer to Admiralty Chart Datum (ACD)
Parameter code: ASLVZZ01 = Surface elevation (unspecified datum) of the water body
Cycle Date Time ASLVZZ01 Residual
Number yyyy mm dd hh mi ssf f f
1) 1985/01/01 00:00:00 1.0300 -0.3845
2) 1985/01/01 01:00:00 1.0400 -0.3884
3) 1985/01/01 02:00:00 1.2000 -0.3666
The first 9 lines are the metadata, which describes the port ID, name
and location of the port, and other information about the data. The
line 10 and 11 are the headers of the data matrix.
First Attempt - Skip Lines
After the glimpse of the data sample, my first thought was to
skip the first 12 lines and treat the rest as a regular data files
that has space as separator. It can be easily done by using
read.table() with skip = 12 option.
read.table(data.file,skip=12)## error
It turned out this approach won't work for some files because
when the way of measuring tidal were changed, the date and port were
highlighted, leaving a second chunk of data matrix but again with
metadata and few other characters. It looks like this:
;; end of first chunk
########################################
Difference in instrument
########################################
Port: P035
;; other metadata
Second Attempt - Remove Lines
Although the first attempt isn't success, I've learnt a bit about the
structure of the data files. And based on that, I came up with a
second approach: read the data files into R as a vector of string, one
element for a line, and then remove all the lines which are metadata.
They start with Port:, Site: or Longitude: etc or the ###
chunk. It can be done using grep function, which tells me exactly
which element of the vector contains the metadata.
This approach works well as long as the metainfo.list contains all
the lines I'd like to remove. The downside is that I won't able to
know I've includes all of them until the whole process is finished. So
when I was waiting for the program to finish, I came up with a third
approach, a better one.
Third Attempt - Capture Lines (RegExp)
The above two approaches are to discard the unnecessary information,
but I may be in the situation that there are other lines that should
be discard but I haven't encounter yet, then the process becomes
tedious try-error and takes quite long.
Equally, another approach is to select exactly what I am interested in
by using regular expression. But first, I have to identify pattern.
Each data point was recorded at a certain point, and therefore must be
associated with a timestamp, for example, the first data point is
recorded at 1926-01-01 00:00:00. They also has an ID values with an
closing parentage's, for example 1.
1) 1985/01/01 00:00:00 1.0300 -0.3845
So the content of my interests are have a common pattern that can
be summarised as: the lines that start with a number of spaces, and
also have
observation ID
few integers, and an ending parentheses,
observation date
few integers with forward slashes that
means year, month and day, and then a space,
observation time
few integers with colons, means hour, minutes and
seconds.
The patterns in RegExp can be formulated as the roi.pattern variable
and the whole process can be implemented as:
To me, there isn't an absolute winner between the second and third
approach, but I prefer to use regular expression because it has more
fun with it; I am a statistician and like to spot patterns.
Also, it is an direct approach and more flexible. Note I can continue
to add components to the regular expression to increase the confidence
in selecting the right data matrix. For example, there are spaces and
then few integers at the timestamp. But it will presumably increase
the run-time.
Code and Sample Data
You can download the exmaple data and run the scripts listed below
in R to reproduce all the results.
#### * Pathdata.file<-"~/Downloads/1985WIC.txt"## to the downloaded data file#### * Approach 1read.table(data.file,skip=11)## error#### * Approach 2s<-readLines(data.file)metainfo.list<-c("Port:","Site:","Latitude:","Longitude:","Start Date:","End Date:","Contributor:","Datum information:","Parameter code:")meta.line.num<-sapply(metainfo.list,function(i){grep(pattern=i,s)})res.2<-s[-meta.line.num]#### * Approach 3roi.pattern<-"[[:space:]]+[[:digit:]]+\\) [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"roi.line.num<-grep(pattern=roi.pattern,s)res.3<-s[roi.line.num]
(defuncounsel--load-theme-action(x)"Disable current themes and load theme X."(condition-casenil(progn(mapc#'disable-themecustom-enabled-themes)(load-theme(internx))(when(fboundp'powerline-reset)(powerline-reset)))(error"Problem loading theme %s"x)));;;###autoload(defuncounsel-load-theme()"Forward to `load-theme'.Usable with `ivy-resume', `ivy-next-line-and-call' and`ivy-previous-line-and-call'."(interactive)(ivy-read"Load custom theme: "(mapcar'symbol-name(custom-available-themes)):action#'counsel--load-theme-action))
It looks almost trivial, the main idea is to disable all current
themes and load the new one. Additionally, try to reset the
powerline, since it has to match the mode-line face, which most
themes customize.
The Interface
The interface of ivy-read is the same as the built-in
completing-read in first two arguments. The difference is that it
also accepts a callback through the :action argument. This callback
will make the completion engine aware of what needs to be done with
the completion result. The presence of the callback allows these completion engine features to work:
ivy-resume will resume the last completion. Very useful if you
change your mind on the candidate, or want to examine a related
candidate.
ivy-next-line-and-call selects the next matching candidate and executes the callback for it.
ivy-previous-line-and-call selects the previous matching candidate and executes the callback for it.
I like to think of ivy-resume as a DEL or
<left> for completion. As you can erase or go back
one character with the same DEL binding, regardless of the
last character inputted (a or B etc), in the
same way you can call the completion again with the same
<f6> binding, regardless of the command that required
completion (counsel-git-grep or counsel-load-theme or
counsel-load-library etc). ivy-resume isn't bound by default,
since it needs to be a global binding. I just use this in my config:
(global-set-key[f6]'ivy-resume)
For the functions that execute the callback while changing the
candidate, the idea is:
C-M-n (ivy-next-line-and-call) corresponds to C-n (ivy-next-line),
C-M-p (ivy-previous-line-and-call) corresponds to C-n (ivy-previous-line).
I've also showed off a generic ivy feature: M-j will yank
the word at point into the minibuffer. Think of it as the mirror of
C-w in isearch. It could not be C-w, since I
like C-w being bound to kill-region.
The command/insert mode split
Finally, I show off the equivalent
hydra-based modal selection
method. So instead of pressing C-M-n C-M-n C-M-n C-M-p
C-M-p, you can press the equivalent C-o c jjjkk.
Luckily, you don't need to remember a lot of bindings for this hydra
mode: just press C-o and read the hints. I'll just list the
exit points, since that's usually the more important stuff:
To exit the "command-mode" completely, press o or the standard C-g.
To exit the "command-mode" with the current candidate, press f or d.
To exit the "command-mode" and once again edit the minibuffer, press i.
You might ask why f and d do the same. They
actually mirror C-j (ivy-alt-done) and C-m
(ivy-done). And if you ask what the difference between those two is,
the answer is that ivy-alt-done will not exit the completion when
selecting directories during file name completion. It may be possible
to extend this to other types of completion where it makes sense to
select something but not to exit the minibuffer.
Outro
If you're using ivy-mode, make sure to try the new features: the
action-using commands should work for any command that starts with
counsel-. For other commands, like package-install, you can only
select one candidate with C-m.
Also do try C-M-n with counsel-find-file: you'll be able
to cycle through all files in a directory without exiting the
completion. Same goes for ivy-switch-buffer, which should probably
be renamed to counsel-switch-buffer for consistency.
I don’t usually dedicate an entire post to something I’ve already done in a
previous one, but this nugget is so useful it deserves the attention. Remember
how you can create Github PRs straight from Magit? Magit 2.1.0 is barely
two weeks away, and it brings so many (awesome) changes that our little snippet is
going to break.
Luckily for you, I’ve been using Magit’s next version for a couple of months
now, so I have a fix all shiny and ready for you. This function is also slightly
more robust, and fixes some corner-case issues in the other. So if you found
that the previous function didn’t cover your use-case, you should definitely try
the new one.
(defunendless/visit-pull-request-url()"Visit the current branch's PR on Github."(interactive)(browse-url(format"https://github.com/%s/pull/new/%s"(replace-regexp-in-string"\\`.+github\\.com:\\(.+\\)\\.git\\'""\\1"(magit-get"remote"(magit-get-remote)"url"))(cdr(or(magit-get-remote-branch)(user-error"No remote branch"))))))
Note that I’ve changed the keybind to v. In Magit 2.1, that key is bound in
buffer overlays, not on the mode-map, which means we can just bind it in the map
and we’ll get a semi “do-what-I-mean” functionality.
Update 24 Jan 2016
The latest Magit from Melpa deals with branches in a slightly different way, so
the code above won’t quite work. If you run into that problem, the update below
should work fine.
(defunendless/visit-pull-request-url()"Visit the current branch's PR on Github."(interactive)(browse-url(format"https://github.com/%s/pull/new/%s"(replace-regexp-in-string"\\`.+github\\.com:\\(.+\\)\\.git\\'""\\1"(magit-get"remote"(magit-get-push-remote)"url"))(magit-get-current-branch))))
To commemorate the release of CIDER 0.9.0, I've just
added the ability to debug-step-in Clojure expressions to lispy.
This ability was present for Elisp for a very long time, and it's
instrumental to
my Elisp output. So now
I've added exactly the same thing to Clojure.
An important thing is that the function needs to be loaded (probably
with cider-load-file), in order for Clojure to know its location. If
you only evaluate the function with C-x C-e, it won't
work. Actually, the same applies to Elisp.
And then you have the corresponding function call (| is the point, as usual):
|(l->rfix10* 2+ 3)
As you press xj (lispy-debug-step-in), the following code is evaluated on the Clojure side:
(do(def a10)(def op1*)(def b2)(def op2+)(def c3))
At the same time, you are taken to the second branch of the body of
l->rfix - exactly the one that corresponds to 5 arguments. And
that's it: you now have a, op1 etc defined to their proper
values. You can now continue within the function body with many
possible follow-ups. I'll just list the eval-related ones:
Use e to evaluate expression at point.
If you want to evaluate a symbol, mark it with M-m, end
evaluate with e. Actually, I prefer to mark stuff with
m and hjkl arrows, using i to mark
the first element of the region. It's also possible to mark with
a, and 2m, 3m etc.
If you want to bind a symbol in a let binding, mark both the
symbol and its value, and press e. If there are many
let bindings, you can navigate to the next one with either
jj or 2j.
You can debug-step-in again if needed with xj.
You can flatten a function or a macro call with xf.
You can eval-and-insert with E.
You can eval-and-commented-insert with 2e.
Just to add, any of these will work properly and switch to an appropriate body branch:
Enjoy the new code, but be mindful that it's really fresh, so it will
certainly have some quirks. For instance, function arguments as a map
won't work since I haven't programmed for that yet. Big thanks to
@bbatsov and all CIDER contributors.
A really cool thing that I hope to get in the future, is to make
Z (lispy-edebug-stop) also work in Clojure. What it does
currently for Elisp, is to use edebug to setup the function
arguments. It may be possible to use
@Malabarba's new debugger
implementation for the same thing.
The advantages of using lispy-style debugger are the following:
You can navigate your code the way you want, not just in a way that
the debugger allows you to.
You can multi-debug, and have sessions last for days.
You can edit the code as you debug. For this new code, I started
debugging when there was only a function name. I added the body code
expression-by-expression, simultaneously debugging it.
The disadvantage is a bit of namespace pollution, but I think it's
more or less acceptable.
This is a feature-packed release consisting of 57 commits done over
the course of the last month by me and many
contributors.
I'm trying something different this time with the
release-notes:
I've started a
Changlelog.org
in the repository, which is much more pleasant to read (and write) in org-mode form inside Emacs:
Then I used pandoc to convert the Org file to Markdown. Too bad, the pandoc install-able through apt-get knows nothing of Org mode,
so I had to
cabal install pandoc
and let cabal do its thing for like 30 minutes. The exported
Markdown wasn't too bad, just had to reformat most things to one line,
since Github's fill-column is like 150 chars instead of the
usual 80. If someone has more experience of exporting Org to
Markdown, please do share: maybe there's a better way to do this.
Anyway, read the release notes either at github or inside Emacs and enjoy the cool new stuff. Big thanks to all contributors.
Also check out the wiki page on customization and other pages.
I think it would be cool to have a separate page for e.g. avy-goto-char with the screenshots (or gifs) for all values of avy-style:
pre
post
at
at-full
de-bruijn
So if anyone is willing to contribute (and check out those overlay
styles in the process), please go ahead: you can either clone the wiki
with:
Like many R users who are not actually programmer, I am afraid of
regular expression (RegExp), whenever I saw something like
I'd told myself I won't be able to understand it and gave up on the
sight.
But I've collected few RegExp patterns that do magical
jobs. My favourites are the dot (.) and dollar ($) sign and I usually
use them with list.files() to filter the file names in a directory. For
example,
The first line returns all the R image files, which have file names
ending with RData, and for the second all the text files which have
file names ended with text. Basically in regular expression, dot
sign (.) means anything, and dollar sign ($) means the end of a
string. By combining these two, I am able to select multiple files
with certain patterns, without manually picking one by
one.
How powerful is that! It is an inspirational example that motivates
myself from time to time to look deeper and get my head on the topic
of regular expression. But I just couldn't have a clear picture of how to
us it.
I think the main problems for me to understand RegExp in R are
The syntax is content-sensitive
A subtle change can lead to random results. For example, the above
pattern can also be \\.RData$, which means file names ended with
.RData. The dot (.) sign here literally means ".". Adding two
backslashes \\ changes the meaning of the pattern completely, but
both gives the same results. It gave me so much frustration when
extrapolating a pattern that works in one case to a similar case
but get random results.
The syntax is hard to read
The RegExp pattern above are reasonably easy to understand, if one
spent 10 minutes reading the manual, but the following is just crazy.
There are 12 parentheses, 6 square brackets and many other symbols.
Even same symbol have different meanings, and it's hard to find out
exactly what they means because
There isn't enough learning materials
I've never seen an R book that mentioned regular expression. This topic
is certainly not a teaching content in university courses or training
workshops.
Even google fails to find any meaningful resource except for the Text
Processing in Wiki, which is the best I could find.
Although there are related questions in StackOverflow, most of the
answers were set in a very specific situation. It's hard make it
applicable to other situations or learn this topic from the discrete Q&As.
It has created a mental barrier that statistician shouldn't teach nor
learn RegExp at all, or at least for me. But my limited experience
suggests that it is such a powerful feature that I've missed a lot.
But
I believe there will be more chances to process text files, for example,
parse the log files of this blog. RegExp can improve the efficiency to
a great extent. So I am considering to invest the time to learn it properly.
Are you a R user? What's your experience with regular expression? Do
you have good learning materials to recommend? If so, please share
your experience on the less-talked area.
Having to confirm-before-save buffers every time you call compile is nearly as
outrageous as having no key bound to compile in the first place. This snippet
takes care of both and, as a bonus, makes the compilation window follow a
predetermined size and ensures that point will follow the output.
;; This gives a regular `compile-command' prompt.(define-keyprog-mode-map[C-f9]#'compile);; This just compiles immediately.(define-keyprog-mode-map[f9]#'endless/compile-please);; I'm not scared of saving everything.(setqcompilation-ask-about-savenil);; Stop on the first error.(setqcompilation-scroll-output'next-error);; Don't stop on info or warnings.(setqcompilation-skip-threshold2)(defcustomendless/compile-window-size105"Width given to the non-compilation window.":type'integer:group'endless)(defunendless/compile-please(comint)"Compile without confirmation.
With a prefix argument, use comint-mode."(interactive"P");; Do the command without a prompt.(save-window-excursion(compile(evalcompile-command)(andcomintt)));; Create a compile window of the desired width.(pop-to-buffer(get-buffer"*compilation*"))(enlarge-window(-(frame-width)endless/compile-window-size(window-width))'horizontal))
Update 16 Jun 2015
Thanks to Clément and abo-abo for the variable suggestions, which also led me to
find the compilation-skip-threshold variable. Warnings are important to fix,
but in some languages it’s common to have warnings you can’t fix, so it’s nice
for the compilation buffer to not stop scrolling on them.
Update 17 Jun 2016
The command now propagates the prefix argument to the compile function, so
that you can start the buffer in comint-mode. See the update on this post for
more information.
Org-mode is great to serve as knowledge management tool, it also has helped
me increase my personal effectiveness. Recently I have been exploring
org-mode for managing small projects in the business environment,
in which collaboration happens occeasionally between me and the project team members.
In this post I summarised my workflow to organise, manage and monitor a
project. The implementation of this workflow revolves around the
collaboration. I have been practise this workflow for a while and can
see my growth in planing and managing skills.
Organising
I use a broad definition of project: as long as a task that requires a series
of sub-tasks to be done, then it is a project. Normally I categories
any tasks that relates to a project into three groups:
Project Tasks
the major tasks that must to been done in order to
deliver the project product.
Tasks
administrative or miscellaneous tasks that keep the project goes
on, like sent out the invoice.
Notes
anything that is important to the project and therefore
worthy keeping a record, like meeting notes or decision
made that that impacts the project progress.
Each category has a corresponding top level section or heading. Once this outline is setup, it
is very convenient to view content under these categories,
regardless of what tasks I was working on, either reading emails,
coding, or writing report. Org-mode can scan all the .org
files in a direcotry, and creates a tree-structure, with the file name
being the root, and headings being the nodes.
An intuitive way to locate a any node is to start from the beginning,
the process is same as finding a section in a text book. It can be summarised as:
first, find the right book by its name,
then find the right part,
then narrow down to the right section,
and continue to the section I am interested in. An more pleasure way is
to use fuzzy match supported by Helm package - I can narrow down the
selection by random nodes. For example, as the images below shows, to
locate headline under this article among 40 org files, I only need to search "Small pro",
because there are only three headlines has "Small" in its name, "small
changes", "small talk", and "small project", and "pro" narrow down to
the unique headline.
It saves me a lot of time in remembering where I saved one notes, and
wandering around the files to find something. I only explain a bit of
the features of Helm, if you want to try out, you can find my
configuration here. I recommend a good tutorial if you want to know
more.
Figure 1: Test image
We usually a couple of projects at the same time. Also, create a new
tasks or notes is easy. org-capture-mode would create a temporary
node and by default it will be saved as a subtree in refile.org, or I
can directly re-locate the headline directly to this project using the
locating mechanism above.
These two features are most enjoyable to use, and make me away from
wandering in multiple directories, trying to find the right files, and
therefore increase my productivity. Never under estimate how long you
will spent in finding in one file.
Managing
Projects usually come with hard deadlines about the product delivery.
Setting and change deadlines in org-mode is pleasurable with org-deadlineC-c C-d.
It brings up a mini-calendar buffer (shown below), I can use
shift+left and shfit+right to move forward and backward for a
day, or shift-up and shift-down to move between weeks, and hit
RET to select a deadline. Apart from navigating, I can also choose to
type the exact date directly, like "2015-07-25" and hit RET.
Once the deadline is set it will show up in that day's calendar. I
don't want to suddenly realise there is a deadline I had on that day.
So it makes sense to have an early warning period to show the tasks if
it is due in days. This behaviour is governed by the
org-deadline-warning-days variable. In my Emacs configuration, I set
to 30 days. It gives me plenty of time to do any tasks.
I also set deadlines for sub-tasks since it is quite easy to do in
org-mode. But coming up with realistic deadlines is difficult. To me,
it must give enough time to do the task properly, to the PM, it must
be fit in the whole project plan and resource. Both are likely to have
different opinion on how long to implement the new features with
documentation. It is quite important skills to have: to me, it
reflects my understand on the problem and also my own technical
capability, to the manager, it is part of their project plan.
My initial estimation may be far from the actual effort, especially
when the problem domain is new to me, or I haven't done similar tasks
before. The more I do, the better I am good at estimating. At this
stage, I practise this skill seriously, and like to have someone with
more experienced to review my estimation.
To make this task easy for them, I'd present an overall view of the
project time-lines, which clearly shows the period allocate to the
specific tasks. org-timeline will generate a time-sorted view for
all the tasks. The recent feedback I received is that I tend to
overlook the time spent on documentation and tests. Someone with more
than 10 years in software development says they usually takes about 3x
times on these two tasks together than actually coding.
time-line view also provides benchmark to the progress and I check it
frequently to make sure I am on track. It gives the PM a reference for
swapping tasks if some becomes urgent.
Monitor
Additional to have the early warning system to prevent sudden surprise,
org-mode provides another way of monitoring the project in terms of resource -
the actual time I spent on the project. This feature is quite useful
when I am given a quite loose deadline but with limited resource,
say 150 hours.
Since the sub-tasks are mostly defined in the early stage, whenever I
start to do it, I clock in first by org-clock-in. The clocking will
be stopped once I manually clock out, or clock in to another task, or
the tasks is completed (marked as DONE.) For each clock entry, it
shows start time, end time and duration.
Multiple clocking logs are accumulated, and each entry shows the start
time, end time, and duration. The durations can be added up and tells
me exactly how much time I spent on each tasks. The whole tasks under
the project and aggregated across the whole project, by one single
function org-clock-report (C-c C- C-r).
Table 1: Clock summary at [2015-06-14 Sun 11:17]
Headline
Time
Effort
Total time
10:41
TODO Use Emacs's org-mode to Manage a Small Project
10:41
TODO Tasks
1:45
DONE add example for org-refile
0:35
0:30
NEXT add example for org-clock-report
0:13
0:15
NEXT proof read
0:11
0:15
NEXT proof read - 2
0:46
1:00
It is normal to underestimate the complexity of an tasks, and spent
too much time in resolve them, and usually I can catch up the in the
later stage, however if I had the feeling the overall progress has
been affected, I need require more sources from the PM, and the quote
I will give is extra hours I had based on my initial estimation.
That's an quick reaction.
Also, the clock-report table tells me the different between my effort
estimation and the actual time I spent on that tasks.
Just a little extension to the old xc (lispy-to-cond)
command that I've added today to lispy:
Previously, xc could only transform a series of nested if
into a cond. You can even chain xcxi to transform a
case statement into equivalent if statements.
Note also that unwanted whitespace is properly removed. What you see
in the picture is actually an
ERT test
that's being run each time I commit new stuff. It makes sure that
starting in the first buffer state and pressing xc really
results in the second buffer state. With this test, it's reasonable to
believe that xc will actually work the same way in an
interactive scenario. Right now, there are 599 of these type of tests
in lispy-test.el. To view them like in the screenshot, you can press
xv.
Final note, I realize that it should be eql instead of eq, but I
like eq more: it's 33% more efficient.
This is the feature I’ve been wanting to show off the most. Anyone who’s
configured Emacs to use more than one package archive knows this problem.
The package menu displays countless redundant entries, as it must list a package
once for each archive that offers it. Even worse, if you install a package from
one archive, the package menu will gladly upgrade it to a newer version on
another archive, clueless to the fact that it may be giving you unstable code.
All of this has been addressed now, with the package-archive-priorities
variable. It is an alist where you can assign a priority number to each archive
you use. Thanks to Jorgen Schaefer, the first thing it does is change the
package-install command to download the package from the highest priority
archive instead of just choosing the highest version.
Since you all know I’m a bit of a menu fanatic, I took my time extending that
functionality to the package menu. Now, instead of listing all entries for a
given package, the menu will only list the package that’s offered by the highest
priority archive. With the above configuration, you will only see Melpa packages
if they are not also available from another archive. More importantly, after
installing a stable version you will never be inadvertently upgraded to an
unstable version.
This behaviour can be permanently configured with the
package-menu-hide-low-priority variable, and you can temporarily toggle it on
and off with (. Therefore, if a package has stable version available but you
prefer the Melpa version, you can display it with ( and then install it as
usual. And what’s more, package.el will identify that situation and will keep
upgrading you on the bleeding-edge for that specific package, even though Melpa
is the lowest priority archive.
Furthermore, the ( key is now a general key for displaying possibly unwanted
packages (much like it hides/displays details in dired buffers). In addition
to the behaviour above, this key will also display packages that are considered
obsolete. That is, those whose version is lower than something you already have
installed. This was not previously possible, and it can be used to downgrade a
package. As long as the older version is available from some archive, you can
mark it for installation and mark the current version for deletion.
This concludes our series on package.el. There are other small improvements here
and there, but it’s time we go back to our weekly code snippets and productivity
tips. Hopefully, this was enough to inspire some of you to clone the Emacs repo
and start building the master branch.
I refer to counsel-describe-variable here, which you can get from
the MELPA counsel package, or from
Github. If you're not familiar, counsel contains specialized functions using ivy-mode completion. Being specialized means that you can do more than usual, for instance here are the exit points of counsel-describe-variable:
RET calls describe-variable: a plain describe-variable with ivy-mode on
looks like counsel-describe-variable, but has only this exit point.
C-. calls counsel-find-symbol: jump to symbol declaration.
C-, calls counsel--info-lookup-symbol: jump to symbol reference in *Info*.
These can also be extended by binding new commands in counsel-describe-map.
I'll just describe one useful scenario that happened just now. I was working on a checkdoc.el change
and was testing it with avy.el. And I got a few style warnings for this code, which I got from a contributor (thanks for the code btw):
(defvaravy-translate-char-function#'identity"Function to translate user input key. This can be useful foradding mirror key. E.g. one can make SPACE an alternative of 'a',by adding:\(setq avy-translate-char-function (lambda (c) (if (= c 32) ?a c)))to allow typing SPACE instead of character 'a' to jump to the locationhighlighted by 'a'.")
One of the warnings that avy-translate-char-function should be quoted while inside the docstring like so:
"`avy-translate-char-function'"
Of course that wouldn't play well with setq, so I wanted to see if
any other variables that end in -function provide an example of use
like here, and, if so, how they quote.
So I did <f1> vfunction$ to describe a variable that ends with function.
Here, $ is a part of a regular expression, which should be a priority to learn if you want to be a (not just) Emacs Power User.
And now the key part: instead of pressing RET to describe just the current candidate, I repeatedly press C-M-n (ivy-next-line-and-call). After each C-M-n, the next candidate will be selected and the *Help* window will be updated.
With this trick I was able to skim through the 346 matches in seconds. The conclusion was that an example isn't usually provided, with nnmail-expiry-wait-function being an exemption.
In the end, I decided not to provide such an explicit example and shorten the doc to this:
(defvaravy-translate-char-function#'identity"Function to translate user input key into another key.For example, to make SPC do the same as ?a, use\(lambda (c) (if (= c 32) ?a c)).")
It took me quite a while to figure out, but finally
counsel-git-grep, which I've described
before works completely
asynchronously, which means it's very fast and smooth even for huge
repositories.
If you don't know what git grep is: it allows you to search for a
regular expression in your Git repository. In a way, it's just a
find / grep combo, but very concise and fast. It's particularly
great for finding all uses and references to a symbol.
It can be useful even for non-programmers. One great application is to
stick all your org-mode files into a Git repository. Then you can
search all your files at once very fast: all my org stuff takes more
than 1000,000 lines and there's no lag while searching. I really
should remove some pdfs from the repository at some point.
The Video Demo
Check out the speed and the various command in this Video Demo. Each key stroke starts two asynchronous shell calls, while canceling the current ones if they are still running. Here are the example calls:
$ git --no-pager grep --full-name -n --no-color \
-i -e "forward-line 1"| head -n 200
$ git grep -i -c 'forward-line 1'\| sed 's/.*:\\(.*\\)/\\1/g'\| awk '{s+=$1} END {print s}'
The Elisp side
I figure since I was interested in this topic, probably a few more people are as well.
So I'll explain below how async processing is done (in this case):
(defuncounsel--gg-count(regex&optionalno-async)"Quickly count the amount of git grep REGEX matches.When NO-ASYNC is non-nil, do it synchronously."(let((default-directorycounsel--git-grep-dir)(cmd(concat(format"git grep -i -c '%s'"regex)" | sed 's/.*:\\(.*\\)/\\1/g'"" | awk '{s+=$1} END {print s}'"))(counsel-ggc-process" *counsel-gg-count*"))(ifno-async(string-to-number(shell-command-to-stringcmd))(let((proc(get-processcounsel-ggc-process))(buff(get-buffercounsel-ggc-process)))(whenproc(delete-processproc))(whenbuff(kill-bufferbuff))(setqproc(start-process-shell-commandcounsel-ggc-processcounsel-ggc-processcmd))(set-process-sentinelproc#'(lambda(processevent)(when(string=event"finished\n")(with-current-buffer(process-bufferprocess)(setqivy--full-length(string-to-number(buffer-string))))(ivy--insert-minibuffer(ivy--formativy--all-candidates)))))))))
Since we're calling a shell command, it's important to set default-directory to a proper value: the shell command will be run there.
The output from the process will be channeled into a buffer. I start the buffer name with a space to make it hidden: it will not be shown in the buffer list and switch-to-buffer completion.
I leave the sync option and use shell-command-to-string for that. Sometimes it's necessary to know the amount of candidates without a delay.
If get-process and get-buffer return something it means that the shell command is still running from the previous input. But the input has changed and that data has become useless, so I kill both the process and the buffer.
Then I start a new process with start-process-shell-command- a convenience wrapper around the basic start-process that's useful for passing a full shell command with arguments.
Finally, I let counsel--gg-count return without waiting for git grep to finish. But something needs to be done when it finishes, so I use set-process-sentinel. Emacs will call my lambda when there's a new event from the process.
Outro
To check out the new stuff, just install or upgrade counsel from MELPA. Or clone the source. I hope that my Elisp explanations were useful, async stuff is awesome, and I hope to see more of it in future Emacs packages.
For several reasons, the Package Menu’s f key has always flown a bit under the
radar for me. Though the package-menu-filter command is great in principle, in
practice its usefulness falls a little short for several reasons.
One reason, which is in no way its own fault, is that Emacs already has great
searching facilities such as isearch or occur. It’s just easier and faster
to fall back on what you know rather than try out something new. This not
exactly something that needs fixing, but it can be improved by focusing the
filter command on doing things what other facilities can’t do.
Now you might be thinking, “doesn’t this command filter by package keywords?
Isearch can’t do that.”
And that takes us to the second (probably most relevant) reason. The list of
keywords used by packages is an ugly mess. And that’s not something easy to fix.
Before investing too much resource on this never-ending struggle against entropy,
Emacs 25.1 will make a simple improvement that goes a long way. You can filter
multiple comma-separated keywords at once. So, for instance, you can mitigate
the redundancy between the theme and themes keywords by just searching for
theme,themes.
Furthermore, sometimes there’s a good improvement available that is completely
orthogonal to the problem you have. Instead of fixing up keywords, we can also
have package-menu-filter offer other features that alternate facilities
don’t —preferably one that doesn’t rely on the supernatural cooperation of a
thousand monkeys typing elisp. As of Emacs 25.1, you may now filter packages by
status or by archive, by writing arc:ARCHIVE and status:STATUS respectively
when you invoke the command.
Of lesser importance, there was also a third usability issue with the command.
It took a couple of seconds to load every time you invoked it. That,
fortunately, was much simpler to solve, and I’ll even leave it as an exercise to
the reader!
In the following (very streamline) function, place point anywhere you want and
then invoke a single keybind (from emacs -q, your custom keybinds don’t
count). If you do it right, you can improve its speed a hundred-fold. Can you
see how?
(defunpackage-all-keywords()"Collect all package keywords";; `package--mapc' simply calls the lambda on each;; known package. `package-desc--keywords' returns;; the package's list of keywords.(let((key-list))(package--mapc(lambda(pkg)(setqkey-list(appendkey-list(package-desc--keywordspkg)))))key-list))
Check in again next Monday for the last post on this series, where we talk about
prioritizing your package archives.
Lispy 0.25.0 came out 2 months ago; 177 commits later, comes version
0.26.0. The release notes are stored
at Github, and
I'll post them here as well.
The coolest changes are the new reader-based M, which:
Gives out very pretty output, with minor diffs for actual code,
which is quite impressive considering all newline information is
discarded and then reconstructed.
Works for things that Elisp can't read, like #<marker ...> etc, very useful for debugging.
Customizable rule sets; rules for Elisp and Clojure come with the package.
The improvements to g and G also great:
Because of caching, the prettified tags can be displayed in less
than 0.15s on Emacs' lisp/ directory, which has 21256 tags in 252 files.
The tags collector looks at file modification time, so you get the updated tags right after you save.
The details for these and other features follow below.
Fixes
C-k should delete the whole multi-line string.
y should work for all parens, not just (.
p should actually eval in other window for dolist.
Prevent pairs inserting an extra space when at minibuffer start.
ol works properly for active region.
New Features
Misc
xf will pretty-print the macros for Elisp.
M-m works better when before ).
Fix ', ^ after a ,.
Improve / (splice) for quoted regions.
Z works with &key arguments.
The new M is used in xf.
Allow to flatten Elisp defsubst.
c should insert an extra newline for top-level sexps.
Paredit key bindings
You can have only Paredit + special key bindings by using this composition of key themes:
(lispy-set-key-theme'(specialparedit))
The default setting is:
(lispy-set-key-theme'(speciallispyc-digits))
New algorithm for multi-lining
M is now bound to lispy-alt-multiline instead of
lispy-multiline. It has a much better and more customizable
algorithm.
See these variables for customization:
lispy-multiline-threshold
lispy--multiline-take-3
lispy--multiline-take-3-arg
lispy--multiline-take-2
lispy--multiline-take-2-arg
They are set to reasonable defaults. But you can customize them if you feel that a particular form should be multi-lined in a different way.
lispy-multiline-threshold is a bit of ad-hoc to make things nice. Set this to nil if you want a completely rigorous multi-line. With the default setting of 32, expressions shorter than this won't be multi-lined. This makes 95% of the code look really good.
The algorithm has a safety check implemented for Elisp: if read on the transformed expression returns something different than read on the original expression, an error will be signaled and no change will be made. For expressions that can't be read, like buffers/markers/windows/cyclic lists/overlays, only a warning will be issued (lispy can read them, unlike read).
d and > give priority to lispy-right
For the expression (a)|(b), (a) will be considered the sexp at point, instead of (b). This is consistent with show-paren-mode. If a space is present, all ambiguities are resolved anyway.
b works fine even if the buffer changes
I've switched the point and mark history to markers instead of points. When the buffer is changed, the markers are updated, so b will work fine.
Extend Clojure reader
In order for i (prettify code) to work for Clojure, it must be able to read the current expression. I've been extending the Elisp reader to understand Clojure. In the past commits, support was added for:
empty sets
commas
auto-symbols, like p1__7041#
Extend Elisp reader
It should be possible to read any #<...> form, as well as #1-type forms.
g and G get a persistent action for ivy
This is a powerful feature that the helm back end has had for a long time. When you press g, C-n and C-p will change the current selection. But C-M-n and C-M-p will change the current selection and move there, without exiting the completion.
This also means that you can call ivy-resume to resume either g (lispy-goto) or
G (lispy-goto-local).
e works with defvar-local
As you might know, the regular C-x C-e or eval-buffer will not reset the values of defvar, defcustom and such (you need C-M-x instead). But e does it, now also for defvar-local.
Improve faces for dark backgrounds
I normally use a light background, so I didn't notice before that the faces looked horrible with a dark background.
The ` will quote the region
If you have a region selected, pressing ` will result in:
`symbol'
Customize the file selection back end for V
V (lispy-visit) allows to open a file in current project. Previously, it used projectile.
Now it uses find-file-in-project by default, with the option to customize to projectile.
Fixup calls to looking-back
Apparently, looking-back isn't very efficient, so it's preferable to avoid it or at least add a search bound to improve efficiency. Also the bound became mandatory in 25, while it was optional before.
Thanks to this, to e.g. get the value of a quoted var in a docstring or a comment, or jump to its definition, you can M-m. Then, you can step-in with i to select the symbol without quotes.
Update the tags strategy
A much better algorithm with caching an examining of file modification time is used now. This means that the tags should be up-to-date 99% of the time, even immediately after a save, and no necessary re-parsing will be done. And it all works fine with the lispy-tag-arity modifications.
1% of the time, lispy-tag-arity stops working, I don't know why, since it's hard to reproduce. You can then pass a prefix arg to refresh tags bypassing the cache, e.g 2g or 2G.
Also a bug is fixed in Clojure tag navigation, where the tag start positions were off by one char.
The fetched tags retrieval is fast: less than 0.15s on Emacs'
lisp/ directory to retrieve 21256 tags from 252 files. Which means
it's lightning fast on smaller code bases (lispy has only 651 tags).
xj can also step into macros
lispy-debug-step-in, bound to xj locally and C-x C-j globally can now step into macros, as well as into functions. This command is very useful for Edebug-less debugging. Stepping into macros with &rest parameters should work fine as well.
p can now lax-eval function and macro arguments
When positioned at function or macro args, p will set them as if the function or macro was called with empty args, or the appropriate amount of nils. If the function is interned and interactive, use its interactive form to set the arguments appropriately.
Again, this is very useful for debugging.
Allow to paste anywhere in the list using a numeric arg
As you might know, P (lispy-paste) is a powerful command that:
Replaces selection with current kill when the region is active.
Yanks the current kill before or after the current list otherwise.
Now, you can:
Yank the current kill to become the second element of the list with 2P
Yank the current kill to become the third element of the list with 3P
...
It's OK to pass a larger arg than the length of the current list. In
that case, the paste will be made into the last element of the list.
Update the way / (lispy-splice) works
When there's no next element within parent, jump to parent from
appropriate side. When the region is active, don't deactivate it. When
splicing region, remove random quotes at region bounds.
This change makes the splice a lot more manageable. For example,
starting with this Clojure code, with | marking the current point:
(defn read-resource"Read a resource into a string"[path](read-string|(slurp (clojure.java.io/resourcepath))))
A double splice // will result in:
(defn read-resource"Read a resource into a string"[path]|(read-stringslurp clojure.java.io/resourcepath))
After xR (reverse list), 2 SPC (same as
C-f), -> (plain insert),
[M (back to parent and multi-line), the final result:
(defn read-resource"Read a resource into a string"[path]|(-> pathclojure.java.io/resourceslurpread-string))
This also shows off xR - lispy-reverse, which reverses
the current list. Finally, reverting from the last code to the
initial one can be done simply with xf - it will flatten
the -> macro call.
Outro
Thanks to all who contributed, enjoy the new stuff. Would also be nice
to get some more feedback and bug reports. Currently, it might seem
that a large part of the features are either perfect or unused.
It was six months ago, to the day, when I alluded to the fact that Emacs’
package menu needed to go async. The time it took to do a simple list-packages
bothered me the most, closely followed by having to go play Minesweeper every
time I did a package upgrade. The latter was partially addressed when I added
asynchronous package transactions to Paradox, but the former took a bit more
work. In Emacs 25.1, at last, the package menu is going async.
You don’t need to do anything special to benefit from this. As soon as you issue
M-xlist-packages, instead of those “Contacting host: ...” messages which
always foretell a many-second hang, the package menu will come up almost
instantly. The download of archive information will go on in the background, and
once it is done the new information is updated in place.
This has two big advantages.
A fraction of a second after issuing the command you’re already in the menu,
free to navigate, search, or mark stuff while the background download is
happening.
If you have multiple archives configured (which you should), they are fetched
simultaneously. So the entire download will be 2–4 times faster now, even if
you decide to sit and wait for it to finish.
It should be noted this only applies to refreshing. Package transactions
(installation, upgrade, and deletion) are still synchronous in package.el.
Async transactions where implemented for a while, but the outcome was quite far
from satisfactory. However, so as not to end on a sad note, you can always go to
Paradox for that.
Lastly, if you’re the kind of person that hates nice things, you can disable
this feature with the package-menu-async variable.
That should be enough about (a)synchronicity for the moment. Come back on
Monday, when we go into some big improvements on the filtering engine.
In Thurday's post on dependency management, I briefly mentioned that
package.el now keeps track of which packages the user explicitly requested,
and which were pulled in as dependencies. But there’s a bit more to this
feature, so it deserves some time in the spolight.
Simply put, there is now a new custom variable package-selected-packages. This
variable stores the names of packages installed explicitly by user. So every
time you do M-xpackage-install or you do i x in the Package Menu, the
name of that package gets added to this list. Packages which get pulled in as
dependencies are not added to this list, and those which are explictly deleted
get removed from the list. This is how package-autoremove knows what to
remove, it just finds packages which (a) are not on this list and (b) are
not required by anything else.
But this variable comes with other benefits too. First, the user can edit it
manually with the usual customize-variable and use it to keep track of their
list of wanted packages. Second, there’s now another command,
package-install-selected-packages, which ensures that all packages on the
list are installed. This means you can safely move to a new computer, or even
just delete your elpa/ subdir. As long as you keep your custom settings you
can just invoke the command and all your packages will be reinstalled.
There’s one small caveat, which some of you may have noticed. This bookkeeping is
done during installation. So, when you finally upgrade to Emacs 25.1, how is it
going to know which of your installed packages were user-selected and which were
dependencies?
Well, it’s just impossible to know for sure, so it makes an educated guess. It
takes all installed packages that are not required by any other installed
package, and considers them to have been explicitly installed. This can (and
probably will) yield both false positives and negatives, but that only happens
the very first time you start Emacs 25. So just keep in mind you may need to
customize-variable and fine-tune this list.
At this point, swiper is only a
fraction of ivy-mode's functionality. Still, it's nice to keep them
all, together with counsel, in a single
repository: counsel-git-grep works much better this way.
Anyway, I'll echo the release notes here, there are quite a few exciting new features.
Fixes
TAB shouldn't delete input when there's no candidate.
TAB should switch directories properly.
require dired when completing file names, so that the directory face is loaded.
TAB should work with confirm-nonexistent-file-or-buffer.
TAB should handle empty input.
work around grep-read-files: it should be possible to simply M-xrgrepRET RET RET.
Fix the transition from a bad regex to a good one - you can input a bad regex to get 0 candidates, the candidates come back once the regex is fixed.
ivy-switch-buffer should pre-select other-buffer just like switch-buffer does it.
Fix selecting "C:\" on Windows.
counsel-git-grep should warn if not in a repository.
C-M-n shouldn't try to call action if there isn't one.
Turn on sorting for counsel-info-lookup-symbol.
ivy-read should check for an outdated cons initial-input.
New Features
Out of order matching
I actually like in-order matching, meaning the input "in ma" will match "in-order matching", but not "made in". But the users can switch to out-of-order matching if they use this code:
ivy-re-builders-alist is the flexible way to customize the regex builders per-collection. Using t here, means to use this regex builder for everything. You could choose to have in-order for files, and out-of-order for buffers and so on.
New defcustom: ivy-tab-space
Use this to have a space inserted each time you press TAB:
(setqivy-tab-spacet)
ignore case for TAB
"pub" can expand to "Public License".
New command: counsel-load-library
This command is much better than the standard load-libary that it
upgrades. It applies a sort of uniquify effect to all your
libraries, which is very useful:
In this case, I have avy installed both from the package manager and
manually. I can easily distinguish them.
Another cool feature is that instead of using find-library (which is
also bad, since it would report two versions of avy with the same
name and no way to distinguish them), you can simply use
counsel-load-library and type C-. instead of
RET to finalize.
Here's another scenario: first load the library, then call
ivy-resume and immediately open the library file.
New command: ivy-partial
Does a partial complete without exiting. Use this code to replace
ivy-partial-or-done with this command:
In regex terms, ^ is the beginning of line. You can now use this in
swiper to filter your matches.
New command: swiper-avy
This command is crazy good: it combines the best features of swiper
(all buffer an once, flexible input length) and avy (quickly select
one candidate once you've narrowed to about 10-20 candidates).
For instance, I can enter "to" into swiper to get around 10 matches. Instead of using C-n a bunch of times to select the one of 10 that I want, I just press C-', followed by a or s or d ... to select one of the matches visible on screen.
So both packages use their best feature to cover up the others worst drawback.
Add support for virtual buffers
I was never a fan of recentf until now. The virtual buffers feature works in the same way as ido-use-virtual-buffers: when you call ivy-switch-buffer, your recently visited files as well as all your bookmarks are appended to the end of the buffer list.
Suppose you killed a buffer and want to bring it back: now you do it as if you didn't kill the buffer and instead buried it. The bookmarks access is also nice.
Here's how to configure it, along with some customization of recentf:
All these commands just forward to their built-in counterparts, only trying not to exit the first line of the minibuffer.
M-DEL calls ivy-backward-kill-word
C-d calls ivy-delete-char
M-d calls ivy-kill-word
C-f calls ivy-forward-char
Allow to customize the minibuffer formatter
See the wiki on how to customize
the minibuffer display to look like this:
100 Find file: ~/
file1
file2
> file3
file4
When completing file names, TAB should defer to minibuffer-complete
Thanks to this, you can TAB-complete your ssh hosts, e.g.:
/ssTAB -> /ssh
/ssh:olTAB -> /ssh:oleh@
More commands work with ivy-resume
I've added:
counsel-git-grep
counsel-git
Others (that start with counsel-) should work fine as well. Also don't forget that you can use C-M-n and C-M-p to:
switch candidate
call the action for the candidate
stay in the minibuffer
This is especially powerful for counsel-git-grep: you can easily check the whole repository for something with just typing in the query and holding C-M-n. The matches will be highlighted swiper-style, of course.
Allow to recenter during counsel-git-grep
Use C-l to recenter.
Update the quoting of spaces
Split only on single spaces, from all other space groups, remove one space.
As you might know, a space is used in place of .* in ivy. In case
you want an actual space, you can now quote them even easier.
Outro
Thanks to all who contributed, check out the new stuff, and make sure
to bind ivy-resume to something short: it has become a really nice
feature.
Doing things in Emacs is superlatively better than having to switch
to another application.
In this case, "doing things" is getting the dictionary definition of
word at point, and "superlatively" is a word that I didn't know - a
straw that broke the camel's back and caused me to finally automate
the process of getting the definition of a word that I encounter in an
Emacs buffer.
The whole process of writing the define-word package took around 30
minutes, I just had to:
Try to get an API key, read their draconian TOS and decide that I don't want
to agree to it just to get their key.
Examine the HTML that it returns and note that it's quite regular.
Write a 10 line function with re-search-forward to extract the word
definitions from a sample page that I saved with wget.
Then just wrap the function in an url-retrieve and done. It's a
good thing that I learned to use url-retrieve when I wrote
org-download.
Here's how it looks like in action, the word under point is "Authors"
and instead of visiting
this page, you can see it
right away in your Echo Area:
The result is displayed simply with message, so it doesn't mess with
your window config. You read it, press any key and the Echo Area
popup will vanish automatically.
Install the package from MELPA or check it out at github.
You just need to decide where to bind it:
At less than 50 lines, the source is very easy to understand. So if
you're looking to write some Elisp that retrieves and parses some HTML
from a web service, it's nice to look at a simple implementation of
how it's done.
Package.el has gotten a series of improvements after the release of 24.4.
Since I’ve found that people like to read about upcoming features, I’m starting
a new series exclusively about our favorite package manager. Today, we talk
dependencies.
Package.el’s dependency management has always been just enough to do the job
it’s supposed to do. If you install a package that has dependencies, those
requirements get installed as well. End of story.
This leaves a little to be desired for two reasons. Firstly, if you later remove
the first package, the dependencies will be left on your computer. Secondly,
even if you notice those dependencies lying around and want to remove them, you
never know if that’s safe because Package.el won’t tell you if another package
depends on them and, what’s worse, it will let you remove it even if another
installed package does depend on them (which will lead to breakage).
The situation is being improved on both accounts, thanks to Thierry Volpiatto.
Package.el will differentiate between packages you’ve installed explicitly
and those which were just pulled along as dependencies. So you get a new
package-autoremove command to cleanup dependencies that are no longer
needed.
Package.el will never let you remove a package if some other package depends
on it. So you’re free to try to delete anything you don’t want, and nothing
is going to break if it happens to be a dependency (you’ll just fail).
And this led to some UI improvements to the package menu, thanks to yours truly.
Packages that were pulled in as dependencies are marked as such in the
package menu.
The description buffer (the one you get when hitting RET on a package) is
now kind enough to list all packages that depend on this one.
As a whole, this is a great user experience improvement. Instead of seeing
installed packages that they never actually installed, the user will see
dependency packages. Every once in a while, after removing some package, they
might get notified that “these dependencies are no longer necessary”, and can
easily clean them up with package-autoremove.
If you want more 25.1 news, there’s also a whole series just about that, and you
can always just check out the emacs-25 tag.
If you're doing some Elisp coding, you should definitely check out the
built-in checkdoc command. Too bad it's (was) interactive-only. With
a small modification, I've made it suitable for batch.
You can check out how I did it in the
avy (or
lispy) repository. Here's my
compile target in the Makefile:
I made compile target a dependency of the all target, so with a
single make I can:
run the tests
check for compiler warnings
check for checkdoc style warnings
compile over ansi-term
There's no reason to use ansi-term (or an external shell) over
compile in this case. Using M-xcompile (or actually
M-xhelm-make) I
can navigate to any compilation or style warning or failed test from
the *compilation* buffer.
For the lazy, it's possible to jump to errors with a mouse, but
navigating errors is a breeze with this Hydra:
I make good use of one of the coolest compilation-mode features:
pressing g will restart the compilation process.
Outro
Check out the new feature, I think it's really cool. Another incentive
to try is that lately the
emacs trunk has been
extremely stable: I'm on Emacs25 almost all the time now, since it
feels a lot faster.
Here are my aliasing settings:
$ which newemacs
/usr/local/bin/newemacs
$ readlink -f `which newemacs`
/home/oleh/git/gnu-emacs/src/emacs
To run the tests with newemacs instead of emacs (since that's the
one for which checkdoc works in batch), use M-xsetenv
-> emacs -> newemacs.
Proof-general is a powerful client for the Coq proof assistant, and Software
Foundations is great interactive tutorial for the language. As I was following
the tutorial, I felt the need to speed things up a little bit. Today’s post is
just some configuration code I wrote for that effect.
Parts of it might be useful for coq in general, but its mostly optimized to
minimize keystrokes in the tutorial, and sometimes it leaves bad indentation or
extra lines all over the place.
;; I appreciate the effort of writing a splash-screen, but the angry;; general on the gif scares me.(setqproof-splash-seent);;; Hybrid mode is by far the best.(setqproof-three-window-mode-policy'hybrid);;; I don't know who wants to evaluate comments;;; one-by-one, but I don't.(setqproof-script-fly-past-commentst)(with-eval-after-load'coq;; The most common command by far. Having a 3(!);; keys long sequence for this command is just a;; crime.(define-keycoq-mode-map"\M-n"#'proof-assert-next-command-interactive);; Proof navigation didn't work for me. So please;; stand aside for my paragraph navigation.;; https://endlessparentheses.com/meta-binds-part-2-a-peeve-with-paragraphs.html(define-keycoq-mode-map"\M-e"nil)(define-keycoq-mode-map"\M-a"nil);; Small convenience for commonly written commands.(define-keycoq-mode-map"\C-c\C-m""\nend\t")(define-keycoq-mode-map"\C-c\C-e"#'endless/qed)(defunendless/qed()(interactive)(unless(memq(char-before)'(?\s?\n?\r))(insert" "))(insert"Qed.")(proof-assert-next-command-interactive)))(defunopen-after-coq-command()(when(looking-at-p" *(\\*")(open-line1)))(advice-add'proof-assert-next-command-interactive:after#'open-after-coq-command)
These are some common abbrevs, and an advice so you don’t have to hit SPC
before M-n.
I have a fondness for Emacs commands and key bindings that you can
get for free. Here, by "free" I mean customizations that don't
require you to change your old workflow (i.e. unbind your old
bindings), but still get the new workflow with comfortable bindings.
For free (almost physical) keys, see my post on
xmodmap.
See the last post
to see how you can get window manipulation commands "for free" when
you call ace-window.
With hydra, you can define
lightweight minor modes for almost free or even completely free key
bindings.
With lispy, you get list
manipulation bindings (single letters, no less) for free when your
point is positioned at a list boundary, or when the region is
active.
With worf, you get heading
navigation and manipulation bindings, also single-letter, when your
point is at a heading or markup start.
Today, I describe a recent addition to
avy: when you call avy-goto-line
and decide that you don't want to use avy-keys to select a line on
screen, and you want to select a line by number, you can just enter
that number.
avy-goto-line will recognize that you entered a digit, and forward
to goto-line with that digit already pre-entered. So basically there's no disadvantage to doing this:
(global-set-key(kbd"M-g g")'avy-goto-line)
Even if you use the avy method zero times, you lose no efficiency
when compared with regular goto-line, you just get a pretty
light-show with each call:
You can customize a lot of things in avy, for instance the keys or
the way the overlays are displayed. See the
new wiki page on
customization for more info.
I kind of forgot to tag the 0.8.0 release on Github, so it's been a
whole 3 months since the last release. In this post, I'll only
describe the newest exciting feature in more detail, see
the archive for
olderpostsonacewindow.
New Features
Display the window decision chars in the mode line
Enable ace-window-display-mode for this. This gives you the
advantage of always being aware which window corresponds to which
char.
New defcustom: aw-ignore-current
This is off by default. When t, ace-window will ignore
selected-window.
Allow to switch the window action midway
Ace-window has many commands available, like:
ace-select-window
ace-delete-window
ace-swap-window
...
But did you wish sometimes when you called ace-select-window that
you should have called ace-delete-window? In the old way, you would
cancel ace-select-window with C-g and call
ace-delete-window.
With the new way, you can, just press x followed by the
decision char. All keys are customizable through aw-dispatch-alist.
(defvaraw-dispatch-alist'((?xaw-delete-window" Ace - Delete Window")(?maw-swap-window" Ace - Swap Window")(?naw-flip-window)(?vaw-split-window-vert" Ace - Split Vert Window")(?baw-split-window-horz" Ace - Split Horz Window")(?idelete-other-windows" Ace - Maximize Window")(?odelete-other-windows))"List of actions for `aw-dispatch-default'.")
The strings beside each command are important: they are used to update
the mode line when you press a char. They also mean that a window
should be selected using aw-keys for the corresponding command. If
there's no string, the command is just called straight away, with no
arguments. To reiterate, for each entry without a string, its command
will be called immediately, and for others the window will be selected
first.
Also, take note of aw-flip-window. Suppose the you have a lot (say
7) windows, but you only want to cycle between the most recent
two. You can do so with n, with no need to press the
decision char.
I call this feature "the dispatch". The dispatch normally happens when:
you're prompted for aw-keys
you press a char that isn't in aw-keys
there's an entry in aw-dispatch-alist for this char
If you want to skip step 1 always (since, by default, you're not
prompted for aw-keys when you have 2 or less windows), use:
(setqaw-dispatch-alwayst)
Be careful though, setting this means that you'll always have to
select a window with aw-keys, even if there are only two. This is a
large toll on the muscle memory. On the other hand, even with one
window, assuming you've bound ace-window to M-p, you get:
split-window-vertically on M-p v
split-window-horizontally on M-p b
delete-other-windows on M-p o
What's also nice is that these commands scale with the amount of
windows: if you have only one window, you get no prompt for M-p
v, so it acts just like C-x 2. But if you have more
windows, you don't have to select the window that you want to split
beforehand: you can select it after you decided to issue a split
operation.
Give the new feature a try. The jump in utility between the new and
old ace-window, I feel, is of the same magnitude as the jump between
other-window and ace-window. However, it doesn't come for free and
the muscle memory needs to be readjusted slightly.
Big thanks to all who contributed, especially to @joedicastro.
If you’ve been following our journey of typography, you must now have pretty
apostrophes all over your org documents. But if that’s the case, you probably
also noticed a drawback. Ispell doesn’t like them very much. Now how are we
supposed to use our amazing auto-correct?
Don’t despair, dear reader, I have the solution. It wasn’t trivial to come up
with, but it’s quite simple to understand.
Tell Ispell that apostrophes are just like hard-quotes, where it comes
to constituting a word.
Convert all ’ to ' as they’re being sent to the Aspell subprocess.
Convert them back when reading the subprocess output. We even take care to
only do that in org-mode, so that it won’t get in the way of your LaTeX
buffers.
The following code uses the new advice system in Emacs 24.4. It can certainly be
made to work on older Emacs by changing it to use defadvice.
(require'ispell);;; Tell ispell.el that ’ can be part of a word.(setqispell-local-dictionary-alist`((nil"[[:alpha:]]""[^[:alpha:]]""['\x2019]"nil("-B")nilutf-8)));;; Don't send ’ to the subprocess.(defunendless/replace-apostrophe(args)(cons(replace-regexp-in-string"’""'"(carargs))(cdrargs)))(advice-add#'ispell-send-string:filter-args#'endless/replace-apostrophe);;; Convert ' back to ’ from the subprocess.(defunendless/replace-quote(args)(if(not(derived-mode-p'org-mode))args(cons(replace-regexp-in-string"'""’"(carargs))(cdrargs))))(advice-add#'ispell-parse-output:filter-args#'endless/replace-quote)
This is, admittedly, a hacky solution. But it’s been working well for me for the
last few weeks. Feel free to shout and scream at me if you run into issues.
This package contains the library on which ace-window and
ace-link now depend, as well as a multitude of navigation
commands.
Intro
Yes, they used to depend on ace-jump-mode, hence the
ace in their names, but as I added features to my packages over time, it became harder to compose
features because of ace-jump-mode inflexibility.
Don't get me wrong, ace-jump-mode is a fine standalone package, and if you're using it only for
jumping to chars, there are few reasons for you to make the switch. However, if you're interested in
the new and juicy features, such as:
jumping to word starts
jumping to sub-word starts
jumping to line beginnings or endings
copying and moving lines
with most of the features coming in -0, -1, or -2 flavor, or if you're interested in building
an ace-jump-mode-based package, then you should consider avy.
The leading chars flavor
All of the commands provided are about jumping to visible characters, and possibly doing some stuff.
Since usually there are a lot of these visible characters, it's advantageous to first narrow them
somewhat. So each command can be divided into two phases: the narrowing phase and the decision phase.
There are 3 variations of the narrowing phase:
The -0 flavor, e.g. avy-goto-word-0 or avy-goto-line doesn't narrow at all. The advantage
that there's minimum context switching (no narrowing phase, only decision phase). The disadvantage
is that the number of candidates in the decision phase can be quite large.
The -1 flavor, e.g. avy-goto-word-1 or avy-goto-char narrows by the first character of the
thing. The disadvantage is the context switch: first you call the command, then you switch context
to scan and input the leading char, then you switch context to scan and input the decision chars.
The advantage is less candidates in the decision phase.
The -2 flavor is my favorite avy-goto-char-2: it reads two consecutive chars, instead of
just one. This doesn't impact the narrowing phase too much, since inputting two consecutive chars
isn't much more than just one. But it greatly decreases the number of candidates in the decision
phase. The small amount of candidates allows me to use this setting for the decision:
(setqavy-keys'(?a?s?d?f?g?h?j?k?l))
That's only 9 decision chars, all of them on the home row.
The decision char overlay flavors
Once again, there are three, here are their corresponding defcustoms:
(defcustomavy-goto-char-style'pre"Method of displaying the overlays for `avy-goto-char' and `avy-goto-char-2'.":type'(choice(const:tag"Pre"pre)(const:tag"At"at)(const:tag"Post"post)))(defcustomavy-goto-word-style'pre"Method of displaying the overlays for `avy-goto-word-0' and `avy-goto-word-0'.":type'(choice(const:tag"Pre"pre)(const:tag"At"at)(const:tag"Post"post)))
If you're used to ace-jump-mode, the corresponding style is at, which displays the overlay at
the target position, one character at a time. I don't like this style most of the time, I prefer
pre which is more similar to vim-easymotion: it will display the full char path at once, before
the target position. This results in less feedback and more efficiency, in my opinion.
Outro
Give the new package a try, it's already been pretty active the last few days, with bugs being fixed
and new features appearing, thanks to all who contributed.
If you want to see some screenshots, there are plenty at the
repository page, and also in an earlier
post.
Finally, the package in on MELPA, so it's convenient to install. The only hassle is to decide where to bind all these commands.
The above two plots show the same data (included below), and if you are going to present one to summarise your findings, which will you choose? It is very likely you are going to pick the right one, because
the linear increasing feature of bars is pleasant to see,
it is easier to compare the categories, the ones on the right has higher value than the ones on the left, and
categories with lowest and highest value are clearly shown,
In this article I am trying to explain how to specify the plotting orders in ggplot to whatever you want and encourage R starters to use ggplot2.
To create a bar plot is dead easy in R, take this dataset as an example,
mode
count
ssh-mode
2361
fundamental-mode
4626
git-commit-mode
4869
mu4e-compose-mode
4964
emacs-lisp-mode
6205
shell-mode
10046
minibuffer-inactive-mode
12624
inferior-ess-mode
25774
ess-mode
47115
org-mode
78195
to get the plot on the right side, reorder the table by count (it is already been done), then
with(df,barplot(count,names.arg=mode))
will do the job. That's simple and easy, it does what you provide. This is completely different to ggplot() paradigm, which does a lot computation behind the scene.
ggplot(df,aes(mode,count))+geom_bar()
will give you the first plot; the categories are in alphabetically order. In order to get a pleasant increasing order that depends on the count or any other variable, or even manually specified order, you have to explicitly change the level of factors.
df$mode.ordered<-factor(df$mode,levels=df$mode)
create another variable mode.oredered which looks the same as mode, except for the underlying levels are in different. It is set to the order of counts. Run the same ggplot code again will give you the plot on the right. How does it work?
First, every factor in R is mapped into an integer, and the default mapping algorithm is
sort the factor vector alphabetically,
map the first factor to 1, and last to 10.
So emacs-lisp-mode is mapped to 1 and ssh-mode is mapped to 10.
What the reorder script can do is to sort the factors by count, so that ssh-mode is mapped to 1 and org-mode is mapped to 10, I.e. the factor order which are set to the order of count.
How does this affects ggplot? I presume ggplot do the plotting on the order of levels, or let's say on the integer space, I.e. do the plotting from 1 to 10, and then add the labels for each.
In this example, the default barplot function did the job. Usually we need to do extra data manipulation so that ggplot will do what we want, in exchange for the plot good better and may fits in the other plots. Without considering the time constraints, I would encourage people to stick with ggplot because like many other things in life, once you understand, it becomes easier to do. For example, it is actually very easy to specify the order manually with only two steps:
first, sort the whole data.frame to a variable,
then change the levels options in factor() to what ever you want.
To show a decreasing trends - the reverse order of increasing, just use levels = rev(mode). How neat!
It was a few years ago that I learned about ace-jump-mode in one of Magnar’s
Emacs Rocks episodes. Over this time, slowly but surely, this one simple command
has completely taken over my workflow. It was only last week that I realised how
ingrained it is on my muscle memory. As I shared this thought on twitter,
@_abo_abo’s avy was mentioned in the conversation and I decided to give it a
try.
Until now, I was under the impression that it was just ace-jump for windows,
but it’s a little more than that. It is actually a full replacement for
ace-jump, with small improvements here and there (plus quite a bit of parallel
functionality which I haven’t explored yet).
A couple of small but important differences.
By default, the buffer text doesn’t get shadowed while you’re doing a jump. I
didn’t like this at first, but now I much prefer it. Shadowing the text is
prettier, but I would commonly get lost if I didn’t focus intently on that
letter before the shadowing was applied.
When more than one key is needed for a jump, it’ll show you the full path,
instead of one key at a time.
This is a feature-packed release with a lot of cool things like:
Partial completion on TAB
Resume the last completion session with ivy-resume
Multi-tier regex matching
The detailed release notes follow.
Fixes
Glob expansion in rgrep
While completing file names, ivy expands the file name to
full. Unfortunately, rgrep uses read-file-name-internal and isn't
receptive to globs being expanded with the current directory. A
work-around this is to use the generic strategy when ivy is in
trouble:
enter the input text as if there was no completion.
exit with C-u C-j (forwards to ivy-immediate-done).
ivy-immediate-done is currently unbound by default. If you want, you
can bind it in your config like this:
This time, they are jabber-chat-mode and elfeed-search-mode.
Fix a flag in swiper-query-replace
You can launch a query replace with M-q from swiper.
Avoid sorting org-refile candidates
By default, ivy completion candidates are sorted with
string-lessp. The sorting can be customized with
ivy-sort-functions-alist. While refiling, the natural order is
actually best, so the sorting is turned off in that case.
Reset to the first candidate when switching directories
With ivy-mode you can select a file anywhere on your file system by
switching directories repeatedly with C-j
(ivy-alt-done). After moving to a new directory, the point should be
on the first candidate.
Fixup the face order
All this time there was an issue with the face order swapping between
1 and 2 groups. This is now fixed. Also, I've made
swiper-match-face-4 inherit from isearch-fail. It's important to
have all 4 faces be different.
Don't error on bad regex
When the current input is a bad regex, just display no candidates and
continue. Don't throw an error.
New Features
Use // instead of / to move to root
While completing file names, you can enter // to move to
the root directory. This change was necessary in order to make it
possible to enter e.g. /sudo: or /ssh:.
Host completion for /sudo: and /ssh:
This feature is a bit flaky for now. But it works well on my
machine. It should get better after a few bug reports. You can start
the completion right after the method, e.g. /ssh:RET, or
after method+user, e.g. /ssh:oleh@RET.
Respect confirm-nonexistent-file-or-buffer
If you set confirm-nonexistent-file-or-buffer to t (it's nil by
default), you'll have to confirm when you create a new file or buffer
with ivy.
Highlight remote buffers with ivy-remote face
Just some extra polish to make things look nicer. The buffers to which
you're connected through TRAMP will be highlighted with the new
ivy-remote face.
Change the prompt for match-required interactions
Sometimes, the Emacs functions that call completion specify to it that
a match is required, i.e. you can't just type in some random stuff -
you have to select one of the candidates given to you. In that case
ivy will appropriately change the prompt like this:
Improve the candidate selection while using history
While completing, you press M-p to select the previous
input. This update tries to select not just the first candidates that
matches, but the actual previous candidate that you selected before.
Use alpha compositing to add ivy-current-match-face
This is only relevant for when the completion candidates have custom
face backgrounds. But in that case, the minibuffer looks a lot nicer.
Add partial completion
Press TAB to do call ivy-partial-or-done to complete the
current thing to the largest possible prefix. When called twice in a
row, it's the same as C-j, i.e. it will finish the
completion.
Improve completion of hidden buffers
In Emacs, hidden buffer names start with a space. To see them all,
press a single space while completing buffers in ivy-mode. You can
toggle between hidden and non-hidden buffers by editing the first
space in your input query. Remember that with ivy-mode the
minibuffer is a proper editable area, so C-a works properly
(unlike in isearch or ido-mode).
Allow to quote spaces while matching
Spaces are wild while matching - they serve as group
boundaries. However, sometimes it would be useful to quote them. From
now on, you can quote N consecutive spaces by inputting N+1
consecutive spaces.
Add multi-tier regex matching
This is actually a really cool feature, so if you're paying attention
to any section, let it be this one.
User side
For example, I cloned
boost-1.58.0
and called counsel-git, which is like a find-file for all files in
a git repository at once.
Initially, it gives 45919 candidate files.
With input "utility", there are 234 candidates.
With input "utility hpp", there are 139 candidates.
Now, the interesting part. If I want to exclude anything with "hpp" in
it, I change the input to "utility !hpp" (with M-b !) to
get 95 candidates (95=234-139, it checks out). I could exclude some
more:
with input "utility !hpp cpp" there are 57 candidates.
with input "utility !hpp cpp ipp" there are 46 candidates.
I can unify the regex to "utility ![hic]pp" and also get 46 candidates.
exclude htm with "utility ![hic]pp htm" to get only 17 candidates.
You can use this strategy anywhere, not just for git file. For example, in describe-function or swiper.
Elisp side
You can customize ivy-re-builders-alist to make ivy-mode complete
in the way that you like. The alist dispatches on the collection
type, so you can have one completion strategy for buffers, another for
files and still another for everything else.
Each function on the alist should turn the string input into a string
regex. So the simplest one would be regexp-quote. If you want to use
multi-tier matching, the function should instead return a list of
regexps of two types:
the ones that should match (will be joined by and).
the ones that should not match (will be joined by or).
Add ivy-resume
This feature is still a bit of a work-in-progress. But it allows you
to resume the last completion to the point before you entered
RET or C-g etc. It only works for features that
specifically passed :action to ivy-read. You could resume other
features, but nothing would be done when you select the candidate,
since ivy-completing-read has no idea what the function that called
it was going to do with the result.
If you're considering to switch from ido-mode to ivy-mode, now is
a good time, since the most glaring gaps have now been filled. You can
find some up-to-date info on the
swiper wiki. I switched the
page syntax from markdown to org-mode, but Github makes the headings
way to big. So it might be easier to clone the wiki and view the org
files in Emacs:
As one thing leads to another, it occurred to me that I didn't know
what __GNUC__ macro does. Which brought me to the manual page on
Common Predefined Macros.
Using org-mode Babel to check the macro values
I decided that just skimming through the page wasn't enough, and I
wanted to brush up on my org-mode babel. So I created this
wiki page
to which you're welcome if you ever need to check __BYTE_ORDER_ or
__UINT_FAST16_TYPE__ on your system.
Here's a small excerpt of it:
#+begin_src C :results verbatim#include<stdio.h>intmain(){// version stuffprintf("__GNUC__ %d\n",__GNUC__);printf("__GNUC_MINOR__ %d\n",__GNUC_MINOR__);printf("__GNUC_PATCHLEVEL__ %d\n",__GNUC_PATCHLEVEL__);printf("__VERSION__ %s\n",__VERSION__);return0;}#+end_src
When you navigate to any place in the code and press C-c C-c, which calls (sic) org-ctrl-c-ctrl-c,
it will re-compile and re-run your program.
I list here some of the config to make it all work:
You can read up on what auto-yasnippet is in this post.
For the wiki page, I had to insert the printf statements a bunch of times, some of them with triply-repeated symbols.
Take, for instance, this statement:
If you're crazy enough (in a good way) to follow oremacs, you'll get the
new wiki page delivered right to your config with
make install
which will also update to the latest org-mode and CEDET.
And here's how to access it:
run xmodmap etc/.Xmodmap to make ; into a modifier
press ;-k to open a dispatch hydra
press w to select the wiki
select C (currently, the only other candidate is git)
Outro
Check out the new auto-yasnippet method, I think it's pretty
efficient. I hope that you'll find the wiki page useful in your C
pursuits. And a big thanks to Eric Schulte for org-babel.
This tip comes from colleague of mine. Ben has a mailing group at work where he
sends weekly Emacs tips. They’re always short and useful, but today’s tip was
quite the gem for me. Emacs has a comment-box command.
I’m sure half the readers know about this already, but I didn’t, so I thought
I’d share with the second half. He also offers an improved version of the
command, originally by Snader, which extends the box up to the fill-column.
This is a slightly obscure feature that I've added today to
ivy-mode. It's obscure, since
very few packages give their candidates a face with a custom
background. Still, when that happens, a conflict arises:
Which face to apply: ivy-current-match or the existing one?
In the image below, I've blended them both in a 50-50 mixture:
By the way, if you're wondering where the backgrounds come from in the
first place, it's lispy-goto
function (if you know what imenu is, it's an advanced version of
that). This function collects all the Elisp tags in the current
directory, and offers to jump to them. The functions that have
interactive in them, are highlighted with lispy-command-name-face.
In the above screenshot, the first 7 functions are interactive, and the other 2 are not.
If you're interested in how color blending is done, I'll describe it
shortly below.
Color-building internals
First, a smaller and more clear example (I've used rainbow-mode and htmlize-buffer):
Here, colir-blend is a very small function that I'll describe below,
and color-values is a function from faces.el that forwards to a C
function xw-color-values with this doc:
Return a list of three integers, (RED GREEN BLUE), each between 0
and either 65280 or 65535 (the maximum depends on the system).
Which is a bit weird, since the last time I checked, the standard for
colors was one-byte per channel, not two. Granted, it was about 10
years ago that I checked, and indeed, the
entry on Wikipedia
says:
High-end digital image equipment are often able to deal with larger
integer ranges for each primary color, such as 0..1023 (10 bits),
0..65535 (16 bits) or even larger, by extending the 24-bits (three
8-bit values) to 32-bit, 48-bit, or 64-bit units
Apparently, Emacs is outfitted to deal high-end digital image
equipment. But I'm still pretty sure that most graphics cards that you
can buy only give 8 bits per each channel. Anyway, here's the code for
the basic functions:
(defuncolir-join(rgb)"Build a color from R G B.Inverse of `color-values'."(format"#%02x%02x%02x"(ashr-8)(ashg-8)(ashb-8)))(defuncolir-blend(c1c2&optionalalpha)"Blend the two colors C1 and C2 with ALPHA.C1 and C2 are in the format of `color-values'.ALPHA is a number between 0.0 and 1.0 which corresponds to theinfluence of C1 on the result."(setqalpha(oralpha0.5))(apply#'colir-join(cl-mapcar(lambda(xy)(round(+(*xalpha)(*y(-1alpha)))))c1c2)))
All pretty simple:
Split each color into red, green, and blue integers with color-values.
Compute a mean for each color channel.
Build the new color by concatenating the hex components into a string.
In the example above:
The result of (color-values "red") is (65535 0 0).
The result of (color-values "blue") is (0 0 65535).
Their arithmetic mean is (32768 0 32768).
Divided by 256 with ash, we get (128 0 128).
Converted to hex, (128 0 128) is (80 0 80).
And the final color is "#800080".
Outro
I hope that you've enjoyed this little venture into Elisp and colors.
People say rough things about Elisp sometimes, but hey: it offers
functions ranging from the assembly-level ash, all the way up to
Photoshop-grade color channel handling.
If only it could get a good optimization bump, like JavaScript got at
one point, and a small threading library, we'd be all set.
“With great power comes great responsibility,” and Emacs is a prime example of
that. The versatility of having an editor that’s a lisp interpreter is truly
empowering, but it can also backfire on you in the most unexpected ways. If
you’ve ever ran into a foggy incompatibility issue between two unrelated
packages, that manifested itself by turning on your mother’s coffee machine every other
weekday, then you know how difficult this can be to track down.
One recurring theme on Emacs.StackExchange is that users will come to us with
some random arcane issue, and either provide no more information or dump their
init file on the question. The best answer we can give in these cases is for
them to bisect their init file. But if that’s always the case, why not automate
it?
The Bug Hunter is an Emacs library that does that for you.
Hunting real errors
If there’s an error being thrown during initialization, all it takes is a single
command.
M-x bug-hunter-init-file RET RET
The Bug-Hunter will do a bisection search in your init file for the source of
the error. Thanks to the magic powers of bisection, it is surprisingly fast even
on huge init files.
Hunting unexpected behaviour
If no actual error is being thrown, but some behaviour is still clearly wrong,
then it’s a little more tricky: you need to come up with an assertion. That is,
you need a snippet of Emacs-Lisp code that will return t if something is wrong
and nil if all is fine.
For instance, I wanted to figure out why the cl library was being loaded even
though I didn’t explicitly require it anywhere. In this case, the snippet
(featurep 'cl) gives me what I need. It returns nil before the library is
loaded, and returns t afterwards.
M-x bug-hunter-init-file RET (featurep 'cl) RET
Interactive debugging
Unfortunately this is not supported yet. Communicating with a background Emacs
process that is not in batch mode is complicated.
Usually, though, you shouldn’t need it. There’s almost always a snippet that
will work for your needs. If your problem is not triggering an error, and you
don’t know enough Elisp to write an assertion, let me know about your problem
and maybe I can write one for you.
Even better, shoot us a question over at Emacs.StackExchange. We’re always glad
to help.
I noticed that the packages that I've published on
Github in the last year did get a nice
boost in quality just from being published, viewed, used and commented
on.
So it makes sense to do the same for my full Emacs config, which lives in the oremacs repository as of today.
The config is personal in the sense that I'm actually using it
verbatim, but it's been configured in a way that's it's easy to
replicate and modify it.
Who can benefit from this config
The audience is people like me, who like to tinker with Elisp just for
fun of it. If you want things to just work, or you find learning
Elisp a chore, it's not for you.
The config won't work initially, it won't work always, but when it
finally does work (hopefully), it will be glorious!
Installation and Running
Requirements
Emacs 24 is required. Obviously, newer versions are better, but the
default emacs24 that you get from the package manager should work.
I'm currently switching between Emacs 24.5.2 built from source and the
current master from git.
Install command
This config doesn't assume to become your main config when you install
it. It installs in-place in the git directory and will start from
there without touching your main config. But you still get access to
all your stuff, like bookmarks stored in your actual ~/.emacs.d/
etc.
cd ~/git
git clone https://github.com/abo-abo/oremacs
cd oremacs
make install
Run command
Run without updating:
make run
Run with an upstream + ELPA update:
make up
Run with an upstream + ELPA + org-mode + CEDET update:
make install
Perks
Standalone
You can try it without messing up your current Emacs config. I actually have multiple versions of this on my system to work-around incompatibility between versions. This way, I can use my full setup even in case I get a bug report for an older Emacs version.
Fast start up
With a SSD, it starts in 1 second. Most features are autoloaded and it's easy to add new autoloaded features.
Tracks the most recent org-mode and CEDET
Since these packages take a long time to byte compile, they are updated not with make up but with make install. They are actually git submodules, which means that they won't update if I don't update them in the upstream.
Bankruptcy-proof
It's hard to become Emacs-bankrupt with this config, since the config is composed of many independent pieces that you can simply ignore if you don't need them.
Anti-RSI QWERTY mod
The config comes with its own .Xmodmap that makes ; into an additional modifier. RSI savers:
;-v instead of Enter.
;-o instead of Backspace.
;-f instead of Shift-9 and Shift-0.
;-a instead of -.
;-s instead of Shift--.
;-q instead of Shift-'.
;-e instead of =.
;-u in addition / instead of C-u.
And obviously the replacements for the two keys that the mod takes away:
;-j instead of ;.
;-d instead of Shift-;.
One more Elisp-level RSI-saver is the swap between C-p and C-h. Moving up/down line is very important, and it's nice to have these keys close, which C-n and C-h are.
It also includes:
a bunch of Hydras that save you key strokes.
lispy-mode which quickens any LISP mode, especially Elisp.
worf-mode which quickens org-mode.
ivy-mode which quickens all completion.
swiper which quickens isearch (by replacing it).
C/C++ is customized with function-args-mode and a bunch of hacks.
Org mode starter
The config starts you off with a fully configured org-mode setup that includes:
gtd.org for getting things done.
ent.org to track entertainment.
wiki folder for quickly starting and selecting wikis.
Outro
I hope that you try and enjoy the new config. Perhaps not verbatim,
but if you find the tips on this blog helpful, this config is their
actual implementation that you can use for reference.
This release packs more than 90 commits, which is quite a lot for me.
The most important part of this release is ivy-mode, which is a nice alternative to ido-mode or helm-mode.
I've mentioned it already in an earlier post,
I'll just review all the little details, changes and customizations here.
Important: remove the old ivy package
One important change related to MELPA is that the ivy package was merged into swiper package. So if you still have a stand-alone ivy package, you should delete it, or you'll get incompatibility problems.
Video Demo of counsel-git-grep
If you like videos, you can watch the this quick demo
which mostly shows off the new counsel-git-grep function.
Fixes
Add work-around for window-start being not current
From now on, you won't encounter some un-highlighted matches when your window is scrolled.
Make thing-at-point work
C-h v and C-h f should select thing-at-point properly.
Don't try to fontify huge buffers
It's a nice feature of swiper that everything in the minibuffer is fontified. However, this can cause a slowdown for buffers with x10000 lines. So this feature is automatically turned off for these large buffers.
Exclude a few modes from font locking
Some modes just misbehave when font-lock-ensure is called. Excluded:
org-agenda-mode
dired-mode
New Features
ivy-mode: complete everything with Ivy
ivy-mode uses swiper's approach to completion for almost all completion in Emacs, like:
execute-extended-command
package-install
find-file
See the wiki page for the details on the key bindings related to ivy-mode, which are especially important to know for find-file. Also see the intro video.
New Counsel functions
counsel-describe-variable - replacement for C-h v.
counsel-describe-function - replacement for C-h f.
counsel-info-lookup-symbol - just a wrapper around info-lookup-symbol, you get the same behavior by just calling info-lookup-symbol with ivy-mode on.
counsel-unicode-char - replacement for ucs-insert.
counsel-git-grep
This is a really cool command for grepping at once all the files in your current git repository. For smaller repositories (<20000) lines, ivy handles the completion by itself. For larger repositories, it defers the work to git grep. It works really well for the Emacs repo with its 3,000,000 lines, especially if you're using Emacs 25 (from emacs-snapshot or self-built).
This function makes use of C-M-n and C-M-p bindings, which switch between candidates without exiting the minibuffer. Also, they highlight the current candidate with the swiper faces. You can think of this command as multi-swiper.
Even for very large repos, it will always display the amount of matches correctly. Also note that swiper-style regex is used here (spaces are wild), and the case is ignored.
The arrows can take numeric arguments
C-n / C-p, C-s / C-r, and C-M-n / C-M-p can all take numeric args, e.g. M-5 or C-u.
Add a recenter binding
C-l will recenter the window in which swiper was called.
Look up Ivy key bindings with C-h m
While in the minibuffer, press C-h m to see the active modes. Ivy also has a paragraph with its bindings.
Use C-v and M-v to scroll
You can use these bindings to speed up your minibuffer scrolling.
Allow C-. to jump-to-definition
For counsel-describe-variable and counsel-describe-function:
pressing C-m will actually describe the current candidate.
pressing C-. will instead jump to definition of the the current candidate.
This is very useful for me, I jump to definitions more often that describe.
Bind arrows
The actual arrow keys are also bound to the corresponding Emacs arrows.
Add a way to exit ignoring the candidates
If your current input matches a candidate, C-m and C-j will exit with that candidate. If you want to exit with your exact input instead, press C-u C-j. This is very useful for creating new files and directories.
Use M-q to toggle regexp-quote
This can be useful while completing file names (with a lot of dots). Pressing M-q toggles between regex and non-regex matching.
Customize ivy-re-builders-alist
This is pretty intricate, look up the variable to see the details. In short, you can use this to apply `regexp-quote' for some types of completion if you want.
Customize what to do on DEL error
The standard behavior on a DEL error (usually empty minibuffer) is to exit the minibuffer. I like this behavior, since it's similar to the behavior to fixing wrongly typed chars: only in this case DEL "fixes" a wrongly typed command.
But if you want to customize it, just set ivy-on-del-error-function to something other than minibuffer-keyboard-quit.
Customize ivy-extra-directories
Customize this if you don't want to see ../ and ./ while completing file names.
Customize ivy-sort-functions-alist
Use this variable to customize sorting, depending on what you're completing. For example:
I’ve never been too pleased with the default behaviour of Backspace during
isearch. If the last key you hit was C-s, then it does the same as C-r
(albeit with less repetition), and if your match failed several characters ago,
you need to hit it that many times to get back on track. Fortunately, asmeurer
took the time to phrase this problem I barely realised I had.
delete the entire portion of isearch string that doesn’t match,
if everything matches, fallback on deleting last char instead of moving backwards.
Finally, John Mastro has yet another improvement for the command, which is the
one I’m using now. I’ll let you follow the link for the code, and just give you
the keybind you need for it to work.
Last weekend I created a new Emacs mode, nasm-mode,
for editing Netwide Assembler (NASM) x86 assembly programs.
Over the past week I tweaked it until it felt comfortable enough to
share on MELPA. It’s got what you’d expect from a standard
Emacs programming language mode: syntax highlighting, automatic
indentation, and imenu support. It’s not a full parser, but it knows
all of NASM’s instructions and directives.
Until recently I didn’t really have preferences about x86 assemblers
(GAS, NASM, YASM, FASM, MASM, etc.) or syntax
(Intel, AT&T). I stuck to the GNU Assembler (GAS) since it’s already
there with all the other GNU development tools I know and love, and
it’s required for inline assembly in GCC. However, nasm-mode now marks
my commitment to NASM as my primary x86 assembler.
Why NASM?
I need an assembler that can assemble 16-bit code (8086, 8088, 80186,
80286), because real mode is fun. Despite its .code16gcc
directive, GAS is not suitable for this purpose. It’s just enough to
get the CPU into protected mode — as needed when writing an operating
system with GCC — and that’s it. A different assembler is required
for serious 16-bit programming.
GAS syntax has problems. I’m not talking about the argument
order (source first or destination first), since there’s no right
answer to that one. The linked article covers a number of problems,
with these being the big ones for me:
The use of % sigils on all registers is tedious. I’m sure it’s
handy when generating code, where it becomes a register namespace,
but it’s annoying to write.
Integer constants are an easy source of bugs. Forget the $ and
suddenly you’re doing absolute memory access, which is a poor
default. NASM simplifies this by using brackets [] for all such
“dereferences.”
GAS cannot produce pure binaries — raw machine code without any
headers or container (ELF, COFF, PE). Pure binaries are useful for
developing shellcode, bootloaders, 16-bit COM programs,
and just-in-time compilers.
Being a portable assembler, GAS is the jack of all instruction sets,
master of none. If I’m going to write a lot of x86 assembly, I want a
tool specialized for the job.
YASM
I also looked at YASM, a rewrite of NASM. It supports 16-bit assembly
and mostly uses NASM syntax. In my research I found that NASM used to
lag behind in features due to slower development, which is what
spawned YASM. In recent years this seems to have flipped around, with
YASM lagging behind. If you’re using YASM, nasm-mode should work
pretty well for you, since it’s still very similar.
YASM optionally supports GAS syntax, but this reintroduces almost all
of GAS’s problems. Even YASM’s improvements (i.e. its ORG directive)
become broken when switching to GAS syntax.
FASM
FASM is the “flat assembler,” an assembler written in assembly
language. This means it’s only available on x86 platforms. While I
don’t really plan on developing x86 assembly on a Raspberry Pi, I’d
rather not limit my options! I already regard 16-bit DOS programming
as a form of embedded programming, and this may very well extend to
the rest of x86 someday.
Also, it hasn’t made its way into the various Linux distribution
package repositories, including Debian, so it’s already at a
disadvantage for me.
MASM
This is Microsoft’s assembler that comes with Visual Studio. Windows
only and not open source, this is in no way a serious consideration.
But since NASM’s syntax was originally derived from MASM, it’s worth
mentioning. NASM takes the good parts of MASM and fixes the
mistakes (such as the offset operator). It’s different enough
that nasm-mode would not work well with MASM.
NASM
It’s not perfect, but it’s got an excellent manual, it’s a
solid program that does exactly what it says it will do, has a
powerful macro system, great 16-bit support, highly portable, easy to
build, and its semantics and syntax has been carefully considered. It
also comes with a simple, pure binary disassembler (ndisasm). In
retrospect it seems like an obvious choice!
My one complaint would be that it’s that it’s too flexible about
labels. The colon on labels is optional, which can lead to subtle
bugs. NASM will warn about this under some conditions (orphan-labels).
Combined with the preprocessor, the difference between a macro and a
label is ambiguous, short of re-implementing the entire preprocessor
in Emacs Lisp.
Why nasm-mode?
Emacs comes with an asm-mode for editing assembly code for various
architectures. Unfortunately it’s another jack-of-all-trades that’s
not very good. More so, it doesn’t follow Emacs’ normal editing
conventions, having unusual automatic indentation and self-insertion
behaviors. It’s what prompted me to make nasm-mode.
To be fair, I don’t think it’s possible to write a major mode that
covers many different instruction set architectures. Each architecture
has its own quirks and oddities that essentially makes gives it a
unique language. This is especially true with x86, which, from its 37
year tenure touched by so many different vendors, comes in a number of
incompatible flavors. Each assembler/architecture pair needs its own
major mode. I hope I just wrote NASM’s.
One area where I’m still stuck is that I can’t find an x86 style
guide. It’s easy to find half a dozen style guides of varying
authority for any programming language that’s more than 10 years old
… except x86. There’s no obvious answer when it comes to automatic
indentation. How are comments formatted and indented? How are
instructions aligned? Should labels be on the same line as the
instruction? Should labels require a colon? (I’ve decided this is
“yes.”) What about long label names? How are function
prototypes/signatures documented? (The mode could take advantage of
such a standard, a la ElDoc.) It seems everyone uses their own style.
This is another conundrum for a generic asm-mode.
There are a couple of other nasm-modes floating around with
different levels of completeness. Mine should supersede these, and
will be much easier to maintain into the future as NASM evolves.
Just got this request a few minutes ago, and now this feature is in
the swiper
repository (available as counsel from MELPA).
The motivation was to write an ivy equivalent of helm-git-grep.
I didn't use this feature before, but the only thing that I needed to get me started was this shell command:
The rest of the code (just 20 lines) followed all by itself:
(defuncounsel-git-grep-function(string&optional_pred&rest_u)"Grep in the current git repository for STRING."(split-string(shell-command-to-string(format"git --no-pager grep --full-name -n --no-color -i -e \"%s\""string))"\n"t))(defuncounsel-git-grep()"Grep for a string in the current git repository."(interactive)(let((default-directory(locate-dominating-filedefault-directory".git"))(val(ivy-read"pattern: "'counsel-git-grep-function))lst)(whenval(setqlst(split-stringval":"))(find-file(carlst))(goto-char(point-min))(forward-line(1-(string-to-number(cadrlst)))))))
Thanks to the push from Stefan Monnier, ivy-read also supports a
function to be passed instead of a static collection of strings. In
this case, it's counsel-git-grep-function that basically takes one
argument: the thing that we're looking for.
After this, shell-command-to-string is my go-to function to quickly
bring some shell output into Elisp. As you can see, it's enough to
pass it a shell command in a string form to get a string response. I
transform the response into a list of line strings with
split-string, making sure to pass the t argument to avoid empty
strings.
One final trick that you can learn for your own Elisp programming is
the let / default-directory / (locate-dominating-file default-directory ".git") combo.
It's quite useful for dealing with git shell commands.
And that's it, it only remains to open a file and jump to the selected line.
I think that counsel-git-grep might complement and slightly displace rgrep or ag in my setup.
So I've given it a nice binding:
(global-set-key(kbd"C-c j")'counsel-git-grep)
I hope that I've made a good case of how easy it is to quickly write
something in Elisp, especially if it's just a shell command
wrapper. So if you're on the fence of whether to learn Elisp or not,
do yourself a favor and learn it: it pays off quickly and is a lot of
fun.
Side note: I've mentioned \bfun\b in this blog 182 times, mostly as a variable representing a function (courtesy of counsel-git-grep).
I am a search-guy: when I want to know something, I use the search functionality to locate to where has the keyword, and I didn't use my eyes to scan the page, it's too slow and harmful.
Emacs provides powerful functionality to do searching. For example, I use these commands very often (with the key-binds),
isearch (C-s), search for a string and move the cursor to there,
helm-swoop (C-F1), find all the occurrences of a string, pull out the lines containing the string to another buffer where I can edit and save,
helm-multi-swoopM-X, apply helm-swoop to multiple buffers, very handy if I want to know where a function is called in different buffers.
projectile-grep or helm-projectile-grepC p s g, find which files in current project contains a specific string, similar to helm-multi-swoop limits the search to files in project directory.
I love doing searching in Emacs, but the problem is to have to remember all the key-binds for different tasks. Also, sometimes, I forgot about what alternatives I have and usually go with the one that I most familiar with, which usually means not the right one. I sometimes realise I use isearch multiple times to do what ace-jump-word-mode can achieve by just once.
Org-mode Hydras incoming! gives me some idea to group all these functions together, and press a single key to perform different tasks, so this can free my mind from remembering all the key-binds. Also, I can write the few lines of text to reminds myself when to do what, and this potentially can solve problem two.
So next time, when I want to search something, I just press F4, and then it brings up all the choices I have, and I don't need to worry about the key-binds or which to use! That's cool!
I am looking forward simplifying my Emacs workflow using hydra package, the key challenge is to identify the logical similarities among the tasks and then group them together accordingly. For hydra-search(), it is "search something on somewhere".
Version control system is a complex issues, and hard to understand the idea of branching and different types of merging. I merely understand the basic of Git, and it already makes my life a lot easier, I am managing about 10 repositories at this moment without much effort.
But my collages are using SVN as the centre storage for scripts. Switching to SVN is not a problem, I just need few weeks to transfer the knowledge and start to use it. I am reluctant to learn something basic and have duplicated knowledge, also, I use GitHub and Bitbucket which are Git based. But sticking to Git make mine work impossible to work with collauges.
Then I found out the Git developer has already made effort to bridge Git and other version control system, like SVN. The git svn allows me to just Git commands for staging, cherry-picking, pull etc, and then upload to the SVN remote repository with just one command line. I really like the idea of transferring the skills from one system to another without any cost, it makes me believe Git is great and I can continue to use Magit in Emacs!
Here is the basic steps and comments for this work flow:
Today I'd like to introduce a function that has completely replaced
ido-mode for me: ivy-mode. It comes with the
swiper package, available both in
MELPA and
GNU ELPA.
The latter is a bit slower to update, so the version there can be a bit outdated sometimes.
The quick video demo
If you prefer listening and watching to reading, check out the video
demo here.
It repeats most of the stuff written down below.
Generic completion
ivy-mode is simply a minor mode that changes
completing-read-function to ivy-completing-read while it's active.
This function is used in most of the places where Emacs requires
completion, like:
M-xexecute-extended-command
C-h fdescribe-function
C-h vdescribe-variable
package-install
C-x bswitch-to-buffer
C-x C-ffind-file
So, while ivy-mode is on, all these functions and more will use ivy completion.
Package-specific completion
org-mode
With the most recent master version, org-mode will obey
completing-read-function, so it should work by default. If you try
it for refiling to headings with similar names, you'll really notice
how much better ivy-mode is at it. helm-mode also does well, if
you don't mind the large window.
Yes, it's also possible, since today. Although you'll have to use my
fork of smex if you want to try it.
I've sent a pull request, it's all backwards-compatible, so hopefully
it'll get merged.
The nice thing is that smex can take care of the sorting all by
itself, since ivy doesn't do that yet.
my packages
lispy and
function-args use ivy by
default. You can also enable it for
helm-make, for which the
default is obviously helm.
File name completion
When ivy-mode is on, find-file will also use it. The completion
is considerably different from all other cases, since it's done in
stages, just like ido-find-file does it.
The key bindings are:
RET will select the current candidate and finish.
C-j will try to continue the completion, i.e. if the
current candidate is a directory, move to that directory. But if the
current candidate is a file or ./, then finish.
/ will switch to completing the sub-directories of /,
but if the candidate is a perfect match, it will act like
C-j.
~ will switch to completing the sub-directories of ~/.
C-n and C-p naturally select the next and the previous candidate.
What's it all for?
Well, for me the advantage is obvious: I get my completion just the
way I like it. You can use it as well if you find that you like it
more than ido. I'll just list a list of features that I like:
The current number of candidates is robustly displayed and updated after each key stroke.
The minibuffer is an actual editing area, where the bindings like
C-a, M-DEL and C-k etc. work just as
you expect. Once you internalize the way that the regex is built, you
can get your match very quickly and intuitively.
The actual regular expressions constructs like \\b or $ and ^ work. The only thing that works differently
is the space: you can't match a single space because any amount of spaces translates into the .* wild card.
But you can use it to your advantage and use the space instead of all the various symbol joining constructs out there, like
snake_case, or kebab-case (yeah, it's totally called that, check the wiki), or whatever/this/is/case.
The familiar M-< and M-> key bindings also work as expected, navigating you to the first and last candidate respectively.
When you press C-p on the first match or C-n on the last match, the match will not change, unlike the behavior
of ido that has annoyed me a lot.
The minibuffer key bindings can actually be properly customized:
just set ivy-minibuffer-map to whatever you like, it won't be
changed. Even in Emacs 24.5, if you customize ido-completion-map,
the change will be reset. That was fixed only in the current master.
You don't have to rely on flx or flx-ido to save you from the overwhelming number of matches.
Instead, you type maybe a bit more, but in return you get a very consistent and predictable result.
No black boxes or hidden variables that dramatically change the order of candidates from time to time.
Outro
Check it out, I hope you like it. And a big thanks to all who
contributed code or bug reports.
A nice thing that guide-key has is that you can set an idle time to
wait until you press a key to activate any head before show the
hydra hints buffer (explained in hydra terms). This is very helpful
when you know the key bindings by memory and you do not need to see
the hints buffer, but if you forget how to activate any head, you
simple press the hydra binding and wait the "idle time" and the
hints buffer is shown to help you to choose the right next binding.
So this option is now also possible. Here's an example, using hydra-toggle from hydra-examples.el:
So the single change from the old code is :idle 1.0, which means:
After a call to hydra-toggle/body, instead of displaying the hint
as usual, start a timer for 1.0 seconds. Once the timer runs out,
display the hint. But if the hydra has exited before that time,
cancel the timer.
Typically, I'm not a huge fan of timers, but it's nice to have the
option. And indeed, I've been noticing that I could do without a hint
for some simple and more often used hydras. But a command can go from
often to barely used quickly, and disabling the hint altogether seems
too harsh. A timer could be a nice middle ground, especially since I
can decide whether to use it or not and the interval for each specific
hydra.
Anyway, thanks for the contributions. Enjoy the new feature, and keep
those good ideas coming!
This is a categorized version of the hydra that @WorldsEndless contributed
today to the wiki.
(defhydrahydra-global-org(:colorblue:hintnil)"Timer^^ ^Clock^ ^Capture^--------------------------------------------------s_t_art _w_ clock in _c_apture _s_top _o_ clock out _l_ast capture_r_eset _j_ clock goto_p_rint"("t"org-timer-start)("s"org-timer-stop);; Need to be at timer("r"org-timer-set-timer);; Print timer value to buffer("p"org-timer)("w"(org-clock-in'(4)))("o"org-clock-out);; Visit the clocked task from any buffer("j"org-clock-goto)("c"org-capture)("l"org-capture-goto-last-stored))
I've bound it like this:
(global-set-key[f11]'hydra-global-org/body)
Previously, I had f11 bound to org-clock-goto, thanks to
the Org Mode - Organize Your Life In Plain Text!
post. I kind of recommend that post, as it has a tonne of useful
stuff. At the same time, it resulted in me copying too much stuff that I didn't understand into my config,
ultimately discouraging me from using org-mode. YMMV.
Screenshot:
A hydra for refile/archive
Inspired by the previous one, I wanted to write a hydra of my own. As
it happened, I was just about to clean up my git notes. So I wanted a
nice refile hydra to speed up the process.
Here's the hydra's code, which is now part of
worf. Worf is a package for
quickly navigating around an org-mode buffer, you can get it from
MELPA.
worf-refile-this will give you a choice of this file's headings for refiling.
worf-refile-other will give you a choice of your org-refile-targets except the current file.
worf-refile-last will refile to the last refile location without prompting.
k will toggle org-refile-keep, which decides if refiling moves or copies the text.
And here's how it looks like:
Outro
Thanks for the contribution, enjoy the new stuff.
Also, if you ever want to dig around someone's git notes, you can read mine,
courtesy of org-mode export.
SX.el, the awesome Emacs client for the StackExchange network, has been stable
and happy for many months now, and it’s about time I mentioned it here. We have
put considerable effort into making the interface intuitive, teaching you how to
use it without the need for explanations. In fact, it would probably be a
disservice to the package for me to post a tutorial, so I won’t just yet.
Instead, I’ll just urge you to give it a try.
After installing it from Melpa, the only thing you need to do is authenticate
with M-xsx-authenticate. Once that is done, you’re ready to invoke any of
the entry commands.
The main command is sx-tab-all-questions. But that’s so many letters, we
simply must to bind it to a key. And if you read the post title, you know where
I’m going with this.
;; Ordered by frequency of use, for no particular reason.(define-keylauncher-map"qq"#'sx-tab-all-questions)(define-keylauncher-map"qi"#'sx-inbox)(define-keylauncher-map"qo"#'sx-open-link)(define-keylauncher-map"qu"#'sx-tab-unanswered-my-tags)(define-keylauncher-map"qa"#'sx-ask)(define-keylauncher-map"qs"#'sx-search)
Before signing off, I will also bring to your attention the magnitude of the
SX.el feature set. This is not just a cute toy that lets you browse or search
questions. The client has so many features it could take the website itself in a
fight. Voting, editing, commenting, answering, asking, deleting, inbox viewing,
and source code font-locking are all things that just popped into my head. And
that’s without mentioning a host of Emacs-oriented features like tabbing through
buttons, or folding (hiding) comments like org headlines.
I’m probably saying too much already, let’s leave some content for future posts.
If you have any questions or would like to give feedback, Sean, Jonathan and I
are always floating around our Gitter chat room (and we do love to hear
compliments).
A lot of the changes for this release aren't very user-visible, although they
are important in improving the usability. The main change is the move from the
standard set-transient-map to my own hydra-set-transient-map.
This change has allowed to remove the part where the amaranth and pink hydras
intercept a binding which doesn't belong to them and then try to forward it back
to Emacs. It was done in this way, which might be interesting for people who
write Elisp:
(define-keyhydra-keymapt'hydra-intercept)
Binding t means that when the keymap is active, any binding which doesn't
belong to the keymap will be interpreted as t. Then, I would use lookup-key
and this-command-keys and try to call the result. That method was quite
fragile:
It didn't work for prefix keys while a pink hydra was active.
It didn't work for some keys in the terminal because of input-decode-map.
The new method solves the mentioned issues by not using t and instead running
this function in the pre-command-hook:
(defunhydra--clearfun()"Disable the current Hydra unless `this-command' is a head."(if(memqthis-command'(handle-switch-framekeyboard-quit))(hydra-disable)(unless(eqthis-command(lookup-keyhydra-curr-map(this-single-command-keys)))(unless(cl-casehydra-curr-foreign-keys(warn(setqthis-command'hydra-amaranth-warn))(runt)(tnil))(hydra-disable)))))
This approach is actually very similar to what the built-in set-transient-map
does from Emacs 24.4 onward. Of course, changing the a large cog in the Hydra
mechanism can lead to some new bugs, or even old bugs to re-surface. So I really
appreciate the help from @jhonnyseven in testing the new code.
As always, if you find something very broken, you can roll back to the GNU ELPA version
and raise an issue.
Fixes
single command red/blue issue
Fix the uniqueness issue, when a single command is assigned to both a
red and a blue head.
Key chord will be disabled for the duration of a hydra
This means that hydras have become much more easy to use with key chords. For
instance, if dj key chord calls a hydra or is part of the hydra, you
won't call the jj key chord by accident with djj.
(defvarfoo"a b c")(defhydrabar()" bar %s`foo"("a"'t)("q"nil))
Previously, it would only work for %s(foo) forms.
:bind property can also be a keymap
If you remember, you can set :bind in the body to define in which way the
heads should be bound outside the Hydra. You also assign/override :bind for
each head. This is especially useful to set :bind to nil for a few heads that
you don't want to bind outside.
Previously, :bind could be either a lambda or nil. Now a keymap is also accepted.
Integration tests
In addition to the abundant macro-expansion tests, integration tests are now
also running, both for emacs24 and for emacs-snapshot. This means that hydra
should be a lot more stable now.
In the tests above, hydra-simple is a defined and bound hydra. "|"
represents the buffer text (empty), where | is the point position.
And (kbd "C-c aabbaaqaabbaa") represents the key sequence that you can normally press by hand.
Finally, "jjkkjjaabbaa|" is what the buffer and the point position should look like afterwards.
If you find a hydra bug, it would be really cool to submit a new integration test to make sure that
this bug doesn't happen in the future.
Basic error handling
I really like the use-package feature where it catches load-time errors and issues a message instead of bringing up the debugger. This is really useful, since it's hard to fix the bug with a mostly broken Emacs, in the case when the error happened early in the load process. So the same behavior now happens with defhydra. In case of an error, defhydra will be equivalent to a no-op, and the error message will be written to the *Messages* buffer.
Use a variable instead of a function for the hint
This leads up to the yet unresolved #86, which asks
for heads to be activated conditionally.
For now, you can modify the docstring on your own if you wish.
Here's some code from the expansion of hydra-zoom to explain what I mean:
(set(defvarhydra-zoom/hintnil"Dynamic hint for hydra-zoom.")'(format#("zoom: [g]: in, [l]: out."78(facehydra-face-red)1617(facehydra-face-red))));; in head body:(whenhydra-is-helpful(ifhydra-lv(lv-message(evalhydra-zoom/hint))(message(evalhydra-zoom/hint))))
Eventually, I'll add some automatic stuff to fix #86. But for now, you can
experiment with modifying e.g. hydra-zoom/hint inside heads, if you want.
Multiple inheritance for Hydra heads
Each hydra, e.g. hydra-foo will now declare its own heads as a variable
hydra-foo/heads. It's possible to inherit them like this:
Here, hydra-zoom-child inherits the heads of hydra-zoom and hydra-arrows.
It adds one more head q to quit. Also, it changes the color to
amaranth, which means that it's only possible to exit it with q or
C-g. This hydra's parents remain at their original (red) color.
A big thanks to all who contributed towards this release and to the wiki.
If you have an idea that would be cool in Hydra, do raise an issue.
And if you written some cool code that uses the already available Hydra features, please share it on the wiki.
It's also a good way to protect your code against regressions (although the best one would be an integration test).
Finally, check out the new version of
pandoc-mode which is
one of the coolest and most elaborate uses of Hydra yet.
If you like my package swiper,
I'm sure you'll also like counsel. It
lives in the same repository, but you can install it separately from
MELPA.
Counsel uses ivy - the same method as swiper to:
Complete Elisp at point with counsel-el.
Complete Clojure at point with counsel-clj.
Open a git-managed file with counsel-git.
Describe an Elisp variable with counsel-describe-variable.
Describe an Elisp function with counsel-describe-function.
Look up an Elisp symbol in the info with counsel-info-lookup-symbol.
Insert a Unicode character at point with counsel-unicode-char.
Below, I'll describe the functions that I added just today.
counsel-describe-function
This is just a replacement for describe-function:
As you can see, regular expressions work here as well. The
space-splitting behavior is the same as in swiper, so don't expect
to be able to match a single space: spaces are wild.
counsel-describe-variable
This is just a replacement for describe-variable:
Well, actually I've used ido-mode with flx matching for these two
functions before. In my experience, ivy handles much better:
you are in charge of which regex you're searching for
At around 40000 candidates, ivy starts to feel clunky (around 0.1s
display delay). Unfortunately, I don't really see a way around this,
except using tricks like while-no-input, which didn't work right
when I tried it earlier. It would be really cool to do the completion
in another thread, but alas.
Outro
Give the new functions a try. I think ivy can become a viable ido
replacement, at least it has done so for me: the only ido functions
that I'm still using are ido-switch-buffer and ido-find-file.
Also, if you're using projectile, you can use ivy completion for it:
Cluttered as our keyboards are with key-binds, it's always nice when we can
combine two disjoint functionalities in the same key. I have M-k and C-M-k
bound to killing the next and previous sexp, respectively, but that is never
something I need inside a string or when typing a file name in the minibuffer.
Then these keys become utterly useless!
The solution to this conundrum is, of course, to find another use for them in
these cases. Something that comes up regularly inside strings (or file-name
prompts) is killing an entire directory name.
The code below, binds M-k to a command that automatically decides whether to
kill a sexp or a directory name. The decision making is a little tricky, but it
has yet to let me down.
;; Note you may want to swap the following two keybinds.;; Emacs' default keymap has `kill-sexp' on `C-M-k'.(global-set-key(kbd"M-k")#'endless/forward-kill-sexp-or-dir)(global-set-key(kbd"C-M-k")#'endless/backward-kill-sexp-or-dir)(defunendless/forward-kill-sexp-or-dir(&optionalp)"Kill forward sexp or directory.
If inside a string or minibuffer, and if it looks like
we're typing a directory name, kill forward until the next
/. Otherwise, `kill-sexp'"(interactive"p")(if(<p0)(endless/backward-kill-sexp-or-dir(-p))(let((r(point)))(if(and(or(in-string-p)(minibuffer-window-active-p(selected-window)))(looking-at"[^[:blank:]\n\r]*[/\\\\]"))(progn(search-forward-regexp"[/\\\\]"nilnilp)(kill-regionr(point)))(kill-sexpp)))))(defunendless/backward-kill-sexp-or-dir(&optionalp)"Kill backwards sexp or directory."(interactive"p")(if(<p0)(endless/forward-kill-sexp-or-dir(-p))(let((r(point))(l(save-excursion(point))))(if(and(or(in-string-p)(minibuffer-window-active-p(selected-window)))(looking-back"[/\\\\][^[:blank:]\n\r]*"))(progn(backward-char)(search-backward-regexp"[/\\\\]"(point-min)nilp)(forward-char)(kill-region(point)l))(kill-sexp(-p))))))
When programming LISP, especially with lispy, it's easy to
generate random one-line expressions. This is, of course, because the results of read or eval
don't contain any whitespace information: all original newlines are lost.
Just to review the multitude of ways to insert generated code into a buffer using lispy I'll list
the shortcuts and the test-based explanations (at around 2000 lines of tests and 54% test coverage,
lispy is pretty well tested).
eval-and-insert
E calls lispy-eval-and-insert.
The image above is generated using the interactive test visualizer lispy-view-test, bound to
xv. If you want to explore how a certain command is intended to behave, just find the
corresponding test (with the same name as the command) and call xv.
eval-and-replace
xr calls lispy-eval-and-replace. This function evaluates the current expression and
replaces it with the result.
The sequence of actions in the test:
e calls lispy-eval to set foo to 42.
j calls lispy-down to move to the next sexp.
xr calls lispy-eval-and-replace.
Ideally, there should have been "xr" instead of (lispy-eval-and-replace) in the test, but
there's a small wrinkle in the lispy-with macro that needs to be fixed before that can happen.
flatten
xf calls lispy-flatten. This function expands in-place the current function or macro
call.
In this test, the misleadingly named function square is evaluated and flattened, to see if the
&optional and &rest argument passing rules indeed work.
The flatten operation works really well for Elisp and quite well for Clojure. The CL implementation
would need to heavily rely on SLIME features (currently absent), since the CL spec doesn't define an
equivalent of Elisp's symbol-function. The same applies to Scheme, I guess.
oneline
O calls lispy-oneline. It's not eval-based, it just deletes the newlines.
If there are any comments present, they are pushed out.
lispy-alt-multiline Demo 1
In the following image, I just press T once, starting from an unchanged buffer:
lispy-alt-multiline Demo 2
Flatten push
Start from this code (the cursor is in the CSS, if you don't see it):
Since push is a macro, macroexpand is used. And since macroexpand doesn't give newline
information, pp-to-string is used, and it gives a reasonable result.
Flatten dotimes
Start with the same code, but with cursor on dotimes this time:
Well, at least some parts look better. It could be make perfect by adding a sort of threshold when
printing each sub-expression. It's less than, say 15 chars, which (setq i (1+ i)) is, no
newlines should be added. I'll add this a bit later.
More on lispy-alt-multiline
lispy-alt-multiline can be used on a LISP expression to re-format it across multiple lines. It
doesn't matter in which shape the expression currently is, since all current newlines will be removed
before the algorithm starts.
This has to be done with some rules, since a one-line expression can transform to multiple viable
multi-line forms. So far, these rules are implemented by customizing these variables:
(defvarlispy--multiline-take-3'(defvardefundefmacrodefcustomdefgroup)"List of constructs for which the first 3 elements are on the first line.")(defvarlispy--multiline-take-2'(deffacedefine-minor-modecondition-casewhileincfcarcdr>>=<<=eqequalincfdecfcl-incfcl-decfcatchrequireprovidesetqconswhenifunlessinteractive)"List of constructs for which the first 2 elements are on the first line.")
The name suggests that there should be lispy-multiline, and there is, bound to M. The
difference between M and T is that M is older and ad-hoc, while
T is newer and rule-based. This means that the latter can misbehave, since it's not yet
fully tested. However, it has the following built-in check to make sure that it doesn't mess up your
code:
The read of the expression before transformation should be equal to the read of the
transformed expression.
If the above check fails, no change will be performed on the source code. So your code should be
pretty safe. One more cool thing that I want to add to other operations is that it checks if the
buffer will be changed after the transformation. If there will be no change, it will just issue a
"No change" message, and no change will be performed. This is really cool if you obsess about the
buffer changed marker in the mode line like I do.
Outro
The functions bound to O, M, and T apply to the current expression.
To be really sure which one, turn on show-paren-mode. You can also call these functions not from
special, although it's not very convenient. The typical strategy in that case would be to bind all
of them on a prefix map, e.g. C-c.
How is it different from special then?
Instead of typing [T you would type C-c T.
But the advantage of the special approach is that [
actually does something (moves point to the start of the current list),
instead of just being a useless part of a key combination like C-c.
And if you're in special already, there's no need for [.
I hope that you enjoy the new update. If it's needed, variables like lispy--multiline-take-3 can
be made buffer-local so that T works appropriately for Clojure, CL and Scheme, instead of
just Elisp. If you'd like to add support for your favorite dialect in this way, I'd be happy to
explain some details if needed and to merge the PR. Happy hacking!
I just finished this feature today. Almost all Paredit functionality is available in terms of
lispy's own functions. The only functions that I didn't
implement were the ones that I didn't find any use for: paredit-backslash (ugh, prompts),
paredit-comment-dwim (covered by regular ;), paredit-forward-down and
paredit-backward-down (both just move out of special, and throw a lot).
Setting the keymap theme
So if you ever thought that lispy could be a good idea, but the non-special key bindings are weird,
and you're too used to Paredit, you can now try this setting:
(lispy-set-key-theme'(specialpareditc-digits))
Here's the setting that I'm using, since I love lispy's default bindings:
(lispy-set-key-theme'(olehspeciallispyc-digits))
The default setting actually is this one, lispy-mode-map-oleh is an additional map for use with
my Xmodmap setup:
(lispy-set-key-theme'(speciallispyc-digits))
Each item in the list passed to lispy-set-key-theme designates a keymap to turn on.
So in a non-LISP buffer you could turn on just the special map:
(lispy-set-key-theme'(special))
This would mean that zero regular bindings get overridden, but when the point is before or after a
paren, you get the lispy bindings. Actually, I've just enabled lispy-mode for markdown-mode,
and it's working great. Unfortunately, I haven't yet figured out how to make a minor mode's keymap
buffer-local. Maybe someone reading has an idea.
The screencast
I went through all of Paredit tests and made ERT tests out of them, to make sure that the features
are actually working. I've recorded a short (<3 minutes)
screencast of implementing one of the tests. Have a
look, maybe you'll something interesting in my setup. There are two short pauses in the video,
that's just me thinking how to write down a function. My ERT and Magit workflows are also shown.
Outro
The new key-theme feature is meant to make lispy super-not-annoying. I think it's useful, since
using lispy in Paredit mode was annoying for me, so the opposite has to be true for people that
are used to Paredit. So if you notice a bug, please report it, bug
reports are very important for my non-annoyance agenda.
Finally, if you're super-devoted to Paredit and don't want to drop it, now you can easily have both
lispy-mode and paredit-mode on. Just use this setting:
(lispy-set-key-theme'(special))
Or this one if you additionally want C-1 to show the inline doc, and C-2 to show
the inline args:
Add minibuffer-inactive-mode to the lispy-elisp-modes list. It means that you can eval there if you want.
V (lispy-visit) should turn on projectile-global-mode if it's not on.
M (lispy-multiline) works better for Clojure: the regexes for vectors, maps and sets have been improved.
C-k should not delete only the string when located at start of string.
M will not turn vectors into lists any more.
the backquote bug for i and M was fixed.
you can flatten Elisp closures as well, at least the plain ones.
New Features
b calls lispy-back
The movement commands, such as:
the arrows hjkl (lispy-left, lispy-down etc.)
f (lispy-flow)
q (lispy-ace-paren)
i (lispy-tab), only when called for an active region
will not store each movement in the point-and-mark history. You can press b to go back in
history. This is especially useful for h, l, and f, since they are
not trivially reversible.
b was previously bound to lispy-store-region-and-buffer, so you could do Ediff with
b and B. Now it's bound to xB.
Hungry comment delete
C-d (lispy-delete) when positioned at the start of a comment, and with only whitespace
before the start of the line, will delete the whole comment.
If you want to un-comment, just use C-u ; from any point in the comment.
Added flatten operation for Clojure
xf (lispy-flatten) now also works for Clojure, before it was only for Elisp.
A bit of a gibberish, but at least we can confirm that map is indeed lazy.
Example 3 (flatten your own function):
Example function:
(defn sqr[x](* xx))
This one requires the function to be properly loaded with C-c C-l (cider-load-file),
otherwise Clojure will not know the location of the function.
Example statement:
(+ |(sqr10)20)
After xf:
(+ |(let [x10](* xx))20)
Added lax eval for Clojure
This is similar to the lax eval for Elisp. If you mark an expression with a region:
asdf [1 2 3]
and press e, you will actually eval this:
(do (def asdf[123])asdf)
You can do this for let bindings, it's super-useful for debugging. The rule is that if the first
element of the region is a symbol, and there's more stuff in the region besides the symbol, a lax
eval will be performed.
e will auto-start CIDER
If CIDER isn't live, e will start it and properly eval the current statement.
2F will search for variables first
Since Elisp is a LISP-2, there can be a function and a variable with the same name. F
(lispy-follow) prefers functions, but now 2F will prefer variables.
2e will eval and insert the commented result
Starting with:
|(filter odd?(map sqr[12345]))
Pressing 2e gives:
(filter odd?(map sqr[12345]));; =>;; (1 9 25)
This works for all dialects, so you can also have:
To do the last eval you need to be in special.
It means that you first have to mark the symbol *MODULES* with a region.
A convenient function to mark the current symbol is M-m (lispy-mark-symbol).
y (lispy-occur) now has an ivy back end
lispy-occur launches an interactive search within the current top-level expression, usually a
defun. This is useful to see where a variable is used in a function, or to quickly navigate to a
statement.
You can customize lispy-occur-backend to either ivy (the default) or helm (if you have it,
since it's no longer a dependency of lispy).
Add ivy back end to lispy-completion-method
Now it's the default one for navigating to tags. You can select alternatively helm or ido if you
wish.
Remove the dependency on ace-jump-mode
Instead the dependency on ace-window will be re-used. This allows for a lot of code
simplifications and better tests.
New custom variables:
lispy-avy-style-char: choose where the overlay appears for Q (lispy-ace-char)
lispy-avy-style-paren: choose where the overlay appears for q (lispy-ace-paren)
lispy-avy-style-symbol: choose where the overlay appears for a (lispy-ace-symbol)
and - (lispy-ace-subword) and H (ace-symbol-replace).
There's also lispy-avy-keys, which is a ... z by default.
Add lispy-compat
This is a list of compatibility features with other packages, such as edebug and god-mode.
They add overhead, so you might want to turn them off if you don't use the mentioned packages.
F works for Scheme
You can navigate to a symbol definition in Scheme with F. This feature was already
in place for Elisp, Clojure and CL.
Outro
Looks like a great batch of features, but I'm the most happy about the stupid backquote bug being
fixed.
Up next: pre-defined key binding themes, featuring:
the beloved default.
the old-school paredit.
the special-only minimal.
and maybe one more.
Feel free to chime in, if you've got a key setup that you think other people will like. Happy
hacking!
Over the last couple of weeks, I’ve been working on a Clojure debugger for Cider
that is strongly inspired by Edebug. Stepping trough code of any form and
injecting values into running code are the features available in this first
release.
This little project started shortly after I my first practical delve into
Clojure. I was impressed at how well Cider integrated Clojure into Emacs, it
almost felt like writing Elisp! When Bozhidar mentioned that they were in need
of a debugger, I figured that would be as a good a chance as any to practice some
Clojure. Since my praise for Edebug is no secret around here, it should come as
no surprise that I’d model cider-debug after it (albeit, a bit more limited).
I should also mention the clj-debugger plugin, which we don’t use anymore but
was a great help while writing this feature.
Usage
First, if you’re still using cider 0.8, you’ll need to upgrade to the latest
stable version cider 0.9, which is available on Melpa. Additionally, ensure
your “~/.lein/profiles.clj” file is consistent with that.
{:user{:plugins[[cider/cider-nrepl"0.9.0"]]}}
Once that is done, using cider-debug could hardly be more straightforward.
Just instrument an expression with C-u C-M-x, and you’ll be taken through the
code step-by-step. There, you can move on with n, quit with q, or inject
values with i. Note that if you instrument a defn the debugger will not
start immediately, instead, you’ll be taken to the debugger each time the
function is executed.
This sort of thing is easier to explain with a gif.
We’d very much appreciate the feedback of any Clojurists out there. File an
issue if you spot something, or just share your thoughts with us on Twitter or
Gitter.
Update <2015-06-16 Tue>
That screen-cast above is a little outdated now. Since it was recorded, the
debugger has acquired even more features:
listing local variables,
evaluating code in the current lexical scope,
injecting values into the code,
a move-out command similar to Edebug.
I’ll make another post eventually, but it’s perfectly intuitive and easy to use.
So give it a try yourself if you use cider.
This post might be interesting to people who use Travis CI to test their Elisp packages.
I've noticed a few days ago that my tests on Travis CI started failing randomly. Turns out that the
widely used ppa:cassou/emacs has become
deprecated. It still sort of works, but sometimes apt-get update just times out, and I get a build
error.
Since it was the only one that I was using, I had to find another one. I went with emacs-snapshot
from ppa:ubuntu-elisp.
I've adopted one more change in the above code: EMACS is replaced by emacs.
The reason for this is that both ansi-term and compile
for legacy reasons set EMACS to strange values,
like "24.4.91.2 (term:0.96)".
So if you have in your Makefile:
EMACS ?= emacs
you're going to have a bad time. So I changed my Makefile to:
emacs ?= emacs
And now I can compile and test while inside Emacs with no issues. The only issue left is that I
don't yet know which PPA I should use for a stable Emacs. Maybe someone has a suggestion.
Yesterday, I've added a new binding to swiper-map: C-l will now call
swiper-recenter-top-bottom.
The implementation is really easy, almost nothing to write home about:
(defunswiper-recenter-top-bottom(&optionalarg)"Call (`recenter-top-bottom' ARG) in `swiper--window'."(interactive"P")(with-selected-windowswiper--window(recenter-top-bottomarg)))
An interesting thing that I want to mention though is the customization of the default
recenter-top-bottom behavior. This is the default one:
(setqrecenter-positions'(middletopbottom))
And this is the logical one that I'm using:
(setqrecenter-positions'(topmiddlebottom))
Try it out, and see if it makes sense to you. For me, when I've just jumped to a function
definition, which usually means that the point is on the first line of the function, the first
recenter position has to be top, since that will maximize the amount of the function body that's
displayed on the screen.
Another use-case is when I'm reading an info or a web page. After a recenter to top, all that I have
read is scrolled out of view, and I can continue from the top.
thoughtbot NYC
1st floor of the WeWork at Bryant Park
54 W. 40th St.
New York, NY
Steve B. from the OSFDA will be presenting Dictionary Lookups:
Parsing HTML with Emacs Lisp:
Steve goes into the Emacs trenches and shows how to use elisp to
reformat HTML markup from the Free Dictionary (one of the finest
open dictionaries and thesauri on the Internet), so now you can
directly query it when you’re at a loss for words in Emacs.
After the presentation, the code will be released for free download
to anyone wanting to use it. Steve is an avid digital currency
application developer, and has used Emacs since 1982 (including Gold
Hill lisp…)
There's actually an outstanding issue to make the hydra
heads appear conditionally. This would be quite useful for the m and , bindings, since
they don't work outside a dired buffer. Maybe I'll get to it on the weekend.
Meanwhile, here's a screenshot for marking files in dired using swiper:
Since the input is "mar 10", swiper transforms it into the regex "(mar).*(10)".
What I do next:
C-o to get into the hydra state.
99m to mark everything. Normally, it would mark 99 candidates, but since there are only 17, that means all of them.
h to go to the first candidate.
j, to to skip one candidate and unmark, then unmark some more, using this method.
If I wanted to move two candidates down at once, I could press 2j...... The .
will repeat the previous command with the previous argument. You can also set the argument later,
e.g. j.2.3.. etc.
The exit points
There are two:
i will bring you back to ivy, so that you can edit the input.
o will quit everything and bring you to the dired buffer.
So you could first mark Mar 10, exit with i, edit the input to Mar 17, press
C-o and mark some more. Then finally exit with o.
The replace and undo operations
These two I've added the latest, so they are still a bit off. The ivy-replace option is similar
to vim's r (I did vimtutor yesterday). It lets you replace the selected candidate. And
u simply calls undo. Strangely, at the moment it will undo several ivy-replace
operations at once, even though I call undo-boundary in ivy-replace.
Outro
I think ivy and the hydra docstring blend in together quite nicely, like
old dogs and new tricks. I don't know which is which.
I forgot to mark the 0.1.0 release, so I'm giving an overview of all the fixes and new features
since the first commit in the release notes.
Fixes
Fix font locking in certain modes
Some major modes try to optimize the font locking (highlighting text with various faces) by only
doing it for the visible portion of the text. But since swiper needs to access all lines at once,
it's necessary to font lock the whole buffer. This is done in swiper-font-lock-ensure. For some
modes, the buffer becomes discolored after calling swiper-font-lock-ensure. In theory, this should
not happen. As a work-around, I exclude these modes from ensuring font lock:
package-menu-mode
gnus-summary-mode
gnus-article-mode
gnus-group-mode
emms-playlist-mode
erc-mode
If you see a discoloration in one of your favorite major modes while using swiper, just let me
know and I'll add it to the list.
Fix face changes in the minibuffer propagating to the main buffer
This was a quite interesting bug. At that moment, I was using add-face-text-property to add faces to the copies of strings in the minibuffer. However, this function destructively modifies the properties, so the change to the properties of a string copy (obtained with concat or copy-sequence) was propagated to the properties of the original string. This was fixed by using font-lock-append-text-property instead of add-face-text-property.
ivy-read returns immediately for 0-1 candidates
An obvious improvement.
Clean up overlays better for C-g
The hidden overlays revealed during the search will be re-hidden if you cancel the search with
C-g.
Ensure that candidates don't have read-only property
This issue was causing a bug while using swiper in erc-mode, since it marks all of the buffer
content to read-only. So once a string with read-only property is inserted into the minibuffer,
you can't delete it unless you set inhibit-read-only.
New Features
Optional initial input
If you call (swiper "fix"), you'll start searching with initial input "fix".
Restore the initial point on canceling
If you cancel the search with e.g. C-g (or DEL when there's no input), the
initial point will be restored.
Inherit standard faces
To give a more standard default appearance, swiper faces inherit the default faces:
isearch-lazy-highlight-face
isearch
match
isearch again
highlight
Most themes customize these faces, so by re-using them swiper blends in better.
If you want the cool (my opinion) original faces, have a look at eclipse-theme.
Reveal invisible overlays
This is quite important for searches in org-mode buffers. I tried to make it as close as possible
to what isearch is doing.
Mark is saved for successful searches
This is the behavior of isearch. After you complete a search, you can go back to the search start with
C-x C-SPC (pop-global-mark).
The current candidate is anchored to the current position
This means that if many candidates are matching the current input, the one which is closest to the
current line (going forwards) is selected. This is important for not losing the context of what
you're searching.
Don't recenter unless necessary
This is similar to the behavior of isearch: a scroll is performed only if the candidate is out of
the window bounds. An alternate strategy of keeping the current candidate always centered in the
window is more distracting.
Decouple helm back end
swiper command uses only ivy now. If you want to use helm, have a look at
swiper-helm.
Add history handling
M-n will select the next history element, and M-p will select the previous
history element.
When there is no input, both C-s and C-r will select the last history element.
This is to make it similar to isearch.
When there is no input, and only once during the search, M-n will select symbol-at-point
as the current input.
Truncate candidates to window width in the minibuffer
If a candidate is longer than the window width, it will be appropriately truncated with ....
You can still match the invisible parts.
Warn for empty buffer
Obviously there's nothing to search for in an empty buffer.
Bring the last history candidate to front
In case of a successful search, the current input will be removed from history, and then re-added to
the front.
Make C-n and C-p differ from C-s and C-r
The arrows will not recall the last history element in case the input is empty.
Otherwise, C-n matches to C-s and C-p matches to C-r.
ivy-read now displays the number of candidates in the prompt
The prompt argument can hold a format-style expression, e.g. " [% 3d] pattern:", and the integer
parameter will be updated with the current amount of matches.
You can also customize ivy-count-format, that defaults to "%-4d ".
Custom amount of chars to start highlighting
Customize swiper-min-highlight for this. It defaults to 2, which means that the current buffer
will be highlighted when the input has 2 chars or more. You can set it to 1 if you want, I found
it slightly distracting, since there will be a lot of highlights for just one char input.
Or you can set it to a larger value if you want the highlights to appear later.
Customize wrapping for C-n and C-p
This feature being on by default in helm-swoop was very distracting for me, and one of the reasons
that I wrote swiper. So it's off by default, but you can set it if you want. Calling
C-p on line number 0 will cycle to the last candidate etc.
Update Copyright
Since swiper was added to GNU ELPA, I had to assign the
Copyright to the FSF. This also means that you also need an FSF Copyright assignment for Emacs in
order to contribute more than total of 15 lines to the swiper code. It's really easy to get and is
already necessary to contribute to any package that is part of Emacs.
Add swiper-query-replace
You can start a query replace operation starting with the current candidate with M-q. If
you want to query replace in whole buffer, just do M-< M-q. And remember that
! will auto-replace all matches for the current query.
The default binding of M-q is fill-paragraph. This function is useless in the minibuffer,
so I chose that binding for query-replace, which is normally bound to sub-optimal M-%.
Although there are worse bindings than M-% (hold three keys at once), for instance, there's
C-M-% (hold four keys at once) that calls query-replace-regexp.
Outro
Enjoy the new features, and a big thanks to all who contributed!
Thanks to Nicolas Petton, Emacs is getting a new built-in sequence library in
25.1, and it’s already available on GNU Elpa for everyone. There’s not much to
be said about this besides the obvious “it’s about time”.
The popularity of dash.el speaks volumes about the need for this feature.
Combined with the recent inclusion of if-let, when-let, and the threading
macros into Emacs core, this should eventually allow most packages to phase out
that extra dependency. Not that there’s anything wrong with dash, of course,
but it’s nice to retire an external dependency if a built-in one becomes
available.
Still, in the field of sequence-processing, there are some noteworthy
differences that may lead you to prefer one over the other.
seq provides a unified interface for lists, vectors, and strings (i.e.,
sequences).
Some functions take arguments in a different order, such as (-take n list)
versus (seq-take list n).
Are there any operations you feel ought to be added to this arsenal? What about
other utility libraries that really should be built-in?
Update <2015-04-28 Tue>
This post used to say that “dash still covers a range of operations that seq
doesn’t”. However, Nico seems very intent on proving me wrong, and we now have
seq-difference and seq-intersection.
With a month's time and almost 50 commits since that last one, a new version of Hydra has emerged.
As usual, I'll just re-state the
release notes.
Fixes
Handling of heads with duplicate cmd was improved.
Don't bind nil in outside keymaps.
Work-around golden-ratio-mode in lv-window.
C-g (hydra-keyboard-quit) should run :post.
Bind [switch-frame] to hydra-keyboard-quit.
:post is called for :timeout.
New Features
hydra-key-format-spec is a new defcustom for the keys format in the docstring. It's "%s" by
default, but you can set it to e.g. "%-4s" if you like.
The key regex was extended to include most common key binding characters.
hydra-repeat is a hydra-specific repeat function. It behaves as you would expect repeat to behave.
New body option - :timeout. Use e.g. :timeout 2.0 to set the timer. After the first head is called,
a timer is started to disable the hydra. Each new head call resets this timer, so the hydra won't disappear as long as you keep typing.
Lines are truncated in lv-message. This is useful for large docstring not to become misaligned when the window
becomes too small.
Allow for a %s(test) spec in the docstring
The spec that's used for e.g. (test) is %S. So if (test) returns a string, it will be quoted.
This may not be desired, hence the new feature.
Example:
(defhydrahydra-marked-items(dired-mode-map"")"Number of marked items: %(length (dired-get-marked-files))Directory size: %s(shell-command-to-string \"du -hs\")"("m"dired-mark"mark"))
The code from this post has very little application. But it's kind of fun, so I'll post it. Star
Trek: TNG is one of my favorite shows, so I've added some TNG characters to one of the Hydra
features that I'm testing.
defhydradio statement
(require'hydra)(defhydradiohydra-tng()(picard"_p_ Captain Jean Luc Picard:")(riker"_r_ Commander William Riker:")(data"_d_ Lieutenant Commander Data:")(worf"_w_ Worf:")(la-forge"_f_ Geordi La Forge:")(troi"_t_ Deanna Troi:")(dr-crusher"_c_ Doctor Beverly Crusher:")(phaser"_h_ Set phasers to "[stunkill]))
The defhydradio macro is akin to a namespace construct that defines multiple variables that can
assume only certain values (either t or nil by default), and functions to cycle those values.
defhydradio implementation
Here's what you may see after a macroexpand:
(progn(defvarhydra-tng/picardnil"_p_ Captain Jean Luc Picard:")(put(quotehydra-tng/picard)(quoterange)[nilt])(defunhydra-tng/picardnil(hydra--cycle-radio(quotehydra-tng/picard)))(defvarhydra-tng/rikernil"_r_ Commander William Riker:")(put(quotehydra-tng/riker)(quoterange)[nilt])(defunhydra-tng/rikernil(hydra--cycle-radio(quotehydra-tng/riker)));; ...(defvarhydra-tng/names'(hydra-tng/picardhydra-tng/rikerhydra-tng/datahydra-tng/worfhydra-tng/la-forgehydra-tng/troihydra-tng/dr-crusherhydra-tng/phaser)))
As you can see, each list passed to defhydradio:
gets a prefixed variable definition
gets a range property for the prefixed symbol
gets a prefixed function definition that cycles the variable value based on the range property
The interesting statement in place of the docstring will actually evaluate to this docstring:
" _p_ Captain Jean Luc Picard: % -3`hydra-tng/picard^^^^ _h_ Set phasers to %`hydra-tng/phaser _r_ Commander William Riker: % -3`hydra-tng/riker^^^^^ _d_ Lieutenant Commander Data: % -3`hydra-tng/data^^^^^^ _w_ Worf: % -3`hydra-tng/worf^^^^^^ _f_ Geordi La Forge: % -3`hydra-tng/la-forge^^ _t_ Deanna Troi: % -3`hydra-tng/troi^^^^^^ _c_ Doctor Beverly Crusher: % -3`hydra-tng/dr-crusher"
The first line overflows a bit, but it's clear what it is. There's some flexibility in using
hydra--table, since you can:
redefine the row-column format (e.g. from 7x2 to 5x3)
add more variables to hydra-tng/names
Note also, that since hydra-tng/names holds all the names, and all the names know their default
values through range, it's possible to reset them all at once with hydra-reset-radios.
Finally, here's a simple implementation of beam-down:
(defunbeam-down()(interactive)(message"Beaming down: %s."(mapconcat#'identity(delqnil(mapcar(lambda(p)(when(symbol-valuep)(substring(symbol-namep)10)))'(hydra-tng/picardhydra-tng/rikerhydra-tng/datahydra-tng/worfhydra-tng/la-forgehydra-tng/troihydra-tng/dr-crusher)))", and ")))(global-set-key(kbd"C-c C-,")'hydra-tng/body)
Outro
And that's it. There actually is an application of defhydradio in
hydra-ox.el. It's not fully finished,
but you can already try it as an alternative to org-mode export dispatch widget, most things are
working.
Lately, I've been spending some time to automate and publish my Emacs config. Being able to quickly
reproduce your config has many advantages, the main one being that you no longer have to spend time
to make your config reproducible.
Happily, most of my config is already published in many packages, I only have to figure out a nice
layer to glue them together. Below, I'll show some automation for the packages managed by
package.el.
Step 1: get the main directory
This is an important step that many other peoples' configs miss, even the ones that are designed to
be distributed. You can't just assume that the config will be located in ~/.emacs.d and rely on
Emacs defaults. Instead, it's nice to be able to clone the config into a random directory and launch
an Emacs from there, without messing with the currently installed Emacs.
It's also useful for having multiple repositories for different versions of Emacs. ELPA packages are
byte-compiled, and the byte code can be incompatible between versions (for instance, 24.3 and 24.4).
Having two independent checkouts with ELPA directory auto-generated really helps in that case.
So here is the code to get the main directory and define an ELPA directory with respect to that:
(defconstemacs-d(file-name-directory(file-chase-linksload-file-name))"The giant turtle on which the world rests.")(setqpackage-user-dir(expand-file-name"elpa"emacs-d))
Step 2: decide what you like
Next, I initialize the package and define some of the packages that I like, omitting the
dependencies that they bring. Note that the code of the whole post is stored in a separate file
packages.el that is not intended to be loaded on start up, so it's fine to call
package-refresh-contents here:
(package-initialize)(setqpackage-archives'(("melpa"."http://melpa.milkbox.net/packages/")("gnu"."http://elpa.gnu.org/packages/")))(package-refresh-contents)(defconstora-packages'(auto-compileauto-yasnippetace-linkace-windowcompanyeclipse-themeflx-idofunction-argsheadlongido-occasionalido-vertical-modelispymagitsmexswiperuse-packageguide-keypowerlineprojectileslimeciderworforg-downloadmake-it-soukrainian-holidaysnetherlands-holidaysj-mode)"List of packages that I like.")
Step 3: install and upgrade
The install step is pretty straightforward: install a package unless it's already installed.
I tried to do something fancier for the upgrade, but in the end it was much more simple to
just call the interactive interface. The last two lines are basically equivalent to pressing
Uxy interactively:
Thanks to the first line, I can issue stuff like this on the shell:
emacs=emacs24 make up
This will use the emacs24 executable, instead of whatever emacs points to. Since the up
target depends on upgrade and run targets, they will be executed in that order:
the upgrade will install / upgrade all packages in a non-interactive Emacs
the run target will start an interactive Emacs with already updated packages
I really like putting stuff in Makefiles, since they are very flexible, yet so easy to call. In the
very same Makefile, I have a profile target from
the post on profiling Emacs start up. I also
wrote two packages related to Makefiles: helm-make and
make-it-so. The latter one is actually very interesting and
deserves its own post, I should maybe just clean it up a bit.
Outro
I'll just cite Gall's law here:
A complex system that works is invariably found to have evolved from a simple system that
worked. A complex system designed from scratch never works and cannot be patched up to make it
work. You have to start over with a working simple system.
I think it applies from both sides w.r.t. my Emacs config: it kind of works, but I really wish it
was reproducible from the start, before making it complex.
Today, I've fixed a few bugs in both swiper and ivy.
Finally, the number of candidates display has also been added. You can see the whole thing in the
one minute video demo.
I've also added some integration tests,
if you're interested. I didn't know how to do exactly this type of testing before (when there's input from minibuffer).
Turns out, it's pretty easy to do using execute-kbd-macro:
(require'ert)(defvarivy-exprnil"Holds a test expression to evaluate with `ivy-eval'.")(defvarivy-resultnil"Holds the eval result of `ivy-expr' by `ivy-eval'.")(defunivy-eval()"Evaluate `ivy-expr'."(interactive)(setqivy-result(evalivy-expr)))(global-set-key(kbd"C-c e")'ivy-eval)(defunivy-with(exprkeys)"Evaluate EXPR followed by KEYS."(let((ivy-exprexpr))(execute-kbd-macro(vconcat(kbd"C-c e")(kbdkeys)))ivy-result))(ert-deftestivy-read()(should(equal(ivy-read"pattern: "nil)nil))(should(equal(ivy-read"pattern: "'("42"))"42"))(should(equal(ivy-with'(ivy-read"pattern: "'("blue""yellow"))"C-m")"blue"))(should(equal(ivy-with'(ivy-read"pattern: "'("blue""yellow"))"y C-m")"yellow"))(should(equal(ivy-with'(ivy-read"pattern: "'("blue""yellow"))"y DEL b C-m")"blue"))(should(equal(ivy-with'(ivy-read"pattern: "'("blue""yellow"))"z C-m")nil)))
Outro
Give the package a try, if you haven't yet. You can get it from MELPA.
(defuncouns-git()"Find file in the current Git repository."(interactive)(let*((default-directory(locate-dominating-filedefault-directory".git"))(cands(split-string(shell-command-to-string"git ls-files --full-name --")"\n"))(file(ivy-read"Find file: "cands)))(whenfile(find-filefile))))
This one will allow you to find a file in your Git repository.
I've bound it like this:
(global-set-key(kbd"C-c f")'couns-git)
Here's how it looks like for selecting a file in the Emacs repo:
The speed isn't an issue for 3500 candidates, although I should try to add the number of candidates
display pretty soon. It's just that there isn't a good spot in the minibuffer to show that.
I've also updated ivy-next-line and ivy-previous-line to switch to the previous history element
in case ivy-text is empty. This is the exact behavior of isearch, so if you bind swiper to C-s
and C-r like I do, you'll find that C-s C-s and C-r C-r work as expected.
Thanks to @johnmastro for the suggestion.
Here's the current state of the keymap:
(defvarivy-minibuffer-map(let((map(make-sparse-keymap)))(define-keymap(kbd"C-m")'ivy-done)(define-keymap(kbd"C-n")'ivy-next-line)(define-keymap(kbd"C-p")'ivy-previous-line)(define-keymap(kbd"C-s")'ivy-next-line)(define-keymap(kbd"C-r")'ivy-previous-line)(define-keymap(kbd"SPC")'self-insert-command)(define-keymap(kbd"DEL")'ivy-backward-delete-char)(define-keymap(kbd"M-<")'ivy-beginning-of-buffer)(define-keymap(kbd"M->")'ivy-end-of-buffer)(define-keymap(kbd"M-n")'ivy-next-history-element)(define-keymap(kbd"M-p")'ivy-previous-history-element)(define-keymap(kbd"C-g")'minibuffer-keyboard-quit)map)"Keymap used in the minibuffer.")
You can also try counsel for completing Elisp and couns-clj for completing Clojure. As you can
see, the implementation is very simple: you just get a list of strings, and you're done.
If you want to implement some ivy completion for your favorite mode, I recommend to find the
corresponding ac-source and see where it gets its list of strings.
In contrast with fetching pull requests, which either requires a minor-mode or
some long-named branches, creating pull requests is as easy as a one-key deal.
*If you’re reading this now, note that this code was written for an older Magit
version. An up-to-date version of the code (working with latest Magit) can be
found on this other post.*
Because sx.el (a package which I really ought to blog about) was developed as a
group effort, Sean and I decided early on to use PRs for everything.
Predictably, the need arose for a quick way to create them, and I promptly asked
Emacs.StackExchange for help. The solution below is a courtesy of Constantine.
(defunendless/visit-pull-request-url()"Visit the current branch's PR on Github."(interactive)(browse-url(format"https://github.com/%s/pull/new/%s"(replace-regexp-in-string"\\`.+github\\.com:\\(.+\\)\\.git\\'""\\1"(magit-get"remote"(magit-get-current-remote)"url"))(magit-get-current-branch))))(eval-after-load'magit'(define-keymagit-mode-map"V"#'endless/visit-pull-request-url))
After hitting V, you just need to press the big green button at the Github
page.
I really liked Marcin Borkowski's post on Info dispatch,
where he describes how to open several *info* buffers for the most commonly used manuals.
However, as I tried to call one of these functions today, I forgot the key binding.
Hydra to the rescue!
Here's what I've come up with:
(defunora-open-info(topicbname)"Open info on TOPIC in BNAME."(if(get-bufferbname)(progn(switch-to-bufferbname)(unless(string-matchtopicInfo-current-file)(Info-goto-node(format"(%s)"topic))))(infotopicbname)))(defhydrahydra-info-to(:hintnil:colorteal)"_o_rg e_l_isp _e_macs _h_yperspec"("o"(ora-open-info"org""*org info*"))("l"(ora-open-info"elisp""*elisp info*"))("e"(ora-open-info"emacs""*emacs info*"))("h"(ora-open-info"gcl""*hyperspec*")))(define-keyInfo-mode-map"t"'hydra-info-to/body)
The Plan
I already have the <f1> i key binding hard wired to my fingers. So after I'm in
any Info buffer, I plan to press t and get this dispatch:
Getting the Hyperspec
I've heard people lauding HTML while dumping on Info.
I'd suggest them to compare the Common Lisp Hyperspec web site
to this info file extracted from there.
Their contents are identical, but it's easier and more pleasant to use Info.
After I downloaded the file, I extracted it to ./etc/info/gcl.info, where . is my emacs-d.
Then I just added this directory to the Info path:
I imagine that if you're reading this blog, you like to tinker with Emacs. And people who like to
tinker with stuff probably also like to learn new programming languages, just for fun. In that
case, if you ever want to learn a non-mainstream language, I highly recommend J.
From its homepage:
J is a modern, high-level, general-purpose, high-performance programming language. J is portable
and runs on Windows, Unix, Mac, both as a GUI and in a console. J systems can be installed and
distributed for free.
The Appetizer
Things J has going for it that are high on my list:
What attracted me to J in the first place is that it consistently has the shortest solutions (and
fast-running) on Project Euler. What kept me going after the initial
wow-effect, was the extremely elegant standard functions implementation dividing things into
verbs, adverbs, and conjunctions.
Something impressive: a Sudoku solver
This code is taken from the ob-J page that I wrote some
time ago (you can find a lot of additional info there):
It's pretty amazing that the whole implementation, not counting the example input matrix, takes only
169 characters. You can also see how functional the language is.
Something simpler: a factorial
To have a more simple example, here's how to write down incrementally the factorial of 20:
The spaces are optional, I've only included them to make the code more clear.
Something visual: a sine plot
#+begin_src J
load 'plot'
plot 1 o. 0.1 *i.200
#+end_src
By simply pressing C-c C-c on this source block you get this image generated and opened in your browser:
The Emacs Tooling
As mentioned above, there's org-mode babel support for J, including session interaction. More
importantly, there is j-mode. The same package also provides a
REPL via jconsole.
Additionally, I wrote down a learning/assisting tool
helm-j-cheatsheet.
Here's one of the screenshots:
It allows to:
insert a function by English name
look up the English name of the function by symbol
J is a very cool language to try and I hope you give it a go. I haven't yet managed to find a nice
use for it, but you could say that, just as learning LISP, learning J can make you better at other
languages.
Until recently, I didn't even know that apropos was an actual word. I thought that it was just
some gibberish that Emacs hackers invented to name a command. Turns out, it actually has a very
appropriate meaning:
As you can see, there are a total of 7 apropos functions available. The most useful ones are
apropos, apropos-command and apropos-variable. But apropos-value is very interesting as
well: it will not match an object name with the input, but instead search for all objects whose
contents match input. So, for instance, if you see something like "error 42" come up, and you
don't know where it's coming from, it's likely that some variable holds the value 42. And that's
when you use apropos-variable.
Apropos is like old-school googling, and I find it highly useful in finding the Emacs information
that I need. I hope that you will find it useful as well. Happy hacking!
I'll describe two new things in swiper and ivy that happened
within the day. Although you can install them separately from MELPA, they still live in a single git
repository.
swiper / ivy faces now inherit the standard ones
It's a good idea by @purcell, and I tend to agree. So now, there are
six faces that inherit highlight, isearch-lazy-highlight-face, isearch and match faces
between them. The advantage is that most themes re-define the above four faces as they see fit, so
the swiper faces will fit in better without customization.
Still, I really enjoy the previous faces that were derived from the Swiper sprite. You can have them
as part of eclipse-theme. The theme creators can just
copy them verbatim to a light theme if they so choose.
By the way, it seems that not all Emacs users are aware that you can customize faces. You can do so
interactively for most popular packages. For example, after M-xcustomize-groupswiper, you get a GUI for selecting most configurable things for swiper, including the faces.
The GUI will generate the code that looks something like this:
The code to complete Elisp code at point is very simple:
(defuncounsel()"Elisp completion at point."(interactive)(let*((bnd(bounds-of-thing-at-point'symbol))(str(buffer-substring-no-properties(carbnd)(cdrbnd)))(candidates(all-completionsstrobarray))(ivy-height7)(res(ivy-read(format"pattern (%s): "str)candidates)))(when(stringpres)(delete-region(carbnd)(cdrbnd))(insertres))))
The only function above that isn't one of the familiar primitives is ivy-read. But even that one
has a similar interface to that of completing-read or ido-completing-read. I was using
helm-lisp-completion-at-point before, but counsel is much less obtrusive, while offering
comparable speed and convenience:
The default minibuffer height for ivy is set to 10 via ivy-height. It's quite reasonable all-around, but
for counsel I've set it to 7 via a let binding.
Just three days ago, I wrote swiper and introduced it in
a post.
And while swiper solved the problem of helm-swoop
being awkward for me to use, swiper's code itself is pretty awkward, since even after using helm
a dozen times in my packages, I'm still not well-versed in its internals.
And, apparently, there are people who don't like helm because it updates a lot. I don't know if
it counts as a viable complaint, but I do agree that the helm update time is quite long. So I
wrote down an alternative completion back end for swiper. It looks quite similar to
ido-vertical-mode, although it is completely unrelated to ido. In fact, the matching algorithm
currently is similar to helm-match-plugin: "for example" is transformed into
"\\(for\\).*\\(example\\)".
The Details
Currently, in the ivy version of swiper only the anchoring algorithm is missing. Anchoring is
what I call the process of selecting a close candidate when the number of candidates changes. It
seems that helm doesn't implement an anchoring algorithm, so both helm-swoop and swiper
implement their own. I don't know how easy it would be to write down the proper generic thing, my
current implementation for the helm version of swiper is quite hacky.
Just to explain to you what needs to be done. Suppose there's an input "ab" that matches 50
candidates, and the candidate number 42 is the current one. Now, if the user types one char to make
the input into "abc", the number of candidates has changed to 10, and the candidate that was current
previously doesn't match any more. It is up to the matcher code to decide which of the 10 candidates
has to be current now. Usually it doesn't matter much, but in the case of swiper it matters since
the point will be moved to the current candidate in the original window.
In any case, there's still a lot of things that have to be done for ivy, but I think that
it's quite usable now. Actually, I've already added it to my bindings:
(global-set-key"\C-s"'swiper)
Here's how it looks like:
And here's an example of completion in dired:
You need to have Emacs 24.4 in order to get a nicer highlight in the minibuffer that uses
add-face-text-property instead of the old propertize.
There's a custom variable that decides the height of the minibuffer window, you can set it as you
like:
(defcustomivy-height10"Number of lines for the minibuffer window.":type'integer)
Outro
Give the new back end a go, see if you like it. It looks quite pretty and minimalist to me, but I'm
biased since I made it.
Today, I'll describe a quite recent addition to ace-window,
which comes from the idea by @deftsp:
Why not show the ace-window dispatch keys in the mode line all the time?
Certainly, there's no reason not to have this option, and it actually makes the whole interface
better, since it becomes less feedback-based:
You glance at a window that you want.
You know which key you have to press before you call ace-window
You make a single-step call instead of:
press ace-window shortcut,
read the dispatch char,
press the dispatch char.
The feature is implemented as ace-window-display-mode - a minor mode that you can toggle on or
off whenever you feel like it.
Here's how it looks like:
As you can see, I've made aw-keys list short on purpose, just to show you that the full path
will be displayed. See, for instance, the *scratch* window: its path is gf.
Implementation
Since the implementation is quite short, I'll post it here and go over a few things that might
interest people who write Elisp (on some level, please don't mock me for being obvious).
As per request of a curious reader, I've updated the code with the comments from below. Don't try
this at home, excessive commenting is bad style.
;; Something that modifies the Emacs behavior should;; preferrably be implemented as a minor mode.;;;###autoload(define-minor-modeace-window-display-mode"Minor mode for showing the ace window key in the mode line.";; And since this minor mode isn't tied to a particular;; buffer, I declare it as global.;;;; An interesting quirk is that I have to put *something*;; between the docstring and the body, otherwise it won't;; work.:globalt;; Dispatch on the variable symbol of the mode -;; `ace-window-display-mode`. `define-minor-mode' will;; define both a variable and function symbol.(iface-window-display-mode(progn;; Update the window parameters(aw-update);; Since `mode-line-format' is a buffer-local;; variable, I set it with `set-default', in order;; for the change to not just happen in the current;; buffer.(set-default'mode-line-format`((ace-window-display-mode(:eval(window-parameter(selected-window)'ace-window-path))),@(default-value'mode-line-format)))(force-mode-line-updatet);; Each time a window is created or deleted, Emacs;; will run the `window-configuration-change-hook' -;; exactly what I need to update `mode-line-format'.(add-hook'window-configuration-change-hook'aw-update))(set-default'mode-line-format(assq-delete-all'ace-window-display-mode(default-value'mode-line-format)))(remove-hook'window-configuration-change-hook'aw-update)))(defunaw-update()"Update ace-window-path window parameter for all windows."(avy-traverse(avy-tree(aw-window-list)aw-keys)(lambda(pathleaf);; Use `set-window-parameter' to store a variable for;; each window. Buffer local variables would not work;; here, since one buffer can be displayed in multiple;; windows, and those would need a different key each.(set-window-parameterleaf'ace-window-path(propertize(apply#'string(reversepath))'face'aw-mode-line-face)))))
Since I'm writing something that modifies Emacs behavior when it's enabled, I first try to implement
it as a minor mode. And since this minor mode isn't tied to a particular buffer, I declare it as
global.
An interesting quirk is that I have to put something between the docstring and the body, otherwise
it won't work. In this case, I put :global t.
Next, follows that standard dispatch on the variable symbol of the mode - ace-window-display-mode.
The define-minor-mode macro will make both variable and function definition for the symbol it's
given:
the variable is used to check if the mode is on.
the function is used to turn the mode on / off.
Note the use of assq-delete-all: this is for when some other package modifies the
mode-line-format after ace-window does. In that case, ace-window's entry wouldn't be the first
one any more.
Since mode-line-format is a buffer-local variable, I set it with set-default, in order for the
change to not just happen in the current buffer.
Each time a window is created or deleted, Emacs will run the window-configuration-change-hook -
exactly what I need to update mode-line-format.
One final trick is to use set-window-parameter to store a variable for each window. Buffer local
variables would not work here, since one buffer can be displayed in multiple windows, and those
would need a different key each.
I really like how the avy-tree / avy-traverse interface ended up as: the same functions are used
for selecting window and setting mode-line-format.
Outro
I hope that you enjoy the update, and keep those interesting ideas coming!
Did you know you can fetch Github pull requests with git by adding a
remote.origin.fetch configuration? That insightful tip is a courtesy of Oleh
at (or emacs, a very active blog that has a habit of unbalancing parentheses
wherever it goes). I like the tip so much I wanted to add something to it.
Instead of manually adding that line to you .git/config file, why not have
Magit do that for you?
The following snippet will ensure that you always have this fetch configured on
Github repositories as soon as you issue magit-status.
(defunendless/add-PR-fetch()"If refs/pull is not defined on a GH repo, define it."(let((fetch-address"+refs/pull/*/head:refs/pull/origin/*")(magit-remotes(magit-get-all"remote""origin""fetch")))(unless(or(notmagit-remotes)(memberfetch-addressmagit-remotes))(when(string-match"github"(magit-get"remote""origin""url"))(magit-git-string"config""--add""remote.origin.fetch"fetch-address)))))(add-hook'magit-mode-hook#'endless/add-PR-fetch)
This means you can always access pull requests from Magit by hitting b b, the
usual branch command. The reference names start with /refs/pull/origin/, but
completion does most of the job here.
This is more lightweight approach than my previous post on the subject, which
used gh-pulls-mode. You should try both and see which one you prefer.
Here, 20 is the issue number. And after merging I finally get access to magit and ediff and
all that jazz to see what the change is actually about. Before I learned this, I had to manually
add remotes with magit-add-remote. And even before that, I was just clicking the merge button
after thoughtfully browsing the diff in Firefox. Trust me, ediff is orders of magnitude better.
Illusion 3: edit a Github wiki in Emacs
Of course, I was always editing it in Emacs with
It's All Text Firefox plugin.
But when I wanted to add an image, I actually read the manual and found out that you can simply
clone the wiki:
Or with the ssh style, in order to not type the user name and password on pushing:
git clone git@github.com:abo-abo/hydra.wiki.git
By the way, today I've made a very large overhaul of hydra's
README.md. So if you wanted to start writing your own hydras, but were hesitant because of the lack
of documentation, now is the time to start.
If you figure out something that you think is worth documenting, you can immediately leave it on the
wiki.
By the way, here's the syntax for including an image, if you were wondering:
You upload a file simply by adding it to the repository. Look Ma, no Imgur!
This is actually an issue that I ran into, when I was less experienced. I included some one megabyte
gifs in the lispy repository to refer to them in README.md. Just a few of them resulted in a
very uncomfortable cloning time, which is an issue for Travis CI and, you know, humans. Finally, I
had to resort to bfg-repo-cleaner to remove the gifs
from the repo history. It could all be avoided if I just posted all images on the wiki and linked to
them in README.md.
Outro
I hope that you find these illusions useful to make you a better Emacs magician. Happy hacking!
I like the idea of
helm-swoop, but it
somehow has minor annoyances that stop me from using it, like
automatic helm-input, and especially the circular candidates. Or maybe it's just NIH talking.
Anyway, I'm rolling my own, and it's called swiper.
Swiper in action
Here's how it looks like:
As you can see, the search string "dec fun pro" is transformed into
a regex "\\(dec\\).*\\(fun\\).*\\(pro\\)", and each group in the
matches is highlighted with a different face. This is quite similar
to the way re-builder does it. In fact it's possible to use swiper
as a poor man's re-builder (since it matches only single lines).
I'm doing my own matching this time, as the part-swapping behavior of
helm-match-plugin is more annoying than useful.
Also note that:
The whole *swiper* buffer is fully syntax highlighted.
The appropriate parts of the matches are highlighted as well.
Swiper in a dired buffer
Here's another screenshot:
It appears that helm is ignoring the display of file attributes, since they have
a sort of invisible property set. I kind of like this behavior.
See how the faces are recycled
I've defined only 4 faces currently (foxes aren't unicorns, the
palette is quite limited), so they loop about if you have many groups:
Yup, the technology is there. Although a powerline theme for the
helm mode line is still missing.
After adding asynchronous operations to Paradox, I saw the need to provide some
visual feedback to the user. In the simplest sense, this could be a fixed
message on the mode-line, such as “Upgrading…” or “Working”, but this is
not enough. I needed movement. Movement implies something is ongoing. It catches
your eye and gives you that subconscious reassurance that progress is being
made. A tiny spinning wheel, hourglass, or rainbow is enough to sooth all your
doubts, unerringly restoring your confidence on the software and those who made
it.
Today, I introduce spinner.el, Emacs’ version of the spinning hourglass.
The spinner is added to the mode-line of a specific buffer, and stays there
until stopped by the program. In the case of Paradox, for instance, you get a
spinner on the *Packages* buffer to indicate background operations are
ongoing, and it is immediately removed once they are finished.
Using the package is as simple as calling (spinner-start 'vertical-rising)
(eventually followed by spinner-stop). There are currently 17 different
spinners available (see above), and you can also specify the animation speed or
even add your own animations.
Spinner.el is available to all Emacs versions running package.el, so hopefully
other packages will make use of it as well.
It was kind of disappointing to summarize in
the last post
that there weren't many new features for
lispy 0.24.0. So I thought long and hard and came up with something
quite obvious: since lispy offers commands to quickly manipulate point and mark, it should offer one
to quickly restore them.
Here's the bulk of the new code:
(defvarlispy-pos-ring(make-ring200)"Ring for point and mark position history.")(ring-insertlispy-pos-ring1)(defunlispy--remember()"Store the current point and mark in history."(if(region-active-p)(let((bnd(lispy--bounds-dwim)))(unless(equalbnd(ring-reflispy-pos-ring0))(ring-insertlispy-pos-ringbnd)))(unless(eq(point)(ring-reflispy-pos-ring0))(ring-insertlispy-pos-ring(point)))))(defunlispy-back()"Move point to a previous position"(interactive)(if(zerop(ring-lengthlispy-pos-ring))(user-error"At beginning of point history")(let((pt(ring-removelispy-pos-ring0)))(if(consppt)(lispy--markpt)(deactivate-mark)(goto-charpt)))))
Here, I'm using lispy-pos-ring made with make-ring to store the
last 200 point and mark positions. In Elisp, a ring is basically a stack backed by a vector.
When the vector space overflows, the older stuff is overwritten.
In this ring I store each time either the point, or a cons of the
point and mark if the region is active. This way possible to restore
the region even if it was deactivated several movement commands ago.
I've put lispy--remember into the most used navigation commands:
j - lispy-down
k - lispy-up
h - lispy-left
l - lispy-right
f - lispy-flow
i - lispy-mark-car (when the region is active)
a - lispy-ace-paren
And lispy-back is now bound to b. The previous binding of
b - lispy-store-region-and-buffer is now bound to
xB (quite close to B - the binding for
lispy-ediff-regions).
I really like the new command, it's especially useful to reverse
h and l. Previously, they could be reversed
with f, but that can get annoying if you have to press it
many times. This results in a much more relaxed editing - I know that
whatever I press, I can restore the point position quickly if needed.
This feature is added to the already present list of safeguards:
j and k are guaranteed not to exit the parent list.
> and < are guaranteed not to exit the parent list.
C reverses itself.
Here's a recipe to copy the third item of the current list and move
the point back: 4mnb. You can use it to copy a function's
docstring if you're in Elisp. You'll need 3mnb for
Clojure, since it weirdly has the docstring before the arguments. I
think it's pretty cool: by typing 3mnb I'm basically
calling (kill-new (caddr (current-sexp))) on my own code.
The last release was a month ago, and there have been 70 commits to
master since then. If you're not familiar with lispy, see
my intro post for version 0.21.0.
I'll just copy the release notes here,
while adding a few things.
Fixes
DEL behaves properly after a string and one space.
C-k works better for expressions preceded with "#".
3 should not add a space when there is one already.
# will not add a space after a comma.
C-j works better in comments after a quote.
lispy--eval-elisp-form first arg is now named lispy-form instead of form. It was impossible
to evaluate an unrelated form variable with the previous behavior.
F again works properly when jumping to a jar (e.g. to defn) from Clojure source.
C-k won't call delete-region in some obscure branches.
Enhancements
P (lispy-paste) will add a newline when called from start of line. This way,
nP becomes equivalent to c (lispy-clone). Of course, it's more flexible: you
can do e.g. nkP.
xb (lispy-bind-variable) now works on regions as well. Use it to bind the current
sexp or region as a let-bound variable: it will put you in iedit. When you're done with iedit, press
M-m (lispy-mark-symbol) to exit iedit. If you need to move the let binding around, use
a combination of C (lispy-convolute) and h (lispy-left).
g will ignore loaddefs.el for Elisp.
M-m works better in unbalanced buffers, which should be an rare thing.
add defhydra to lispy-tag-arity: now g will recognize defhydra statements.
The tag logic was improved to do less parsing.
lispy-outline was updated to match the standard ^;;; outline regex. Try pressing I
in e.g. org.el, it's quite beautiful.
All lispy-eval functions will preserve the match data.
> will delete the extra whitespace while slurping.
Added undercover/Coveralls test coverage report.
H (lispy-ace-symbol-replace) is now a Hydra: type h to delete more, type
u to undo.
Q (lispy-ace-char) now uses avy to jump. This change allows to cover this
function with a test.
This is another step toward edebug-less debugging, adding to special behavior for let, cond
and labels. Remember that you can drop out of edebug with Z (lispy-edebug-stop).
This function will take the current function arguments that edebug provides, store them in
top-level, and exit edebug.
This is really cool for setting up entry conditions for a function that you want to debug,
or even a function with an empty body that you want to write. Why am I so eager to exit edebug?
Because it puts the code in read-only mode, which is quite restrictive.
Incompatible changes
lispy-helm-columns is now a list '(60 80). The first number is
the width of the tag name column, the second number is the width of
both tag name and tag file. The tag name column is left-aligned, while the file column is right-aligned.
j and k should now move to outline when at
beginning of comment. The previous behavior was to look for the first
sexp in the direction. You can still do that with f.
I (lispy-shiftab) is now a two-way cycle, instead of
three-way, like org-mode. The contents can be obtained with
C-u I or C-u C-TAB.
Outro
It seems that lispy is winding down feature-wise, which is a good
thing, because I'm almost out of keys - there's only Y and
U left.
Possible next steps would be to improve the test coverage (currently
48%) and the documentation. Perhaps I'll try to implement some
automation for generating
the function reference, or learn some
texinfo and write an actual manual, see how ox-texinfo holds up.
I hope that you'll grow to enjoy lispy as much as I do. Happy hacking!
This means that when I press < from the start of the
line, a Hydra will be called instead of inserting <, otherwise <
will be inserted.
As the default insert method for org-mode blocks is already pretty
convenient, this Hydra is more of an illustration than anything,
especially of the new :hint nil feature.
Just to remind you, each head has four placeholders:
key binding
body
hint
plist
When a Hydra is active, it will show its doc in the echo area in the
bottom of the frame. This doc is composed of two parts: the body doc
and the heads' doc. The body doc you specify yourself, the heads' doc
is built by concatenating the key binding and the hint for each head
into a (single) line.
If you don't specify a hint for a head, it's assumed to be ""; this
head's binding will still be in the heads' doc. If you don't want a
head's binding to be in the heads' doc, set the hint to nil. This is
commonly done because a head is already documented in the body doc.
It can sometimes become tedious to set all the hints to nil, for
instance in the Hydra above, I would need to do it 13 times. Hence
the :hint nil shortcut.
Here's how it looks like:
I'm not a Scrabble pro: clash for word score 10 is my result, although
a longer word would break the nice column layout.
The first two columns contain begin/end templates, while the
third one contains the one-line templates.
It's strange that I haven't implemented this feature before, as it's
quite a common usage pattern:
You select a window with ace-window.
You do some stuff there.
You want to return the previous window.
aw-flip-window
In the step 3, you have to go though the whole aw-keys dispatch,
only to select a window which can be pre-determined. Not any more, if
you call aw-flip-window:
(defunaw-flip-window()"Switch to the window you were previously in."(interactive)(aw-switch-to-window(aw--pop-window)))
So now, you could have a grid of 10 windows, select one of them with
ace-window, and switch indefinitely between it and the previous
window with ace-flip-window, while ignoring the other 8.
aw-ignored-buffers
Remember that if you have some window that you never want to switch to
with ace-window, you can add it to aw-ignored-buffers:
(defcustomaw-ignored-buffers'("*Calc Trail*""*LV*")"List of buffers to ignore when selecting window.":type'(repeatstring))
It's not a big deal, but it's convenient at least for this scenario:
I start with one active window.
M-xcalc; now I have three windows.
I can toggle back and forth between calc and the main window with
ace-window without having to type aw-keys, since *Calc Trail* is
ignored, so that makes only two total windows.
Selecting last window during the ace-window dispatch
This is a really cool feature, in my opinion: for all three actions -
aw-select-window, aw-swap-window, and aw-delete-window, you can
select the previous window as a target with the same key n.
This is, of course, customizable:
(defcustomaw-flip-keys'("n")"Keys which should select the last window.":set(lambda(symval)(setsymval)(setqaw--flip-keys(mapcar(lambda(x)(aref(kbdx)0))val))))
So you could have a whole list of bindings that select the previous
window during the aw-keys dispatch. This is cool because there's no
visual feedback necessary, so this binding can be easily added to the
muscle memory. The bindings don't necessarily need to be single keys,
anything with one chord, e.g. C-f, is acceptable.
This function will quietly run a batch Emacs with your current config
to see if it errors out or not.
in case that there were no start up errors, it will echo "All is well"
when there's an error, it will pop to a *startup error* buffer with the error description
The nice thing about this is that in case of an error you have a
functional Emacs to fix that error, since fixing errors with emacs
-Q is quite painful.
Another approach could be to just start a new Emacs instance, and
close the window in case there isn't an error. So all that the code
above does is automate closing the window (sort of, since the window
never opens). Still, I think it's pretty cool. And you could attach it
to the after-save-hook of your .emacs, or a timer.
You could even configure Emacs to send you an email in case it notices
that there will be an error on the next start up. Or add the test to
before-save-hook and abort the save in case it results in an
error. That's some HAL 9000 level stuff right there:
I started using Emacs around 2010-2011 when I needed an environment
for C++ and LaTeX. The default color theme was horrendous (it still
is), and I don't fancy myself a designer, so I just copied a color
theme of the thing that I was using previously: Eclipse.
This theme modifies almost
nothing except the font lock faces, and looks reasonable while called
with -nw, although I don't see why anyone wouldn't want to take
advantage of what graphical Emacs has to offer.
Here's a sampler:
It's shown together with my fork of
powerline. I'll try to merge
it as a theme eventually, I'm delaying it because powerline isn't
very easy to understand/modify.
Eclipse theme should be on MELPA soon, I hope you'll enjoy it. I've
tried probably 30 themes on MELPA, but I just can't part with
eclipse-theme. I'm guessing that it's a feeling that most theme creators share.
I just closed a really helpful
issue for
ace-window. Turns out that there's a plugin for vim called
Easymotion that's very similar to ace-jump-mode. And that plugin
doesn't highlight the leading chars one by one, but instead gives them
all at once. Which is a pretty good idea, since it's more convenient
to read the whole path at once and type it in at once rather than:
read one char
type one char
read one more char
type one more char
... (maybe more steps)
To turn on the new behavior:
(setqaw-leading-char-style'path)
Although this will only have an effect once you have more than 10 windows.
But this method really improves the functions from avy-jump.el.
I've added some more commands and renamed the old ones since yesterday.
avy-jump demos
avi-goto-char-2
Here's how I like to bind avi-goto-char-2:
(global-set-key(kbd"C-'")'avi-goto-char-2)
And here's the result of C-' bu:
As you can see, nothing is overwritten by the overlay - it's appended
after the search chars. In the screenshot above, I have avi-background at nil; you can set it to
t if you want a gray background.
avi-goto-char
After binding avi-goto-char:
(global-set-key(kbd"π")'avi-goto-char)
Here's the result of π b:
avi-goto-line
After binding avi-goto-line:
(global-set-key(kbd"M-g f")'avi-goto-line)
Here's the result of M-g f:
I've also added:
avi-copy-line
avi-move-line
avi-copy-region
These functions use the method of avi-goto-line to copy/move stuff.
It might be useful for line-based text.
avi-goto-word-0
Here's the simple definition and the binding:
(defunavi-goto-word-0()"Jump to a word start in current window."(interactive)(let*((avi-keys(number-sequence?a?z))(candidates(avi--regex-candidates"\\b\\sw")))(avi--goto(avi--processcandidates#'avi--overlay-pre))))(global-set-key(kbd"M-g e")'avi-goto-word-0)
There might be quite a lot of candidates for this one, but there's no call to read-char.
Here's what happens after M-g e:
avi-goto-word-1
Here's a version of it that reads one char:
(defunavi-goto-word-1()"Jump to a word start in current window.Read one char with which the word should start."(interactive)(let((candidates(avi--regex-candidates(concat"\\b"(string(read-char"char: "))))))(avi--goto(avi--processcandidates#'avi--overlay-pre))))(global-set-key(kbd"M-g w")'avi-goto-word-1)
Here's what happens after M-g w b:
Outro
Give the new functions a go and see if you like them. New ideas are
welcome, especially pertaining to ripping off vim plugins. I did the
vimtutor once, but I have no idea how vim plugins work. But it turns
out that I like Easymotion.
I probably spend about 75% of my programming time looking up
documentation. That’s really easy in Emacs Lisp—my documentation
is built into my editor—but in most other languages I’m not so
lucky.
I wrote engine-mode to help fix this problem. It’s a simple minor
mode that lets me define arbitrary search engines and send snippets
of text to them from within Emacs. Now I don’t have to copy and
paste text between my browser and editor! So fancy.
We’ll be talking about how to get started with engine-mode (it’s
easy!) and then digging into the implementation and discussing the
process of committing a package to MELPA.
Org-capture is a great way to take notes and plan in Emacs. Capture
templates provide a flexible way to extend org-capture and
personalize the way you record various bits of information. This
talk will cover how Jonathan uses org-capture and provide an intro
to writing your own org-capture templates.
Now that you’ve started your journey on the Typography Express by using round
double quotes, take a seat and extend that to your apostrophes as well. This
snippet binds a round apostrophe to the ' key, but also inserts a pair of
single round quotes with a prefix.
Finally, like the previous one, it also falls back on self-insert-command
inside a code block.
(define-keyorg-mode-map"'"#'endless/apostrophe);; (eval-after-load 'markdown-mode;; '(define-key markdown-mode-map "'";; #'endless/apostrophe))(defunendless/apostrophe(opening)"Insert ’ in prose or `self-insert-command' in code.
With prefix argument OPENING, insert ‘’ instead and
leave point in the middle.
Inside a code-block, just call `self-insert-command'."(interactive"P")(if(and(derived-mode-p'org-mode)(org-in-block-p'("src""latex""html")))(call-interactively#'self-insert-command)(if(looking-at"['’][=_/\\*]?")(goto-char(match-end0))(if(nullopening)(insert"’")(insert"‘’")(forward-char-1)))))
Today, ace-window is dropping
the dependency on ace-jump-mode. Most of the dependency was already
dropped in 0.7.0, when I had to fix ace-window to work better with
defhydra.
The change will not be user-visible, unless you relied on some
customizations of ace-jump-mode to transfer to ace-window. You'll
be able to transfer your customizations to similarly named variables.
The reason for the move is that, due to the implementation of
ace-jump-mode, it's hard to wrap its calls, since the function exits
before any of the red chars are selected by the user. Also,
ace-jump-mode uses its own defstructs for candidates while I'd
rather have plain lists, but that's a minor issue.
New back end: avy.el
I tried to pinpoint the most generic algorithm that ace-jump-mode implements and wrote it down
in avy. Here's the core of the implementation:
(defunavy-subdiv(nb)"Distribute N in B terms in a balanced way."(let*((p(1-(floor(lognb))))(x1(exptbp))(x2(*bx1))(delta(-nx2))(n2(/delta(-x2x1)))(n1(-bn21)))(append(make-listn1x1)(list(-n(*n1x1)(*n2x2)))(make-listn2x2))))(defunavy-tree(lstkeys)"Coerce LST into a balanced tree.The degree of the tree is the length of KEYS.KEYS are placed appropriately on internal nodes."(let((len(lengthkeys)))(cl-labels((rd(ls)(let((ln(lengthls)))(if(<lnlen)(cl-pairliskeys(mapcar(lambda(x)(cons'leafx))ls))(let((ks(copy-sequencekeys))res)(dolist(s(avy-subdivlnlen))(push(cons(popks)(if(eqs1)(cons'leaf(popls))(rd(avy-multipoplss))))res))(nreverseres))))))(rdlst))))
The first function, avy-subdiv, tries to split a number in terms of the base in a way that
the most leaves have the lowest level:
(avy-tree'("Acid green""Aero blue""Almond""Amaranth""Amber""Amethyst""Apple green""Aqua""Aquamarine""Auburn""Aureolin""Azure""Beige""Black""Bronze""Blue""Burgundy""Candy apple red")'(1234));;=>((1(1leaf."Acid green")(2leaf."Aero blue")(3leaf."Almond")(4leaf."Amaranth"))(2(1leaf."Amber")(2leaf."Amethyst")(3leaf."Apple green")(4leaf."Aqua"))(3(1leaf."Aquamarine")(2leaf."Auburn")(3leaf."Aureolin")(4leaf."Azure"))(4(1leaf."Beige")(2leaf."Black")(3leaf."Bronze")(4(1leaf."Blue")(2leaf."Burgundy")(3leaf."Candy apple red"))))
I think the library turned out to be pretty clean, since it knows
nothing of points, buffers or overlays, and imposes no restrictions on
the type of leaf items and keys.
Some cool avy-based commands
I'll list them together with the code, so it's easier to see what they do.
The basic customizable variable is this one:
(defcustomavy-keys'(?a?s?d?f?g?h?j?k?l)"Keys for jumping.")
Note that, while ace-jump-mode has 52 selection chars by default, I
prefer to have only the 8 chars on the home row. This means that I'll
usually have to go around one level deeper, but the characters are
easy to find and press.
avy-jump-double-char
(defunavy-jump-double-char()"Read two chars and jump to them in current window."(interactive)(avy--process(avy--regex-candidates(string(read-char"char 1: ")(read-char"char 2: "))(selected-window))#'avy--goto))
This one will read two chars and then offer avy-selection for the
matches. This is a pretty sensible approach for a pool of 8 keys,
since usually 3 chars total will be necessary, with the first two
being in natural succession.
Here's a screenshot of me typing in a natural two-char sequence "do":
As you see, with 8 keys, 8 candidates will have depth 1, and another 8
candidates will have depth 2. The sorting preference is for the first
candidates to have lower depth.
avy-jump-line
(defunavy-jump-line()"Jump to a line start in current buffer."(interactive)(let((we(window-end))candidates)(save-excursion(goto-char(window-start))(while(<(point)we)(push(cons(point)(selected-window))candidates)(forward-line1)))(avy--process(nreversecandidates)#'avy--gotot)))
This one is quite nice, since I always have less than 8*8=64 lines in any window.
Here's how it looks like:
I removed the gray background, since the leading chars are always in an expected position.
avy-jump-isearch
Saving the most clever one for last:
(defunavy-jump-isearch()"Jump to one of the current isearch candidates."(interactive)(let((candidates(mapcar(lambda(x)(cons(1+(carx))(cdrx)))(avy--regex-candidatesisearch-string))))(avy--processcandidates#'avy--gotot)(isearch-done)))(define-keyisearch-mode-map"'"'avy-jump-isearch)
I don't mind not being able to isearch-forward-regexp for a single
quote without using C-q (quoted-insert). In return, I
get the ability to very quickly jump to a search candidate on
screen. I like this command the most. In case when there's only one match, it's a faster way
to call isearch-done (than C-m).
Here's the result of C-ssen':
Outro
I've used the new functionality for a few days already, so it
shouldn't be fragile. If the newest MELPA version bugs out for you,
you can fall back to version 0.7.1 on
MELPA Stable and post
an issue. Finally, I
hope that some people will take advantage of avy.el simplicity and
come up with some cool new commands to share with me. Happy hacking!
Did you know that you can repeat the most recent Emacs command with repeat:
Repeat most recently executed command.
If REPEAT-ARG is non-nil (interactively, with a prefix argument),
supply a prefix argument to that command. Otherwise, give the
command the same prefix argument it was given before, if any.
The default binding to repeat is C-x z. You can then
continue with just z: C-x zzzzzzzz.
Unfortunately, since defhydra defines new command names based on the
ones that you give it, passing repeat as one of the heads will not
work. So I've added hydra-repeat that's supposed to work in the same way.
It's slightly annoying that the buffer contents have to shift to
accommodate for a larger character, but there's no way around it.
Besides, you can customize the face features other than height, like
foreground and background colors etc.
Note that in the screenshot, I'm using my usual ace-window setup,
i.e. home row keys and no background:
As a follow-up to my older post on Elisp linting options,
I've made check-declare-file and check-declare-directory give out warnings similar to those
that e.g. byte-compile-file gives:
each warning now has a location in a file-line-column format
you can click on the link to jump to the warning location
first-error / next-error / previous-error work as well
the warning line blinks momentarily with each jump
Here's how it looks like:
You can already use this feature if you're on Emacs trunk, otherwise
you'll have to wait for 25.1 to come out.
There's also a new custom variable that you can set to get a more strict check:
(setqcheck-declare-ext-errorst)
Unless you set it, the standard behavior is to issue a "skipping
external file" message when checking a statement like the one above.
The reason is that the external package might not be loaded or
something. As long as I'm checking, I prefer to check everything, so
there's no reason not to have check-declare-ext-errors always true.
As you see, I first find where slime.el is located using locate-library, and then add the
contrib sub-directory to the load-path. After this, it's finally possible to (require 'slime-repl).
Today, I'll show a very useful hydra that I found yesterday on the
hydra wiki.
The idea is by @zhaojiangbin, I've
made some minor changes to get the arrows to work the way that I like,
and made all keys into plain letters.
There was a lot of screen estate left over, so I added some ASCII-art. I wanted something
related to syrup, or at least pancakes, but instead I found a cat. Apparently, it's very
easy to find pictures of cats on the internet. Who knew.
Here's how it looks like in-action:
The pink variation is pretty useful here, since it doesn't get in the
way of e.g. DEL or C-n or C-e or inserting spaces.
I've been using it today for editing some table data in org-mode,
and it feels pretty efficient.
Some explanations
What does what:
d deletes rectangle; it's similar to C-d.
n copies rectangle; it's similar to M-w.
o exits; it's very easy to press.
e exchanges the point and mark; it's also quite useful to re-activate the region
if you disabled it with n or r.
s fills the selected rectangle with a string.
y yanks the rectangle that you saved before with n.
r deactivates or activates the rectangle at point.
The package that I'm using,
profile-dotemacs.el
comes from David Engster, one of the major contributors to CEDET, so
you know it's got to be good. Once you launch it as I showed above,
it will give you the times that various sections took to load in an
overlay.
On very sunny days I can almost get Emacs to start under one second,
although when I checked just now, it turned out to be 1.20s. I
value short start up time very highly, since due to my messing with
Elisp, the Emacs global state can become so messed up that it's a lot
faster to just restart it, rather than spend 10 minutes surgically
restoring the state. And, of course, it's a bonus for using Emacs on a
laptop, which I do at home.
I've used two methods to shorten the boot time, I'll describe them below.
use-package:idle option
This one is a big time saver, since org-mode just takes so long to load:
(use-packageorg:defert:idle(require'oleh/org))
I'm cautious not to abuse :idle, since if it happens that I start
Emacs and start frantically typing all over the place, Emacs will
never get to be idle and the :idle statements won't load. So I only
use it for org and one or two other things.
This is one of the simpler ones, I think it would take around a week
of posts to cover what I have in magit.el. The file Info.el never
gets loaded until I open a buffer in Info-mode.
Here's what gets loaded every time:
My loaddefs.el is just over 1000 lines and is automatically
generated by update-all-autoloads. And here are the lines in
loaddefs.el responsible for loading Info.el:
Almost every directory I work with, is either directly under “~/Dropbox/” or
under “~/Dropbox/Work/”. However, I never actually visit these two directories,
only the directories inside them. The number of possible targets for find-file
is approaching the outer borders of two-digit land, so there's no hope for my
brain to remember registers for all of these.
The best solution I've found is to compile a list of all possible targets
(directories inside those directories) and offer them using ido. This is
somewhat similar to using a bookmark for each possible destination, except it
it's always up to date with the directory contents and doesn't clog up my actual
bookmarks.
(require'ido)(require'cl-lib)(defcustomendless/favorite-directories'("~/Dropbox/Trabalho/""~/Dropbox/")"List of favorite directories.
Used in `endless/visit-favorite-dir'. The order here
affects the order that completions will be offered.":type'(repeatdirectory):group'endless)(defunendless/visit-favorite-dir(files-too)"Offer all directories inside a set of directories.
Compile a list of all directories inside each element of
`endless/favorite-directories', and visit one of them with
`ido-completing-read'.
With prefix argument FILES-TOO also offer to find files."(interactive"P")(let((completions(mapcar#'abbreviate-file-name(cl-remove-if-not(iffiles-too#'file-readable-p#'file-directory-p)(apply#'append(mapcar(lambda(x)(directory-files(expand-file-namex)t"^[^\.].*"t))endless/favorite-directories))))))(dired(ido-completing-read"Open directory: "completions'ignorednil""))));; Note that C-x d is usually bound to dired. I find;; this redundant with C-x C-f, so I don't mind;; overriding it, but you should know before you do.(define-keyctl-x-map"d"#'endless/visit-favorite-dir)
Some random notes:
Having a quick key for this is fantastic. Whenever I want to get work done, I
just hit C-x d and I'll my options are presented before me.
With a prefix argument it also shows files instead of just directories.
If ido is not your completion engine of choice, that's trivial to change.
(hydra-cleanup): Should not delete buffer first when window is dedicated.
(lv-message): Should not deactivate mark.
quit signals are intercepted, so that proper clean-up is done.
no defuns with the same name will be generated in case some heads' cmd parts coincide
New Features
Pink body color
When body color is pink, and hydra intercepts a binding which isn't a head, instead of quitting the
hydra and executing the binding (default red behavior), it will execute the binding and not quit
the hydra. This type of hydra is very similar to define-minor-mode.
So now, pink and amaranth are body colors that are a flavor of the default red that, instead of
quitting when a non-head binding is intercepted, respectively run the binding or issue a warning.
Teal body color
Teal color behaves similarly to amaranth (issues a warning when intercepting foreign bindings),
but the default behavior for each head is to quit.
So if you want to write less, use teal color in this case.
Alternative color-less syntax
The color syntax is working quite well, since I was able to keep an almost full backward-compatibility:
once you pinpoint a color it will have that behavior, all the new behaviors will be released under a new color.
Some might prefer an alternative set of switches rather than colors. New compat switches are:
- ":exit nil" for ":color red"; You don't have to specify either
option in the body, since they are the default behavior that was
available on Hydra release
- ":exit t" for ":color blue"
- ":foreign-keys warn" for ":color amaranth"
- ":foreign-keys warn :exit t" for ":color teal"
- ":foreign-keys run" for ":color pink"
The property :exit can be inherited and overridden in the same way as :color red or blue.
(defhydrahydra-marked-items(dired-mode-map"")"Number of marked items: %(length (dired-get-marked-files))Auto-revert mode is: %`auto-revert-mode"("m"dired-mark"mark"))
To enable this behavior, the docstring has to start with a newline (which will be stripped). This
makes sense, since you usually want to align extensive docstrings, like I did this one. The escape
syntax for variables is such that you can auto-complete them with company-mode. Both variables
and sexps can use format style width specifiers. See the documentation of format for more
details.
Examples 7-9 added to hydra-examples.el
These examples explain Ruby-style interpolation.
Updated faces
I've updated each face to match their color by default. Remember, that you can customize each
face interactively with M-xcustomize-grouphydra.
Alternative grey-scale or custom faces
If you don't want colors for some reason, you can customize hydra-fontify-head-function to your
taste.
This function is already defined in hydra.el, but you could define a similar one on your own:
(defunhydra-fontify-head-greyscale(headbody)"Produce a pretty string from HEAD and BODY.HEAD's binding is returned as a string wrapped with [] or {}."(let((color(hydra--head-colorheadbody)))(format(if(eqcolor'blue)"[%s]""(%s)")(carhead))))
New switch :body-pre
This sexp or function will be prepended to the prefix/body function.
Here's an example of use:
Here, upon entering the hydra, the point position will be saved. And upon quitting, it will be
restored.
Outro
I hope that you're enjoying the new updates as much as I do.
Since the last release, there are two new blog posts about using hydra that I'm aware of:
by Howard Abrams and
John Kitchin.
Additionally, @hura,
@kaushalmodi, and
@glucas have made some additions to the
Hydra wiki. Anyone is welcome
to share their useful hydras there. Thanks a lot and happy hacking!
This is a new feature I've added recently to lispy
that's useful when debugging regex-related code.
The gist of it is that the match data is a single global object in Emacs.
So if you call string-match with your C-x C-e, there's no
guarantee that e.g. match-beginning will return the proper thing with another C-x C-e,
since any package running in your Emacs could mess with the match data (packages that use timers
or post-command hooks etc.).
After getting annoyed by this a few times, I've finally added a
fail-safe to e (lispy-eval) and p
(lispy-eval-other-window). Here's how it looks like:
(defvarlispy-eval-match-datanil)(defunlispy--eval-elisp-form(formlexical)"Eval FORM and return its value.If LEXICAL is t, evaluate using lexical scoping.Restore and save `lispy-eval-match-data' appropriately,so that no other packages disturb the match data."(let(val)(fset'\,#'identity)(set-match-datalispy-eval-match-data)(setqval(evalformlexical))(setqlispy-eval-match-data(match-data))(fset'\,nil)val))
There's also a little dance of ignoring comma operators in the rare
case when I want to eval inside a backquoted list. The two
functions that you can take away from this exercise are match-data
and set-match-data which appropriately return and store a list of
integers. Keeping the string separate from the regex match is a neat
way to improve performance.
In yesterday's post
I showed how to embed variable values into the docstring, among other
things. Today, I've extended this approach to work with s-expressions.
Here's how it looks like:
(defhydrahydra-marked-items(dired-mode-map"")"Number of marked items: %(length (dired-get-marked-files))"("m"dired-mark"mark"))
This piece of code will remind you how many files you've marked so far each
time you press m.
By the way, this is the 64th post on the blog; I found out by pressing
tm in dired buffer or the _posts directory.
Also, I don't think that I've showed passing "" as the keyboard prefix parameter
before. Apparently, it works and just translates to this:
Here's a little code I've made today for Buffer-menu-mode:
(defhydrahydra-buffer-menu(:colorpink)" Mark Unmark Actions Search-------------------------------------------------------------------------_m_: mark _u_: unmark _x_: execute _R_: re-isearch_s_: save _U_: unmark up _b_: bury _I_: isearch_d_: delete _g_: refresh _O_: multi-occur_D_: delete up _T_: files only: %`Buffer-menu-files-only_~_: modified"("m"Buffer-menu-marknil)("u"Buffer-menu-unmarknil)("U"Buffer-menu-backup-unmarknil)("d"Buffer-menu-deletenil)("D"Buffer-menu-delete-backwardsnil)("s"Buffer-menu-savenil)("~"Buffer-menu-not-modifiednil)("x"Buffer-menu-executenil)("b"Buffer-menu-burynil)("g"revert-buffernil)("T"Buffer-menu-toggle-files-onlynil)("O"Buffer-menu-multi-occurnil:colorblue)("I"Buffer-menu-isearch-buffersnil:colorblue)("R"Buffer-menu-isearch-buffers-regexpnil:colorblue)("c"nil"cancel")("v"Buffer-menu-select"select":colorblue)("o"Buffer-menu-other-window"other-window":colorblue)("q"quit-window"quit":colorblue))(define-keyBuffer-menu-mode-map"."'hydra-buffer-menu/body)
You can change the color on the top to your taste: either red or pink or amaranth will work.
For extensive Hydras I tend to pick pink, since I don't want to quit by accident, while still keeping
the non-head bindings. For small ones, red is better.
Here's how the result looks like:
You can install the cow with:
apt-get moo
I didn't list it in the source, since Jekyll would wrap the long lines. The source is already in hydra-examples.el.
You only have to:
I've been hearing some opinions lately that the growing number of
Hydra options makes it intimidating or unclear. I hope that's not the
majority's feeling and that the gentle repetitiveness of this example
proves otherwise.
On the other hand, I've started reading "The Reasoned Schemer" this
week. Now that's intimidating.
Two new colors are being added up to a total of five: red, blue,
amaranth, pink and teal.
I should carefully restate what they do to avoid confusion.
The three rules of Hydratics
1. A hydra may not injure a human being or, through inaction, allow a human being to come to harm.
Seriously though, see below.
Rule 1: Hydra heads are either red or blue
Once you're in a Hydra state:
calling a red head will call the command and continue the state
calling a blue head will call the command and stop the state
They may have a reddish or a bluish face that isn't exactly red or
blue, but that's what they are underneath. I hope you get what I mean.
Rule 2: red or blue is inherited from the body color
This is merely a convenience, you can still explicitly override each
head to be blue or red:
if the body is red, amaranth or pink, the heads inherit red
if the body is blue or teal, the heads inherit blue
Rule 3:
When you call a binding which isn't a head:
amaranth, teal and pink Hydras will intercept it
red and blue Hydras will quit and let Emacs execute your binding
Finally, on intercepting a non-head, amaranth and teal will issue a
warning and do nothing without quitting. And pink will try to call the
intercepted command without quitting. Currently only non-prefix
bindings can be called, since I haven't figured out how to do it for prefixes.
A nice table to sum things up
Thanks to @kaushalmodi for pointing me in this direction:
|----------+-----------+-----------------------+-----------------|
| Body | Head | Executing NON-HEADS | Executing HEADS |
| Color | Inherited | | |
| | Color | | |
|----------+-----------+-----------------------+-----------------|
| amaranth | red | Disallow and Continue | Continue |
| teal | blue | Disallow and Continue | Quit |
| pink | red | Allow and Continue | Continue |
| red | red | Allow and Quit | Continue |
| blue | blue | Allow and Quit | Quit |
|----------+-----------+-----------------------+-----------------|
The extra-awesome Ruby-style Hydra docstrings
Turns out learning Ruby wasn't a complete waste of time, at least I
learned about string interpolation. And now I'm sticking it into Elisp
packages, first tiny, and now
hydra.
Here, using e.g. "_a_" translates to "a" with proper face. More
interestingly, e.g.
"foobar %`abbrev-mode"
translates roughly to
(format"foobar %S"abbrev-mode)
This means that you actually see the state of the mode that you're
changing. The escape syntax was chosen with another intent in mind:
because of the backquote, if you have company-mode on, you can
complete the symbols while in string.
See how it looks like in action:
Small note on pink Hydras
It's useful for instance it the above example, when I don't care about
self-inserting, but I still want to do navigation. Basically pink
Hydra is the closest thing to an actual minor mode. Thanks to
@angelic-sedition for the idea.
Small note on teal Hydras
It provides an interface similar to magit dispatch: pressing
appropriate keys does things and pressing the wrong keys issues a
warning. The only difference between teal and amaranth is the color
inheritance, otherwise they behave exactly the same. This means that
if you want a non-quitting Hydra that will end up with more blue
heads, start with teal, otherwise, start with amaranth.
Thanks to @ffevotte for the idea.
Outro
It feels like things are finally falling into place with this package.
I hope that you like the new changes and find new cool uses for the added abilities.
Happy hacking!
It happens to me frequently when making a demo, or just experimenting
around that I want to revert the buffer to the last saved
state. Obviously such a command exists in Emacs, and it's
unsurprisingly called revert-buffer:
Replace current buffer text with the text of the visited file on disk.
This undoes all changes since the file was visited or saved.
But the poor old revert-buffer isn't bound by default in Emacs. The perfect binding for it is C-x C-r:
I believe the lambda is used to get the "no questions asked"
treatment. Anyway, by default Emacs binds C-x C-r to
find-file-read-only, which really is a trash-tier command.
Why is find-file-read-only useless?
Because C-x C-q is bound to read-only-mode - a really
good command (made even better with wdired and wgrep). So
find-file-read-only is just C-x C-f C-x C-q. And I don't
ever recall needing to use that, since v
(dired-view-file) is almost equivalent if not better.
As a bonus, C-x C-r is mnemonic for "revert".
Auto-reverting
While on the topic, let me mention auto-revert-mode. I have this in my config:
(global-auto-revert-mode1)
This way, if any opened and saved file was modified outside of Emacs, it will be updated
in a short while. I was able to use this to a great advantage when writing the SVG picture
for the Xmodmap post:
I opened the SVG in the default image mode in one Emacs instance
and opened the same document in XML mode in another (mode is toggled with C-c C-c)
With that setup, I was able to see the picture update as I was inputting the XML.
Here's some Elisp for XML generation, if you're interested
(defunmake-row(row-strixiyfill)(mapconcat(lambda(x)(format"<g><text x=\"%d\" y=\"%d\" style=\"font-size:10px;fill:%s;font-family:Deja Vu Sans Mono\">%s</text></g>"(+ix(*x30))iyfill(let((y(eltrow-strx)))(if(stringpy)y(make-string1y)))))(number-sequence0(1-(lengthrow-str)))"\n"))(make-row"QWERTYUIOP"95140"#2b2828");; =>
The last statement would generate:
Q
W
E
R
T
Y
U
I
O
P.
As usual, I'll just re-state the
release notes,
while maybe adding a bit of flavor.
New features
Define Hydra heads that don't show up in the hint at all
This can be done by setting the head's hint explicitly to nil, instead of the usual string. For
instance, if you always tend to bind the arrows to hjkl, there's no point to show a hint
for them.
Use a dedicated window for Hydra hints
Since version 0.10.0, setting hydra-lv to t (the default setting) will make it use a dedicated
window right above the Echo Area for hints. This has the advantage that you can immediately see any
message output from the functions that you call, since Hydra no longer uses message to display
the hint. You can still have the old behavior by setting hydra-lv to nil.
How it looks like:
Here, an error was triggered by previous-line and the message is displayed without interrupting
the hint.
Allow duplicate functions in heads
Duplicate functions will be concatenated in the hint.
This was already covered in yesterday's post
Add option to font-lock defhydra
If you want to nicely font-lock your defhydra statements, just add this to your config:
(require'hydra)(hydra-add-font-lock)
Additionally, defhydra is now to be indented as a defun, so it will be indented like this:
Note that the indentation of the body argument is as if it was data
and not code, i.e. the proper one. As you see, I even added defhydra
to the blog's Pygments list for Elisp.
Incompatible changes
The macro hydra-create, as well as the variables that were supposed
to be used with it (hydra-example-text-scale,
hydra-example-move-window-splitter, hydra-example-goto-error,
hydra-example-windmove) were removed. All the functionality is still
there in hydra-examples.el with the better defhydra macro.
Outro
I hope that you like the new changes. And if you're byte-compiling
your code that uses defhydra, don't forget to re-compile. Also, the
latest version of ace-window will ignore *LV* buffer while
switching, as it did for *Calc Trail* before.
I've just released Paradox version 2.0. If you've been following the blog, you
already know this features the ability to do background upgrades. Below is a list
of other features.
However, I'd first like to say this release requires an Emacs version of at
least 24.4. package.el changed a lot in that release, and the trouble of
supporting two separate versions was keeping me from writing new features. I may
implement backwards compatibility if I see people asking for it.
Some new features
paradox-execute-asynchronously variable
Set to t or nil depending on
whether you want this feature.
paradox-upgrade-packages command
Upgrade everything, wherever you are.
Combines well with the above.
paradox-after-execute-functions variable
A hook run after every
transaction. Remove functions from it to streamline your experience, or add
your own functions to extend functionality. For instance, if you don't like
the new *Paradox Report* buffer, you can prevent its creation by removing
the relevant functions from the hook:
Now, when you hit x in the Paradox Menu to execute a
set of installations and deletions, the buffer narrows to display only
those packages (see below). Much nicer than the usual message in the echo
area.
These are the main changes. You can also view package commit lists, but that's
been the case for a while.
Here, the entry points are <f2> g, <f2> l, and <f2> 0, the
others aren't bound in the global map.
You can also have the same function in both red and blue versions. Here's how the hint will look like:
The multiply-defined functions will be neatly grouped together.
If you remember, (text-scale-set 0) uses a sexp syntax for a Hydra
head: it will be wrapped in (lambda () (interactive) ...) automatically.
Somehow, I accumulated a few files owned by root that should have been
owned by me. Below, I'll describe a solution to get rid of them based
on dired and tramp.
Step one: open the directory with sudo privileges
I'll obviously need the sudo privilege to change the owner of a file owned
by root. Might as well start with that.
Thankfully, I do have a shortcut:
(defunsudired()(interactive)(require'tramp)(let((dir(expand-file-namedefault-directory)))(if(string-match"^/sudo:"dir)(user-error"Already in sudo")(dired(concat"/sudo::"dir)))))(define-keydired-mode-map"!"'sudired)
The function above will open the current directory in sudo mode. I
decided to bind it to !, since the default &
seems strictly better than !. The function will ask you for
the password once. Afterwards, you can open other directories without
having to enter the password.
Step two: find-dired
So now, while in sudo mode of the directory in question:
Call M-xfind-dired. It's not bound by default, and I
don't bind it myself since it's quite situational (I do bind
find-name-dired to F in dired-mode though).
It prompts you for the directory. I just press RET, since the default one is what I need.
Next, it prompts for find args. These are the arguments to the
UNIX command find. Here, I need to pass -user root.
The command has finished and I can see the results. I can select
some of them by pressing m (dired-mark) a few times, or
mark them all at once with t (dired-toggle-marks).
Finally, call O (dired-do-chown) on the marked files and enter oleh.
Done. I'm sure that the admin types can cook up some find-exec-chown combo to do the same thing,
but the find-dired approach is both simpler and more flexible, since I can confirm and edit the list
of files being changed.
Today, I'll share my keyboard layout, based on QWERTY, that
facilitates all tech-related activities, especially Emacs. I've been
using a version of this for about three years now, and only a single
key has been changed in the last two years. Without further ado,
here it is:
The most visible change
I've made ; stop inserting ; and instead act as
Mode_switch: a modifier similar to control or meta.
I'm pressing ;-j to enter ;
I'm pressing ;-d to enter :
So I'm giving up the ability to press ; with a single key,
and remapping : from one two-key combination to another
two-key combination.
In return, I'm getting a QWERTY with all these wonderful shortcuts.
The RSI-savers
The general ones are:
RET on ;-v (symmetric with C-m)
DEL on ;-o
If you're writing a bunch of C++, _ on ;-s is very good.
Likewise, - on ;-a is great for LISP.
For LaTeX, \ on ;-w is good since I switched from
a keyboard that had it near z to a keyboard that has it near RET.
I'm guessing ~ on ;-t isn't bad for shell, but I don't use it that much.
The basic remaps
These are just char-for-char remaps. They are composable, for instance
I have dired-jump on C-:, which means that I'm pressing
C-;-d.
Some math symbols are mnemonic: e-equal,
l-lesser, g-greater.
If you're using lispy, it might become more clear for you
why I've put lispy-slurp on > and lispy-barf on <. Although
pressing these keys with a shift isn't the worst thing, having them on the home row is pretty cool.
The pairs
Here are the translations meant for pairs:
q - θ,
f - φ,
r - ρ,
c - σ.
These are meant for Emacs and .inputrc. In both cases, e.g. φ should insert () and
go backward one char. Here's my .inputrc:
set input-meta on
set output-meta on
set convert-meta off
"θ":""\C-b""
"ω":"'\C-b'"
"\e\C-l":"\C-e | less\C-m"
"υ":">\C-b<"
"σ":"}\C-b{"
"φ":")\C-b("
"ρ":"]\C-b["
set completion-ignore-case on
With this .inputrc, it's like I have some kind of electric mode in my bash.
In Emacs, the pairs are bound in a straightforward way:
These are bound to various Emacs functions. In fact, I initially started with all Greek chars, and
bound e.g.:
(global-set-key(kbd"ε")"=")
When these remap-type bindings became mature, I've put them directly
into .Xmodmap, which gave the advantage of being able to compose the
keys.
These bindings are as good and usable as the control plus lower case
chars. Use them wisely and you can get a very ergonomic setup. Some
ideas:
switching buffers on ;-h
switching windows on ;-n
jumping to bookmarks on ;-m
The shifted digits
I find it much easier to press e.g. ;-4 to insert $,
rather than S-4. I also restored the justice of 0 coming before1 by mapping
;-` to 0.
Outro
I hope that you'll find some use and enjoyment in these ramappings, if
you're not afraid to experiment a little. The cost of losing the
semicolon is pretty minor, the bigger issue is in finding the
unmodified QWERTY extremely sluggish after using this approach.
I discovered today that I was using declare-function in a wrong way.
So I'll share how to use it properly.
declare-function
I was just using it to shut up the byte compiler's "not known to be
defined" warning. Turns out that it can also be used to check if the
functions actually exist in the file to which they point to. You can
use check-declare-file to check one file, or
check-declare-directory to recursively check the whole directory.
Here's an example output:
Warning (check-declare): helm-info.el said `Info-goto-node' was defined in info.el.gz: arglist mismatch
Warning (check-declare): helm-info.el said `Info-find-node' was defined in info.el.gz: arglist mismatch
Warning (check-declare): helm-plugin.el said `Info-goto-node' was defined in info.el.gz: arglist mismatch
Warning (check-declare): helm-plugin.el said `Info-find-node' was defined in info.el.gz: arglist mismatch
Warning (check-declare): helm-emms.el said `with-current-emms-playlist' was defined in emms.el: function not found
Warning (check-declare): projectile.el said `ggtags-ensure-project' was defined in ggtags.el: file not found
Warning (check-declare): projectile.el said `ggtags-update-tags' was defined in ggtags.el: file not found
Warning (check-declare): async-bytecomp.el said `package-desc-reqs' was defined in package.el.gz: function not found
My mistake was assuming that the FILE argument of declare-function
was somehow related to require. But of course it had to be simply
the name of the file that contains the said function. If the referenced file is in an external package,
e.g. (declare-function cider-repl-return "ext:cider-repl") can be used.
checkdoc
This one is actually very useful once you embrace it. It will
tsk-tsk you until all your functions are documented. And since
you're already writing a docstring, might as well make it
good. Sometimes this leads me to removing a function that's called
only once, just so that I don't have to document it.
byte-compile-file
Leaving the obvious for last. This will speed up the code in addition to checking for errors.
In dired you can use:
several m (dired-mark) followed by B (dired-do-byte-compile).
a single *% (dired-mark-files-regexp) el$ followed by B.
As I was refactoring my .emacs, I found a silly wrapper around
describe-function that uses ido-completing-read. I think I was
aware at the time that
ido-ubiquitous
could simply make describe-function use ido-completing-read. The
problem was that it also affected the completing-read in functions
that became headlong-bookmark-jump from
headlong
(see the corresponding post
if you missed it).
A bit of code to make things right
(defunido-occasional-completing-read(promptcollection&optionalpredicaterequire-matchinitial-inputhistdefinherit-input-method)"Use `ido-completing-read' if the collection isn't too large.Fall back to `completing-read' otherwise."(let((filtered-collection(all-completions""collectionpredicate)))(if(<=(lengthfiltered-collection)30000)(ido-completing-readpromptfiltered-collectionnilrequire-matchinitial-inputhistdefnil)(completing-readpromptcollectionpredicaterequire-matchinitial-inputhistdefinherit-input-method))));;;###autoload(defmacrowith-ido-completion(fun)"Wrap FUN in another interactive function with ido completion."`(defun,(intern(concat(symbol-namefun)"/with-ido"))(),(format"Forward to `%S' with ido completion."fun)(interactive)(let((completing-read-function'ido-occasional-completing-read))(call-interactively#',fun))))
The only thing that ido-occasional-completing-read does is to
pre-filter collection with predicate and pass it on to
ido-completing-read. And with-ido-completion is just a convenience wrapper.
Here, with-ido-completion will generate
e.g. describe-function/with-ido, which will subsequently be bound to
<f1> f. The good-old describe-function is left unaffected.
Note, that if I turn on helm-mode at this point, it will override
describe-function with its own completion, but it will not touch
describe-function/with-ido which I bound. This can be useful e.g. if
I want to use helm-mode (or icy-mode or icomplete-mode) for some
completion, but not all.
thoughtbot NYC
1st floor of the WeWork at Bryant Park
54 W. 40th St.
New York, NY
Jonathan Magen will be presenting How I Use org-capture and Stuff:
Org-capture is a great way to take notes and plan in Emacs. Capture
templates provide a flexible way to extend org-capture and
personalize the way you record various bits of information. This
talk will cover how Jonathan uses org-capture and provide an intro
to writing your own org-capture templates.
Harry Schwartz will be presenting on Searching the Web with
engine-mode:
I probably spend about 75% of my programming time looking up
documentation. That’s really easy in Emacs Lisp—my documentation
is built into my editor–but in most other languages I’m not so
lucky.
I wrote engine-mode to help fix this problem. It’s a simple minor
mode that lets me define arbitrary search engines and send snippets
of text to them from within Emacs. Now I don’t have to copy and
paste text between my browser and editor! So fancy.
We’ll be talking about how to get started with engine-mode (it’s
easy!) and then digging into the implementation and discussing the
process of committing a package to MELPA.
After the previous post, I got in touch with the nice fellas at emacs-devel
about including comment-line in Emacs. Understandably, there was a wee bit of
concern with using up one of the oh-so-important C- binds, so it's been put
under C-x C-; for now.
On the bright side, the entire previous code-block gets replaced by a single
keybind.
I've been cleaning up my .emacs lately, with the intention to put my
whole config on Github. Today, I'll show you a useful function for
describing the current buffer's key bindings. The function will
generate something like this (with all letters, of course):
Since the entry point of the whole thing is keys-describe-prefixes, only it needs to be autoloaded.
Once an autoloaded function is called, it will load the whole buffer. So if e.g. keys-latin is
not being used anywhere else outside this file, it doesn't need an autoload.
Redundant functions
keys-latin and keys-greek are actually very small and not used anywhere else. It might be better
to just inline them into keys-describe-prefixes.
Redundant libraries
Here, dash is required just for the ->> macro, which can actually
be obtained from a core library subr-x as thread-last.
But even then, it's just better to unwind the whole thing. After that, it becomes clear that
the three consecutive mapcars could be folded into a single mapcar with the help of a let binding.
After the fold, it starts to look silly, since I'm consing just to
take a car and cdr later:
I hope that this sort of analysis can be useful for people starting to
learn Elisp. And if you have corrections for the final version, do
let me know, I don't mind being schooled, as long as I get better in
the end.
The output of keys-describe-prefixes can be used to learn some
bindings that you didn't know about, and also as a template to
redefine some bindings that you don't need.
Heh, I guess I was so busy at work that I missed the opportunity to
make an especially good post for the 50-post landmark. At least
according to M-= (count-words-region) in my _posts'
dired buffer, this is the 53rd post.
ido-backspace
This echoes to my very second post,
easy helm improvement,
which was about implementing a similar function for helm.
So here it is:
(ido-mode)(require'delsel)(defunido-backspace()"Forward to `backward-delete-char'.On error (read-only), quit without selecting."(interactive)(condition-casenil(backward-delete-char1)(error(minibuffer-keyboard-quit))))(define-keyido-common-completion-map(kbd"DEL")'ido-backspace)
With this setup, when e.g. calling ido-switch-buffer, you can cancel
with DEL. This, of course, will also work for
smex and
someother packages that allow for
ido completion:
Note that the define-key part of the code takes an advantage of a
recent patch to ido that's available in the trunk. You can find the
old way of binding keys for ido in an earlier post,
tilde in ido-find-file.
That post actually played a small role in speeding up the patch, for
which we should thank @tarsius.
Note that this behavior will not affect ido-find-file, since it has its own map.
As I was releasing lispy 0.23.0, which added
an option for methods of jumping to semantic tags other than helm, I've discovered
ido-vertical-mode. I also made some changes to
it, so now it looks like this:
The changes are:
the current text is highlighted in all matches
the current number of total matches is displayed in the first line
I also customized the face for the first match and the only match (you can't see that one here)
Helm is still the big gun when it comes to completing lists with hundreds of candidates,
but with the above changes, ido-vertical-mode comes close to what I need when jumping to tags.
The only thing missing now is that the selected candidate is always on the first line,
even when I press C-n, which isn't very intuitive.
Stuff related to lispy release
Other completion methods available in lispy-goto
Depending on the modes you have enabled, you can complete with:
helm
ido
ido-vertical-mode
icomplete-mode
icy-mode
no mode, just plain completion
lispy-eval-other-window now uses ace-window
This function bound to p is very convenient for debugging
Elisp: it allows you to eval the current sexp in the context of other
window. This is very powerful: you can run and debug your function as
you write it, statement by statement. Previously, this setup was
viable only in a two-window split, because using other-window, it
wasn't predictable in which window the eval would take place. Now, if
there are more than two windows, aw-select will select the window
for the eval. This selection will be remembered, so you don't have to
select for the following evals, unless the window configuration
changes.
Outro
There are more changes that you can find in
the release notes
if you're interested. You can find the new ido-vertical stuff in
my fork, until it
gets merged.
Paradox has had this feature
for a while, but I've never blogged about it. When you're in the Packages Menu, and you're about to upgrade your
packages with the always reliable U x, do you ever wonder what's actually
changed in them? When I do it, there's usually something between 5 and 20 of
them just waiting for me to answer y, but rarely do I see any actual
difference in the new versions.
With Paradox you can find out what's been happening to a package, by listing all
of its commits that happened since the version you have installed. Of course,
this isn't a proper Change-Log —for starters, it's a lot noisier— but since
most packages don't have one, this is the closest thing we have.
Open the menu as usual with paradox-list-packages (which really ought to be
in your launcher map).
Hit f u to filter by upgradeable packages (did you know you could do that,
by the way?).
Hit l on any package to list the commits.
Here's an example of what you might see.
Note how the first three commits are highlighted. These are the ones I'm about
to install. It even tells me there a comments discussing one of them, so I might
want to have a closer look. Hitting RET on a line will visit the relevant
commit on Github.
As you may have guessed, this only applies to packages on Github. But, at this
point, that's where most of them are.
The body, like the heads will recognize the :bind property in PLIST.
The heads will inherit it, just like they do with :color.
The :bind property can be nil or a lambda of global-set-key format.
Here, bind-key will be used to bind goto-char to M-g c,
since c head has inherited body's :bind property. Note
that since bind-key is a macro, it was necessary to wrap it in a
lambda.
However, global-set-key will be used to bind goto-line to M-g g, this :bind property
was overridden in the g head.
Since this commit, it's not possible to pass a lambda instead of the
whole BODY argument, as was advertised before. Just put it on
:bind now.
hydra/body will pass the initial current-prefix-arg along
Now, C-u C-z l will result in (forward-char 4). All the other l
will normally call (forward-char 1), unless an additional prefix is given. The previous
behavior allowed only for C-z C-u l to get (forward-char 4).
Here, r and w heads are using this feature.
Here's what will be generated, if you're interested:
(defunhydra-launcher/lambda-wnil"Create a hydra with no body and the heads:\"h\": `man',\"r\": `(browse-url \"http://www.reddit.com/r/emacs/\")',\"w\": `(browse-url \"http://www.emacswiki.org/\")',\"s\": `shell',\"q\": `nil'The body can be accessed via `hydra-launcher/body'.Call the head: `(browse-url \"http://www.emacswiki.org/\")'."(interactive)(hydra-disable)(catch(quotehydra-disable)(call-interactively(function(lambdanil(interactive)(browse-url"http://www.emacswiki.org/"))))))
Obsolete declarations
hydra-create and all old examples in hydra-examples.el are now obsolete.
You can still use them for a short while, but they will be removed soon.
You should take the time to switch from hydra-create to defhydra. All the old examples
are provided in the new style in hydra-examples.el. However, they will not be evaluated
through (require 'hydra-examples) unless you (setq hydra-examples-verbatim t)
beforehand. This is because I have no idea what kind of bindings will work for you, you
should decide yourself. But I am providing you with a template. The number of examples
has also grown to six.
You can see the release notes here.
There's not a whole lot of user-visible changes, but a large portion of the code was re-written
to facilitate the use of the API. I hope that no new bugs were introduced with this change,
you can still fall back to 0.6.1 and send
me an issue if something broke. There's a benefit
in the long run, just see how simple the code has become:
On very rare occasions it happens that a question matches two packages at once.
Here is such a question:
Is there a yasnippet producing a prepopulated doxygen comment?
For the following C++ function:
bool importantStuff(double a, double b);
It should output the following snippet, perhaps without the tags:
/**
* <Insert description of importantStuff>
*
* @param a <Insert description of a>
* @param b <Insert description of b>
* @return <Insert description of the return value>
*/
Well, I've got a package called function-args
for C++ that uses CEDET to:
display tool-tips with function arguments
jump to a semantic tag in current file
do a bit of completion
do a bit of generation (like inherited function signatures)
I think it might be useful to look at the code, since it's small and shows how to use CEDET's and
auto-yasnippet's API:
(defunmoo-doxygen()"Generate a doxygen yasnippet and expand it with `aya-expand'.The point should be on the top-level function name."(interactive)(move-beginning-of-linenil)(let((tag(semantic-current-tag)))(unless(semantic-tag-of-class-ptag'function)(error"Expected function, got %S"tag))(let*((name(semantic-tag-nametag))(attrs(semantic-tag-attributestag))(args(plist-getattrs:arguments))(ord1))(setqaya-current(format"/*** $1*%s* @return $%d*/"(mapconcat(lambda(x)(format"* @param %s $%d"(carx)(incford)))args"\n")(incford)))(aya-expand))))
The bonus
If you're interested in Doxygen, here's a
bit of code that I found laying around. It will prettify
e.g. <tt>Numerical Recipies</tt> to Numerical
Recipies in the comments.
This code uses a similar approach to the one used in my posts
about prettifying Elisp regex,
and ElTeX.
(deffacefont-lock-doxygen-face'((nil(:foreground"SaddleBrown":background"#f7f7f7")))"Special face to highlight doxygen tags such as <tt>...</tt>and <code>...</code>.":group'font-lock-highlighting-faces)(font-lock-add-keywords'c++-mode'(("\\(<\\(?:code\\|tt\\)>\"?\\)\\([^<]*?\\)\\(\"?</\\(?:code\\|tt\\)>\\)"(0(prog1()(let*((expr(match-string-no-properties2))(expr-len(lengthexpr)))(if(eq1expr-len)(compose-region(match-beginning0)(match-end0)(arefexpr0))(compose-region(match-beginning1)(1+(match-end1))(arefexpr0))(compose-region(1-(match-beginning3))(match-end3)(arefexpr(1-expr-len)))))))(0'font-lock-doxygen-facet))))
Outro
I hope that the code listed here will be useful to someone other than
me. It's a nice highlight of how a language can be made much cooler
by just having a package manager that allows to quickly glue various
things together. I mean, what would happen to JavaScript without npm
(besides stopping people requiring Angular just to use arrays)?
C++, on the other hand, is too cool to care about such things as
package managers and modules and stuff. But we're stuck with it for
performance reasons, so might as well try to make the experience more
bearable by adding some niceties to c++-mode.
An amaranth planted in a garden near a Rose-Tree, thus addressed it:
"What a lovely flower is the Rose, a favorite alike with Gods and
with men. I envy you your beauty and your perfume." The Rose
replied, "I indeed, dear Amaranth, flourish but for a brief season!
If no cruel hand pluck me from my stem, yet I must perish by an
early doom. But thou art immortal and dost never fade, but bloomest
for ever in renewed youth."
Prompted by the "Avoid exiting the hydra on hitting a wrong key" issue,
I decided to add a new body color for Hydra. Turns out that there exists a color that represents
immortality, and it's called amaranth. It's also very pretty.
So what amaranth Hydras do basically, is make it impossible to quit them.
Of course, it would be totally lame if there wasn't at least one way to quit them.
And what better to utilize for this purpose,
than the good-old quitters - the blue Hydra heads.
In addition to the usual arrows on hjkl, a and
e move to beginning and end of line respectively.
Outro
I hope that you like the new idea, also thanks to
@vkazanov for the push in this
direction. Things should be backwards-compatible, so this feature
costs you nothing if you're not using it. See the release notes for 0.8.0.
One more thanks goes to Sridhar Ratnakumar for this suggestion
that lead to an improvement in auto-yasnippet. Thanks to this improvement,
I was able to quickly wrap words like this:
Only three commits happened since 0.6.1, but already 0.7.0 has to
be released, since the behavior has changed slightly.
The ultimate window switching setup, revised
The code from the last post had to be changed.
Note that only ace-window-related heads are affected, since ace-window uses set-transient-map as well.
This was necessary to fix issue #15 (unable to use goto-line in a Hydra, since it requires input).
You can also see an awesome addition sneak in, with transpose-frame
on t.
Technical note
If you look closely at the code, you'll see the ace-window-related
lambdas throwing the hydra-disable symbol. If the Hydra code catches
that symbol, it will not call set-transient-map again, effectively
making the head that threw the symbol blue. In this example, though,
ace-window-related functions are still red, since the Hydra is
resumed by calling hydra-window/body in ace-window-end-once-hook.
This is a trick that I have to play because of how ace-jump-mode
works. Hopefully, it won't be needed for other commands. But still,
it's a simple trick that you can use if you want to have a head quit
conditionally, or something.
In this example, the cursor color will change for the duration of the Hydra.
Both :pre and :post should match to a single Elisp statement; you can use progn to tie
a bunch of statements together if you want.
A more evil helm
Lit Wakefield has a nice article on
using hydra and helm together. Do check it out, I'll certainly steal some of his code.
Outro
I hope that you don't have an objection to Hydra being developed at a
fast pace with many features popping up. I try my best to keep things
backwards compatible, but I really want to quickly fix the areas where
the code is lacking. Thanks to all the people contributing
issues, especially
@atykhonov and
@nandryshak for the last two, which are the core
of this version bump.
For that case, I would get the hint immediately after C-M-s and would be able to give the numeric argument immediately,
but I wouldn't be able to bind anything else on C-M-s as a prefix, e.g.:
The code above would give the error "Key sequence C-M-s z starts with non-prefix key C-M-s".
So you can pick the method that you prefer, the choice is there.
The credit for this monster goes to bcarell, I just refined his approach
with making ace-window not quit the hydra-window Hydra.
This setup needs the latest ace-window.
Here's the result of C-M-o xvxv starting from a single window:
From here, I can quickly maximize the current window with o
while simultaneously quitting the Hydra; i will maximize a
window as well, but it will select it with ace-window, instead of
maximizing the current one.
Note that, since numerical arguments are working now, 4a is
the same as s (swap) and 16a or C-u C-u
a is the same as d (delete).
When your computer is feeling slow and you decide to upgrade it, where do you
start? You start by finding the bottleneck, of course. That awesome CPU won't do
you any good with crappy RAM disk. The same logic holds for your coding skills.
Whenever you're writing code, there are two main processes going on —this is a
very rough model, but it gets the job done.
You think of what you'll write,
and you write it.
As you become more experienced with a language, the first part gets faster and
easier, to the point where you can immediately envisage the solution to any
problem you encounter. On the other hand, your actual writing speed is much
slower to improve, and caps at a very disappointing maximum.
The solution? Upgrade the bottle-neck, i.e., your hands. speed-of-thought-lisp
(or sotlisp) is all about helping your hands keep up with your brain. The mode
itself is quite simple, it provides two sets of tools to help your fingers follow
your thought-flow. To turn it on, just issue M-xspeed-of-thought-mode.
While you're at it, make sure you've got something like electric-pair-mode or
paredit active, sotlisp is a lot more fluid with these.
Minimal typing with abbrevs
A large number of intelligent abbrevs which expand a function's initials to its
name. A few examples:
i → insert
a → and
r → require '
wcb → with-current-buffer
efn → expand-file-name
It's pointless to list all of them here, there are 136 of them at the moment.
The whole purpose of these abbrevs is that you don't need to learn them. Instead
of typing insert SPC or maybe ins TAB, you just write i SPC for the same
effect. You'll be surprised how quickly this abbreviations become more natural
to you than the actual names.
And before you start worrying, these are defined in a way such that they only
expand in a place where you would use a function. So hitting SPC after (r
gives (require ', but hitting SPC after (delete-region r will not expand
the r, because that's obviously not a function. You can safely use short
variable names without running into conflicts.
You may I have noticed I've only mentioned functions. Variable abbrevs are
on the way!
Speedy keybinds
It also defines 4 commands, which fit tightly into this “follow the
thought-flow” way of writing.
M-RET
Break line and insert () with point in the middle.
C-RET
Do forward-up-list then do the above.
Hitting RET followed by a ( was one of the most common key sequences for me
while writing elisp, so giving it a quick-to-hit key is a significant
improvement.
C-c f
Find function under point. If it is not defined, create a
definition for it above the current function and leave point
inside.
C-c v
Same, but for variable.
With these commands, you just write your code as you think of it. Once you hit a
“stop-point” of sorts in your thought-flow, you hit C-c v on any undefined
variables. Then you just write its definition, and hit C-u C-SPC
(pop-global-mark) to go back to where you were.
Conclusion
sotlisp is floating around the Elpas, so you can issue the usual M-xpackage-installRETsotlisp, and activate it in your init file.
(speed-of-thought-mode)
Then you'll be able to do magic such as writing
(with-temp-buffer(inserttext)(buffer-string))
by typing (wtbM-RETiSPCtextC-RETbsSPC.
Also coming up is a sotlatex package and perhaps even a sotclojure.
This post advertises the release of hydra.el 0.5.0.
The package was introduced in an earlier post and received some positive feedback.
Currently, it's available in both
GNU ELPA (slightly behind)
and MELPA (always current).
The newfound utility
As I originally released the Hydra, I was asked if it was similar to
this approach
described on Endless Parentheses. I
said that it's not, since Hydra commands are meant to be repeatable
instead of being called just once.
Now, after I've designated the repeatable heads with red color and the one-off heads with blue color,
Hydra can reproduce the mentioned approach. Here's how it looks like:
And this is what I see in the echo area after pressing C-c C-v:
At this point:
q will cancel without doing anything. I think it's more
convenient than doing the equivalent C-g with the
global-set-key approach.
a, d, f, t, and a will call the appropriate function.
The advantage of blue Hydra heads over global-set-key is that you immediately get the hint: this can be useful
for rarely used commands.
as with the old red Hydra heads, pressing anything other than
a, d, f, t, a,
or q will vanquish the Hydra and call the key binding
that you just pressed. This is different from the global-set-key
approach, which would e.g. error to your C-c C-v C-n with
"C-c C-v C-n is undefined", unlike the Hydra, which would call C-n
(next-line).
The full defhydra syntax
Note that defhydra looks intentionally like a defun, so that it's
easier to remember how it works.
Here's a more verbose Hydra that's largely equivalent to the one above:
The final function calls the Hydra's body, displaying the hint in the echo area and setting
the transient map. It's the return result of defhydra, that's why it was possible to pass it
to global-set-key in the first example.
Second argument: Hydra body
The Hydra body consists of:
a map used for binding, like global-map or c++-mode-map or projectile-mode-map
the body prefix: it's a string passable to kbd that will be used
in conjunction with heads' prefixes to bind the functions
an optional plist, which currently recognizes only the :color key,
which in turn can be either red (the old behavior) or blue.
It's possible to omit the map and the body prefix simultaneously (it
doesn't make sense to omit one but not the other), or just pass ()
if you want to get a red Hydra body that you can bind yourself.
Third argument: Hydra hint
This string will be used in the echo area to distinguish the current
Hydra. This is optional, it case you don't provide it, it will default
to "hydra".
Fourth argument: Hydra heads
Each Hydra head is a list of:
the key extension
the function
optional hint
optional plist.
Again, the plist recognizes only :color currently. The color is inherited from the body, if
you don't specify it. In turn, the body color is red if you don't specify it.
The code above:
binds C-c a, C-c d etc with the usual global-set-key approach.
binds C-c C-v a, C-c C-v d with the new
approach, which shows you the hint right after C-c C-v and
allows you to cancel easier.
binds C-c v to be repeatable, i.e. you can press C-c v v v. Same with C-c C-v v v v.
Some more ideas for blue Hydras
Here's one for some helm-related functions, don't ask how
projectile-find-file ended up here:
In Emacs, of course. As I mentioned before, this blog is run using
Jekyll and a fork of the
lanyon theme.
It started out pretty convenient: thanks to jekyll serve I would see
the live updates at http://localhost:4000/ as I was editing the
markdown. But after a few posts, I've experienced a drastic decrease
in refresh time: it went from under a second to 15 seconds. Below, I'll show how I've managed
to speed it back up.
No posts - no problem
Thanks to (require 'dired-x) I can jump from the current post to the
posts directory with C-x C-j. It even puts the point on the current file. Now:
m to mark to current file
t to invert the mark, making the current file unmarked and all others marked
D to delete all marked files. No need to worry, since
they are all version-controlled (except the original one, which isn't
being deleted), dired even asks you for confirmation, so I hit
y.
Now, that there is only one post in the whole blog, jekyll serve is
much faster at refreshing.
Even faster jekyll serve
Which brings me to the next point. Since I'm calling it so much, might
as well wrap it in some Elisp. Elisp is like frosting - it makes everything better:
Start the commit with C (magit-commit-add-log). I'm using my own modification of magit-commit-add-log
that arranges the whitespace in a nice way, when aimed at a file instead of a hunk.
I get _posts/2015-02-01-blogging-about-blogging.md: string auto-generated by C.
I just amend it to look like _posts/2015-02-01-blogging-about-blogging.md: add and finalize the commit with
C-c C-c (git-commit-commit).
Now I have around 40 files marked as deleted, but not staged. I create a stash with zz and name it foo
or something. Then delete the stash with magit-discard-item, which I like to bind to d to be similar
to dired. Now it's as if these files were never deleted.
Whoa, now that I look at it, it's a lot of steps. But somehow dired
and magit are very similar to
Super Mario Bros: it
takes long to explain what you're doing, but as you play it, it's very
simple and natural.
I did try to make Elisp abbrevs work, but after a few times of getting
is expanded to indent-sexp in strings or comments, I just could
take it no longer.
Luckily and timely, Artur Malabarba revealed his
speed-of-thought-lisp. This
package does many things, but the main idea that I liked was that the
abbrevs should only expand when in the function position, i.e. the
abbrev is:
right after opening parenthesis
not in a comment
not in a string
not in the arguments list
With that idea, I've "refactored" my old abbrevs into
Abel. It's a minor mode add-on for
abbrev-mode. When you activate it, around a hundred abbrevs are
added to your abbrev list for emacs-lisp-mode. When you deactivate
it, these abbrevs are disabled, and your original abbrev list is
restored. It even overrides the mode-line description to Abbrev ->
Abel while it's active, thanks to
diminish.
So while you should try sotl, which provides many other things, keep
abel in mind: it's a simple upgrade to abbrev-mode that binds no
bindings and asks no questions.
Here's the current list:
(defcustomabel-abbrevs'(;; basics("a""and")("bp""boundp")("c""concat")("fp""fboundp")("ii""interactive")("i""insert")("l""lambda")("m""message")("n""not")("f""format")("u""unless")("up""unwind-protect")("w""when")("wl""while")("r""require")("ci""call-interactively")("cc""condition-case")("pg""plist-get")("sa""save-excursion")("sr""save-restriction")("smd""save-match-data");; defines("de""declare-function")("df""defface")("da""defmacro")("du""defcustom")("dv""defvar");; everything with char("bc""backward-char")("scb""skip-chars-backward")("scf""skip-chars-forward")("gc""goto-char")("fc""forward-char")("dc""delete-char")("ca""char-after");; everything with region("ra""region-active-p")("rb""region-beginning")("re""region-end")("ntr""narrow-to-region")("dr""delete-region")("ir""indent-region");; error related("ie""ignore-errors")("e""error");; regex match related("la""looking-at")("lb""looking-back")("mb""match-beginning")("me""match-end")("ms""match-string")("msn""match-string-no-properties")("rm""replace-match")("ro""regexp-opt")("rq""regexp-quote")("rr""replace-regexp-in-string")("rsb""re-search-backward")("rsf""re-search-forward")("sf""search-forward")("sm""string-match");; words("fw""forward-word")("bw""backward-word");; lines("eol""end-of-line")("fl""forward-line")("lbp""line-beginning-position")("lep""line-end-position")("nai""newline-and-indent");; buffer("bfn""buffer-file-name")("bn""buffer-name")("bs""buffer-substring")("bsn""buffer-substring-no-properties")("cb""current-buffer")("wcb""with-current-buffer")("wtb""with-temp-buffer")("efn""expand-file-name")("ff""find-file")("ffn""find-file-noselect");; window("ow""other-window")("sw""selected-window");; string("ssn""substring-no-properties")("ss""substring")("si""split-string")("se""string=")("sl""string<")("sp""stringp");; point("pi""point-min")("pa""point-max")("p""point");; key("gk""global-set-key")("dk""define-key");; rest("ah""add-hook")("atl""add-to-list")("bod""beginning-of-defun")("bol""beginning-of-line")("dm""deactivate-mark")("fs""forward-sexp")("jos""just-one-space")("kn""kill-new")("lp""load-path")("mm""major-mode")("sic""self-insert-command")("sn""symbol-name")("tap""thing-at-point")("tc""this-command")("ul""up-list"))"List of (ABBREV EXPANSION) used by `abel'.":set(lambda(symbolvalue)"Update abbrevs accoring to `abel-abbrevs'."(setsymbolvalue)(mapc#'abel-definevalue)):group'abel)
I wonder, when the code isn't touched in a long time, is it good (no
need for changes) or bad (became obsolete)? Let's find out. I'll
explain here
auto-yasnippet, my
second package in MELPA out of more than a dozen currently that saw
almost no changes since the initial commit two years ago, and see if
you like it.
Short description of yasnippet
YASnippet is a template
system for Emacs. It allows you to type an abbreviation and
automatically expand it into function templates. Bundled language
templates include: C, C++, C#, Perl, Python, Ruby, SQL, LaTeX, HTML,
CSS and more.
Snippet step-by-step
Here's one of the snippets that I use for emacs-lisp-mode:
# -*- mode: snippet -*-
# name: function
# key: d
# --
(defun $1 ($2)
$0)
The name of the snippet, function is more like a comment than
anything else.
On the other hand, key is very important: it's what I have to
insert in the buffer to get the expansion with M-xyas-expand.
Everything after # -- is the snippet body.
This particular snippet has two fields, in places of $1 and $2.
$0 is where the point will be when the snippet expansion is finished
As I expand, pressing TAB will move from field to field
until the expansion is finished.
Snippets are mode-local
Here's the corresponding snippet for clojure-mode:
As you see, the key here is the same; you're allowed to overload
them based on the current major-mode.
Even after quite a few posts, I still keep forgetting
Jekyll's syntax for the post header. This is my reminder:
# -*- mode: snippet -*-
# name: post
# key: post
# --
---
layout: post
title: $0
---
Mirrors in snippets
This simple snippet introduces a powerful concept, and an important
yasnippet feature that auto-yasnippet uses:
# -*- mode: snippet -*-
#name : class ... { ... }
# --
class $1$2
{
public:
$1($0)
};
This is a snippet for a class declaration in c++-mode; $1, the
name of the class, is mirrored in the name of the constructor. This
way, you don't have to enter it twice.
What auto-yasnippet does
All the snippets listed above are pre-configured, persistent and very
rarely changed. They are like plain functions in the source
code. Each of them needs their own file and so on.
What auto-yasnippet provides are throw-away lambdas, that don't
need a file and aren't persistent.
Basic install of auto-yasnippet
To get a usable install, you just need to bind aya-create, which is
similar in spirit to M-w (kill-ring-save):
(global-set-key(kbd"H-w")'aya-create)
and aya-expand, which is similar to C-y (yank):
(global-set-key(kbd"H-y")'aya-expand)
I also like to bind:
(global-set-key(kbd"C-o")'aya-open-line)
I'm using C-o to do all of these:
expand-abbrev
yas-expand and yas-next-field-or-maybe-expand
open-line
Example 1: JavaScript
Let's say that you have this code and want to generate more like it:
field1=document.getElementById("field1");
Let's even assume that you know how auto-yasnippet works and wrote down a slightly modified code beforehand:
field~1=document.getElementById("field~1");
Here, ~ are meant to represent yasnippet's mirrors, they will be
consistent across every expansion. Now you type H-w (aya-create), which works on the current
line when there's no region. Your code becomes the initial one without ~, and aya-current variable now holds:
Since the code spans multiple lines, as is often the case with Java, you need to mark it
with a region before H-w.
Here's the final result:
classLightOnimplementsRunnable{publicLightOn(){}publicvoidrun(){System.out.println("Turning on lights");light=true;}}classLightOffimplementsRunnable{publicLightOff(){}publicvoidrun(){System.out.println("Turning off lights");light=false;}}
No need for AbstractLightFactoryAdapterProvider when we can just
copy-paste stuff with auto-yasnippet. In fact, I should probably
emphasize that when describing auto-yasnippet: it's just an advanced
copy-paste tool.
Example 3: C++
Suppose that I want to generate curl from the grad. I can start with this template:
constPoint<3>curl(grad[~2][~1]-grad[~1][~2],
Here, I need less than a line, so region needs to be marked again. The result:
Note how annoying it would be to triple check that the indices match.
This time, I just had to check the first line.
Example 4: taking ~ out of the equation
This works only for one-line snippets with a single mirror parameter.
In the JavaScript example, you can leave a $ instead of each
occurrence of $1, and with the point in place of the last occurrence
you call H-w (aya-create). Here, | represents the point:
I hope that this package will lessen your suffering when dealing with
verbose programming languages and repetitive text, and that a day will
come when repetition is no longer needed and auto-yasnippet will
become obsolete.
I was inspired by Sacha Chua's recent post
explaining her window bindings, that combine both ace-window and windmove.
So I wrote down an update to Hydra in order to get a similar setup.
The new code should already be available in
MELPA. I'll update the code in
GNU ELPA soon, when I make
sure that there were no bugs introduced by the change.
If anyone wants to see how the defhydra macro expands, you can check
out
hydra-test.el.
I just added a Travis CI setup, so if you're interested in starting
to test your Elisp code, you can have a very simple example.
How the defined Hydra works
With this setup:
to swap two windows (i.e. call C-uace-window), I can do any of:
C-M-o s
C-M-o ua
yys
yyua
to delete one window (i.e. call C-u C-uace-window), any of:
C-M-o d
C-M-o uua
yyd
yyuua
to move one window down, two windows right, and one window up:
C-M-o jllk
yyjllk
Although every other shortcut except the Hydra heads will vanquish the
Hydra, sometimes I have nothing on my mind that needs doing. For that
case, as you can see above, I enter o in its own list
without a function, so that o will dismiss the Hydra
without doing anything.
I am a Statistician, coding in R and write report is what I do most of the day. I have been though a long way of searching the perfect editor for me, tried Rstudio, SublimeText, TextMate and settled down happily with ESS/Emacs, for both coding and writing.
There three features that have me made the decision:
Auto Formatting
Scientists has reputation of being bad programmers, who wrote code that is unreadable and therefore incomprehensible to others. I have intention to become top level programmer and followed a style guide strictly. It means I have to spent sometime in adding and removing space in the code.
To my surprise, Emacs will do it for me automatically, just by hitting the TAB and it also indents smartly, which make me conformable to write long function call and split it into multiple lines. Here's an example. Also, if I miss placed a ')' or ']' the formatting will become strange and it reminders me to check.
I frequently search the command history. Imaging I was produce a plot and I realised there was something miss in the data, so I go back and fix the data first, then run the ggplot command again, I press Up/Down bottom many times, or just search once/two times. M-x ggplot( will give me the most recent command I typed containing the keyword ggplot(, then I press RET to select the command, which might be ggplot(gg.df, aes(lon, lat, col = city)) + geom_line() + ...... If it is not I want, I press C-r again to choose the second most recent one and repeat until I find right one.
Literate Programming
I am a supporter of literate statistical analysis and believe we should put code, results and discoveries together in developing models. Rstudio provides an easy to use tool for this purpose, but it does not support different R sessions, so if I need to generate a report, I have to re-run all the code from beginning, which isn't particle for me with volumes data because it will take quit long.
ESS and org-mode works really well via Babel, which is more friendly to use. I can choose to run only part of the code and have the output being inserted automatically, no need to copy/paste. Also, I can choose where to execute the code, on my local machine or the remote server, or both at the same time.
These are only the surface of ESS and there are lot more useful features like spell checking for comments and documentation templates, that makes me productive and I would recommend anyone uses R to learn ESS/Emacs. The following is my current setting.
;; Adapted with one minor change from Felipe Salazar at
;; http://www.emacswiki.org/emacs/EmacsSpeaksStatistics
(require 'ess-site)
(setq ess-ask-for-ess-directory nil) ;; start R on default folder
(setq ess-local-process-name "R")
(setq ansi-color-for-comint-mode 'filter) ;;
(setq comint-scroll-to-bottom-on-input t)
(setq comint-scroll-to-bottom-on-output t)
(setq comint-move-point-for-output t)
(setq ess-eval-visibly-p 'nowait) ;; no waiting while ess evalating
(defun my-ess-start-R ()
(interactive)
(if (not (member "*R*" (mapcar (function buffer-name) (buffer-list))))
(progn
(delete-other-windows)
(setq w1 (selected-window))
(setq w1name (buffer-name))
(setq w2 (split-window w1 nil t))
(R)
(set-window-buffer w2 "*R*")
(set-window-buffer w1 w1name))))
(defun my-ess-eval ()
(interactive)
(my-ess-start-R)
(if (and transient-mark-mode mark-active)
(call-interactively 'ess-eval-region)
(call-interactively 'ess-eval-line-and-step)))
(add-hook 'ess-mode-hook
'(lambda()
(local-set-key [(shift return)] 'my-ess-eval)))
(add-hook 'inferior-ess-mode-hook
'(lambda()
(local-set-key [C-up] 'comint-previous-input)
(local-set-key [C-down] 'comint-next-input)))
(add-hook 'ess-mode-hook
(lambda ()
(flyspell-prog-mode)
(run-hooks 'prog-mode-hook)
;; (prog-mode)
))
;; REF: http://stackoverflow.com/questions/2901198/useful-keyboard-shortcuts-and-tips-for-ess-r
;; Control and up/down arrow keys to search history with matching what you've already typed:
(define-key comint-mode-map [C-up] 'comint-previous-matching-input-from-input)
(define-key comint-mode-map [C-down] 'comint-next-matching-input-from-input)
Ah, much better. The indentation does change a bit in other places as
a result of this. To my experience, it can be fixed on a case-by-case
basis by declaring the indent level for the offending function or
macro.
By default, functions behave as if their indent was declared to nil.
Zero here means that we expect zero arguments on the same line, so that this indentation follows:
Well, "refactoring" is for the Java people, I simply rename things.
Just today, I had to do a big rename operation related to the
release of lispy 0.22.0.
Below, I'll share some functions and packages that I used for that.
My Github setup
Here's it (tree -da -L 1) is:
.
├── .cask
├── gh-pages
├── .git
└── images
I've cloned my gh-pages branch into its own git repository inside
the original lispy repository. This way, I can rename functions in
the code and the documentation simultaneously.
After version "0.21.0" hits, any time you call lispy-left by its
now obsolete alias lispy-out-backward, you'll get a message:
`lispy-out-forward' is an obsolete command (as of 0.21.0); use `lispy-right' instead.
So now, since I'm releasing version "0.22.0", I can remove even the
alias declarations. I gave people one week of warnings to adjust (just
rename to the new name) any of their code that's calling the currently
obsolete functions.
Renaming obsolete functions in the documentation
Taking advantage of the repository setup, I can:
Step 1: call rgrep
rgrep is a fine function, I wonder why it's not bound by default; I
bind it to C-<, taking advantage of my weird key
mappings (I'm actually pressing the C-;-l physical keys).
It requires 3 inputs:
Symbol to search for: I call it with the point positioned on
the symbol that I want to rename in the code, in the middle of the
define-obsolete-function-alias tag, so rgrep picks up the
symbol name as the default and I just type RET to select
it.
File pattern: *.el, the current file extension is the
default. I type *RET, since I want to match the org and html
files as well.
Base directory: this directory will be will be recursively
searched for files that match the file pattern; the default
~/git/lispy/ is fine here, RET.
Step 2: call wgrep
wgrep is a fine
package, I wonder why it's not more popular. Since it's so similar to
wdired (one of the best things since sliced bread, btw), I like to bind
the starter to C-x C-q and the finisher to C-c C-c
as well:
iedit is an amazing package, it's crazy-good.
Here's how I bind it, since once iedit-mode is on, you can move to the next occurrence with C-i:
(global-set-key(kbd"C-M-i")'iedit-mode)
In order to change really every occurrence in the buffer, I need to
mark the thing that I want to change, before C-M-i.
Otherwise, iedit will automatically add symbol bounds (a nice
feature, actually), so that e.g. =lispy-out-forward= will not match
lispy-out-forward.
Finally, I interactively, char-by-char, rename e.g. lispy-out-forward to lispy-right.
The experience is similar to the popular multiple-cursors,
which I also like to use, just for different purposes.
Step 4: exiting
When I'm done:
I exit iedit-mode with C-M-i.
I exit wgrep with C-c C-c
I save all affected files if I want, since they aren't saved yet and
it's still possible to revert everything.
Outro
Woah, that's a lot of steps!
True, but do note that all three tools can be used on their own for
various other tasks. See for instance my other
"refactoring" demo, that uses
iedit-mode to unbind a let-bound variable in Elisp (should also
work for Common Lisp, since the syntax is the same).
There's beauty and utility in having such composable tools. A lot of
the time, it's better than to just have one "Rename" button. For
instance, when only one buffer is involved, the rgrep-wgrep step
can be skipped and I can rename stuff with iedit-mode only.
Or, when the playground for renaming is less than a buffer, I can:
C-x nd - narrow-to-defun or C-x nn -
narrow-to-region (both are equivalent to N in lispy)
Why we don't have a comment/uncomment-line function is beyond me. While we fix
that, might as well make it as complete as possible.
(defunendless/comment-line(n)"Comment or uncomment current line and leave point after it.
With positive prefix, apply to N lines including current one.
With negative prefix, apply to -N lines above."(interactive"p")(let((range(list(line-beginning-position)(goto-char(line-end-positionn)))))(comment-or-uncomment-region(apply#'minrange)(apply#'maxrange)))(forward-line1)(back-to-indentation))
Short and sweet, isn't it? In analogy to M-;. I find C-; to be a great
binding for it. This way I can quickly comment 3 lines by hitting C-3 C-;.
(global-set-key(kbd"C-;")#'endless/comment-line)
Also, because it moves forward after commenting, you can conveniently do things
like comment every other line by repeatedly doing C-; C-n.
Update <2015-02-01 Sun>
The previous version didn't quite work for negative prefix arguments, so I've
updated the above snippet to fix that.
Also, quite a few people requested (or proposed) a version which acts on region
if active. I already use M-; for that (and I also its other modes of
operation), but I can see why people would want that so I've added it below.
It's similar to the one suggested by Kaushal Modi.
(defunendless/comment-line-or-region(n)"Comment or uncomment current line and leave point after it.
With positive prefix, apply to N lines including current one.
With negative prefix, apply to -N lines above.
If region is active, apply to active region instead."(interactive"p")(if(use-region-p)(comment-or-uncomment-region(region-beginning)(region-end))(let((range(list(line-beginning-position)(goto-char(line-end-positionn)))))(comment-or-uncomment-region(apply#'minrange)(apply#'maxrange)))(forward-line1)(back-to-indentation)))
I'd like to share a little snippet that I like to use instead of a
plain M-s o (occur).
smarter-than-the-average-occur
(defunoccur-dwim()"Call `occur' with a sane default."(interactive)(push(if(region-active-p)(buffer-substring-no-properties(region-beginning)(region-end))(let((sym(thing-at-point'symbol)))(when(stringpsym)(regexp-quotesym))))regexp-history)(call-interactively'occur))
It will offer as the default candidate:
the current region, if it's active
the current symbol, otherwise
It's pretty good, since I actually want the default candidate in 95%
of the cases. Some other functions, such as rgrep are smart like
this out-of-the-box.
A Hydra that goes nicely with occur
Since occur, like some other functions (grep, rgrep, compile),
works with next-error, it's possible to use this
Hydra to navigate the occur
matches:
Hydra was introduced in an
earlier post,
and is available in GNU ELPA.
What goes around comes around
I usually do a quick internet search to see where I got the code that
I'm posting here, reason being that it's nice to list the source, as
well as there could be more advanced stuff out there on the same
topic.
I saw a few results for "occur-dwim" so I thought that I just
copy-pasted it from somewhere. Slight disappointment turned into a
small sense of pride, when I found that the source of the copy-pastes
was my old Stack Overflow answer:)
Currently, MELPA hosts over 2250 packages. That's
quite a large number, although it's still feasible to try most of the
popular ones. In comparison,
apt-cache pkgnames | wc
gives me the number 50250. I don't recall calling apt-cache
pkgnames ever before, while I call package-list-packages all the
time. I suppose that this will have to change when the number of
packages grows further.
I usually explore packages just as they show up as new in package-list-packages.
I'll share the two tools that I use for exploring them.
Smex
Smex is an M-x enhancement for Emacs.
In addition to executing commands faster, it gives you an option to jump to the current command's definition
with M-..
I don't feel bad at all for unbinding transpose-chars,
C-t is such a prime binding, that only the best commands
can deserve it. And since I touch-type, it's easier for me to hit
M-DEL and retype the word when I notice a mistake, than to
carefully navigate to the mistake location and call transpose-chars
with surgical precision. On the other hand, the number of Emacs
packages is only going to grow, so it becomes less and less feasible
to bind everything. Instead, seldom used stuff can be called though
smex. For instance, I don't bind
markdown-toc or
speed-type and call them via
C-t instead.
Back to the point, C-t ... C-.
(smex-find-function) allows me to quickly jump to the source of the
package that I want to explore.
Semantic tags
Once I'm in the source code, I can quickly get an overview with
g (lispy-goto) from
lispy. It uses CEDET's semantic
module to get a list of tags in current directory with
helm for completion. In my
opinion, it's much prettier than helm's own helm-semantic. It also
explores more tags for Emacs Lisp, for instance it will capture
use-package and global-set-key tags.
A new cute feature that I added recently is a different background for
user-visible functions, i.e. the ones that are callable by
M-x or smex. Here's how it looks like, for the
speed-type package (interactive functions have a purple background):
If the image text looks too small, you can right click / view image in
your browser. Here, I can see at a glance, that there are three
commands that I can call. I can also see a few variables, two of them
don't include --, so they're probably customizable.
I saw this question today at Stack Overflow -
Emacs: How to enable toolbar mode and menubar mode only under a certain mode?.
Myself, I never use the menubar or the toolbar. But maybe it's just
because it's useless in the modes to which I'm used to. What if there
was an option to have the menu bar on just for a few select
modes? For instance, if I decided that I want to learn Haskell
tomorrow, I wouldn't mind to have the menu bar on for a while,
especially if there was some useful stuff there in the menu.
The Code
I'm not aware of a convenient hook for this job; I tried
after-change-major-mode-hook, and it doesn't work when switching
windows. The universal post-command-hook would be totally lame for this task.
Then it occurred to me, that select-windowhas to be called
eventually in most circumstances. Even
ace-window calls
select-window.
It's necessary at least for magit. Doing, for instance,
ll operation switches windows many times.
So I added a check that the current buffer isn't selected twice.
In Emacs, a ring is like a stack with a limited length: you push onto
the head, and if the stack becomes too large, the stuff gets removed
from the tail. Perfect for debugging this code, since I don't care
what happened 20 window switches ago, I'm only interested in the most
recent state. One small annoyance is that the following code will
throw, if the ring is empty:
(ring-refmenubar-last0)
I would prefer if it just returned nil instead. Hence the trick of
initializing menubar-last with a dummy item. Note that this
annoyance propagates to other areas: (current-kill 0) will throw if
you haven't yet copied text since Emacs started.
Outro
I'm not sure if this code is more annoying than useful, but, at least,
I learned something new while writing it, and I hope you did too. If
you know of a better way to accomplish the original task, please do
share.
Seriously though, it's my little project of writing a major mode for
Emacs. I've written many minor modes, but this is my first major
one. Granted, it's only a small derivation of emacs-lisp-mode.
How the ElTeX looks like
Here's how a sample document outline looks like:
Here, homogenization, smoluchowski-equation, and
model-description-geometry are simply Elisp functions that should
produce a string on call. They are defined here.
On calling M-xeltex-compile, the corresponding LaTeX
document will be written to the file to which eltex-filename points
to.
How the original ElTeX looks like in emacs-lisp-mode
(setqeltex-filename"~/tpaper/tpaper.tex")(require'eltex-macros);;* Document(documentclass"article"(usepackage'("amsmath""fleqn")"amsthm""enumerate""amsfonts""xcolor"'("biblatex""style=numeric,natbib=true,backend=bibtex"))(bibliography"analysis")(document(section"Introduction"(homogenization)(smoluchowski-equation))(section"Notations and Assumptions"(model-description-geometry))"\\printbibliography"))
Why this?
Well, why not? At one point I was frustrated with LaTeX not allowing
me to define mathematically rich entities. I did hack up a few TeX
macros for this eventually, but it was very awkward.
So I thought that I could make Emacs Lisp generate simple LaTeX in a
similar way that C generates machine code. No more intricate LaTeX
macros, only plain LaTeX, generated from (intricate) Elisp.
This is still very much in a toy stage, I don't currently use it for
anything serious. Still, in case you're interested in defining
derived major modes, you can look at my
implementation,
it's only 25 lines. The interesting thing about it, if you noticed, is
that it replaces all double quoted strings visually with single quoted
strings, so that they don't jump out as much.
More examples
Here's how the typesetting of some mathematical logic looks like:
And here's the corresponding LaTeX:
Note how simple the resulting LaTeX is, considering how many variables
were used to generate it. Now, I can update anything in the with
Elisp binding, and the corresponding LaTeX will be appropriately
regenerated, error-free.
where args are determined by the presence of argument literals
taking the form %, %n or %&. % is a synonym for %1, %n designates
the nth arg (1-based), and %& designates a rest arg. This is not a
replacement for fn - idiomatic used would be for very short one-off
mapping/filter fns and the like.
I hope that this gets accepted, although there are some conservative
people that protest against this change. Let me know if you would
like to have this option for your Elisp setups. And remember that you
can try out the patch if you're familiar with
building Emacs from source.
I use this one quite frequently: i is somehow mnemonic to
C-i, which means completion. Remember that you can select
the current candidate with C-m and the current text
(usually to create a new file) with C-j.
Huh, I always thought this was the default binding. I have no idea
where this came from, but I use this plus x to get rid of
the garbage produced by a LaTeX run:
This little function is essential if you want to do perform some
operation on all files in the current directory and its sub
directories that match a pattern. Basically the same thing that you
would do with the UNIX find, just better.
There probably is a default function that does this. I've been using
this one for years, probably because it reuses the current dired
buffer of opening the parent directory.
Outro
Phew, that's a lot of bindings. I don't necessarily encourage you to
use the same bindings as me; I just want to bring your attention to
some of these functions, so that you can work them into your workflow.
If you notice that I'm using some obsolete stuff, do let me know, I'm
always looking to improve.
When developing a package, package-install-from-buffer is a very useful
command. It installs the current buffer as an Elpa package, so you can test
installation, byte-compilation, autoloading, and activation, all in one fell
swoop. If your package has multiple files, however, it gets a little more
complicated.
Earlier this month, I asked on Emacs.SE whether there was a simple way to
manually install a multifile package. As discussed in Phils' answer, you need to
create a -pkg.el file, tar the package, and then invoke a command on the tar
file. That's two steps too many for me, so a push to master was in order.
** package-install-from-buffer and package-install-file work on directories.
This follows the same rules as installing from a .tar file, except the
-pkg.el file is optional.
There are no new commands to remember. Just issue package-install-from-buffer
from a dired buffer, or invoke package-install-file and give a directory.
Whichever package is contained in that directory will be read and installed, be
it single or multifile, no taring or -pkg.el file necessary.
I managed to spike a lot of interest for sticky key bindings in my earlier post,
Zoom in/out with style.
So now, I've refactored this method into a convenient library
hydra.el.
The Concept
This package can be used to tie related functions into a family of
short bindings with a common prefix - a Hydra.
Once you summon the Hydra through the prefixed binding (the body + any
one head), all heads can be called in succession with only a short
extension.
The Hydra is vanquished once Hercules, any binding that isn't the
Hydra's head, arrives. Note that Hercules, besides vanquishing the
Hydra, will still serve his orignal purpose, calling his proper
command. This makes the Hydra very seamless, it's like a minor mode
that disables itself auto-magically.
An Example
This code will accomplish the task of the previous post:
Now, <f2> is the Hydra's body: you need to press it
only once, together with one of the heads (g or
l), to summon the Hydra.
Afterwards, you can call the heads in succession without the body
prefix, i.e. <f2> g g g l will work. To vanquish the
Hydra, just call up Hercules: any key binding that's not
g or l, e.g. C-f or whatever you
wanted to do.
Note that you can still assign an unrelated binding to
e.g. <f2> f: the Hydra does not take over
<f2>, only over <f2> l and
<f2> g.
The Infrastructure
hydra-create will create new interactive functions for you with the proper docstrings:
hydra-<f2>-text-scale-increase is an interactive Lisp function.
It is bound to <f2> g.
(hydra-<f2>-text-scale-increase)
Create a hydra with a "<f2>" body and the heads:
"g": text-scale-increase,
"l": text-scale-decrease.
Call the head: text-scale-increase.
An exciting new Hydra: move window splitter
Zooming is old news, Hydra bundles a new application:
This will allow you to move the window splitter, after you issue
C-x 2 or C-x 3 one or more times, with
e.g. C-M-oh h j k j l k l h. You can, of
course, customize both the body and the heads of this Hydra to your
preferences.
The docstrings for this Hydra look more impressive, too:
hydra-C-M-o-move-splitter-up is an interactive Lisp function.
This year I made it my resolution to learn clojure. After reading through the
unexpectedly engaging romance that is Clojure for the Brave and True, it was
time to boldly venture through the dungeons of 4clojure. Sword in hand, I
install 4clojure.el and start hacking, but I felt the interface could use some
improvements.
For starters, it's annoying that you need two commands to check the answer and
move to next question. Sacha has a nice suggestion on this matter, a single
command which checks the answer and moves to the next question. Nonetheless, I
needed more.
Having to manually erase the __ fields to type my answer is absurd. Not to
mention some answers are cumbersome to type inline. With the following command,
you type your code at the end of the buffer; a lone archer lining up a shot to
slay the problems above.
(defunendless/4clojure-check-and-proceed()"Check the answer and show the next question if it worked."(interactive)(unless(save-excursion;; Find last sexp (the answer).(goto-char(point-max))(forward-sexp-1);; Check the answer.(cl-letf((answer(buffer-substring(point)(point-max)));; Preserve buffer contents, in case you failed.((buffer-string)))(goto-char(point-min))(while(search-forward"__"nilt)(replace-matchanswer))(string-match"failed."(4clojure-check-answers))))(4clojure-next-question)))
The second encounter is simple in comparison. Just sharpen the blade, polish the
shield, and we're ready for battle.
(defadvice4clojure/start-new-problem(afterendless/4clojure/start-new-problem-advice()activate);; Prettify the 4clojure buffer.(goto-char(point-min))(forward-line2)(forward-char3)(fill-paragraph);; Position point for the answer(goto-char(point-max))(insert"\n\n\n")(forward-char-1);; Define our key.(local-set-key(kbd"M-j")#'endless/4clojure-check-and-proceed))
These two snippets have me cleaving effortlessly through the initial questions
and I'm eager for the final challenges.
Both of these websites were recommended to me by Michael Fogleman. Do you know
any other good clojure resources?
The last release was more than a month ago, and there have been more
than 130 commits to master since then. Somehow, I've been dragging my
feet with this release: the (3 pages of)
release notes
were in the draft stage for 10 days now, while I kept committing on
top of them.
Introduction
This project is my vision of efficient LISP editing. According to
Github, I started it more than a year ago, although the initial commit
already contained around 1000 lines of code. After a year, it's more
than 100 interactive commands in 5000 lines of code and 1000 lines of
tests.
Initially, I started the project because, while I wanted to learn
Paredit to get more efficient, I did not want to learn Paredit's
cumbersome bindings. Having established a skeleton that allows to call
Paredit-like commands with plain letters, over time, I've tacked on
anything LISP-related here. In this way, it's very similar to
org-mode, which starts with an outline and TODO skeleton, and then
adds everything else in the world on top of that. Heck, I actually
tacked on org-mode's outline features on top of lispy: when at an
outline, i is equivalent to org-mode's TAB, and
I is equivalent to S-tab.
Among the other packages with which lispy
integrates/cooperates/coexists are: edebug, ediff, eldoc, ert,
outline, semantic, semantic/db, ace-jump-mode, iedit,
delsel, helm, multiple-cursors, fancy-narrow, projectile,
god-mode, auto-complete, company.
Main idea behind lispy
The idea is to have plain letters, like h or r,
call commands instead of self-inserting, but only if the point
position is such that you wouldn't want to self-insert anyway. When
this situation occurs, I like to say "the point is special"; this
shortcut is all over the code and the docs.
This is a bit similar to vi's normal/insert states, but instead of
"holding" the current state in your head, it's visible through just
the point position. And instead of having just the
Esc/i combination to toggle normal/insert state,
you can do it with any command that moves point,
e.g. C-f, or C-a, or even any custom command
that you write. lispy does provide [, ], and
C-3 key bindings for getting into special, but you are free
to use any other binding or command that you want.
For instance, starting from this code and point position (which is already special):
(when(=arg0)(setqarg2000))
you can move to the when statement with h (lispy-right):
(when(=arg0)(setqarg2000))
or you can remove the when statement altogether with r (lispy-raise):
(setqarg2000)
or you can:
evaluate (setq arg 2000) statement with e (lispy-eval); the result will be displayed in the echo area; works for multiple LISP dialects
evaluate and insert with E (lispy-eval-and-insert); the actual value 2000 will be inserted below the expression
evaluate and replace with xr (lispy-eval-and-replace); the expression will be replaced with 2000
copy the statement to the kill ring with n (lispy-new-copy)
delete the statement with C-d (lispy-delete)
insert a copy of the statement below with c (lispy-clone)
insert 3 copies of the statement below with 3c (digit-argument, lispy-clone)
move the statement on the previous line with DEL (lispy-delete-backward)
mark the statement with m (lispy-mark-list)
mark only arg with 2m (digit-argument, lispy-mark-list)
get help for setq with xh (lispy-describe)
get help for arg with 2mxh (digit-argument, lispy-mark-list, lispy-describe)
copy 2000 to kill ring with 3mn (digit-argument, lispy-mark-list, lispy-new-copy)
move the statement outside of when with oh (lispy-other, lispy-left)
swap (= arg 0) with (setq arg 2000) with w (lispy-move-up)
move it back with s (lispy-move-down)
put the whole when expression on one line with hO (lispy-left, lispy-oneline)
narrow the buffer to current sexp with N (lispy-narrow)
widen the buffer with W (lispy-widen)
a lot of other things, there are 52 plain letters, after all
Note that in the following code, the point is not special:
(when(=arg0)(setqarg2000))
so if you would type h, it would not call lispy-left, but would insert "h" instead, yielding whhen.
to get the point into special before (when, you can use either C-a or [ or C-M-a
to get the point into special after arg 0), you can use either C-e or ]
to get the point into special after 2000)), you can use C-3
Evolution of special
Initially, the special state was only for the point before an open
paren or after a close paren, since you almost never-ever want to
insert characters at those positions. Over time, other point states
were added to special:
region is active (supersedes
expand-region for LISP
dialects)
the point is at the start of a comment
the point is at the start of an outline
Region selection is especially important, since it is super-useful for
manipulating (move, copy, eval, get-help, goto-definition) symbols
inside lists. Since these symbols aren't delimited with parens, the
only way to get to them with special is though region selection.
Happily, region selection will fix the largest source of Paredit
unbalanced paren errors: while using lispy region-manipulating
commands, you can't copy an unbalanced expression, and thus you can't
yank an unbalanced expression. You just have to use m and
M-m instead of C-SPC; and h,
j, k, l, i,
>, and < instead of e.g. C-f and
M-f.
Since there are only 26 lower-case letters and 26 upper-case letters,
the state of the command bindings in lispy quickly turned into
survival of the fittest:
the commands that were used the most got the priority bindings of
lower case letters on the home row
the second tier got the other lower-case bindings
the third tier of commands were put on upper-case letters and on
x + lower-case letters
the fourth tier of commands are not bound at all, and I'm
considering to obsolete some of them, just to keep things simpler.
ADD: the Annoyance-Driven Development
The other part of evolving and refining the commands, consisted of
noticing small annoyances for when some generic command wasn't working
as intended, or it was working in a sub-optimal way in a certain
situation. After this, I would fix the command and put a test on it,
to make sure that the annoyance does not surface in the future.
Notes on LISP dialects
My priority is Elisp, since that's what I'm using to implement
lispy, but the following dialects are also supported:
To be supported, all a LISP dialect needs is just to use ( or { or
[ as the opening delimiter; and ) or } or ] as the closing
delimiter, no actual adaptations in the lispy code are necessary.
The only thing that needs to be implemented on a per-dialect basis is
the language-specific eval:
e (lispy-eval)
E (lispy-eval-and-insert)
xr (lispy-eval-and-replace)
xj (lispy-debug-step-in)
Also, though the jump-to-definition functionality could be implemented
via CEDET, environments like SLIME can do it much better, so
F (lispy-follow) and M-. (lispy-goto-symbol) use the
appropriate environment's facilities. Somehow, I still haven't managed
to implement this for Geiser.
Drinking from the lispy fire hose
I'll get you started with the most basic and composable commands
below. You can find the rest in the
function reference or by just calling
xv (lispy-view-test) on should statements of
lispy-test.el:
In the screenshot above, I start with the code and point position on
the top. Then, after typing miji, I should end up with the
state below: the point and mark position have moved. In this
particular situation, I could follow-up with e to see the
value of auto-mode-alist. Here's how to decipher miji:
m - lispy-mark-list: marks current expression
i - lispy-tab (mnemonic for indent or inner): marks the car of current expression
j - lispy-down (vi shortcut to move down): moves the point and mark down by one sexp, selecting the quoted expression
i - lispy-tab: selects the car of the quoted expression, i.e. auto-mode-alist
As you see from the screenshot, I have show-paren-mode always
on. It's even on in the tests visualization!
The most basic lispy commands: the arrows
h is left
j is down
k is up
l is right
The directions are literal only if you have your code properly
indented, with newlines after each sexp. Otherwise, it may be the case
that j moves literally right, instead of down; still, it's
down figuratively.
arrows like digits
All of them take digit arguments, so that e.g. 5j is equivalent to jjjjj.
h and j maintain the guarantee of not exiting the
current list, so you can use e.g. 99j to move to the last
element, if your list length is smaller than 99.
arrows like regions
When the region is active, the arrows move the mark appropriately with
the point. You can activate and deactivate the region by repeatedly
pressing m.
You can also mark a symbol with M-m
(lispy-mark-symbol). There's no need to be in special for this
command to work. I call these type of bindings global, while the
bindings that only work in special I call local.
arrows like outlines
When located at the outline, j will call
outline-next-visible-heading, and k will call
outline-previous-visible-heading. l will move to the
first list of the outline, while h will jump between the
top-level sexp and the containing outline.
switching to a different side of the expression
Arrows can't do this easily; this can instead be done with
d (lispy-different). Works for lists and regions.
Moving the code instead of moving around the code
The most basic commands are:
w is lispy-move-up
s is lispy-move-down
These will "hold on" to the current expression while moving it in the
appropriate direction. There's no need to worry to mess up with them,
since they cancel each other out perfectly.
Note that, just like with the arrows, if you don't have an opening or
closing delimiter to "grab", you can mark a symbol M-m to
be in special and use w / s.
Modified arrows can move too
o will modify the arrow keys temporarily, just for one
command, with a minor mode. You can think of it as making the arrows
move the point and the sexp in the usual direction, instead of
moving just the point.
ol: move current sexp outside of the parent list, forwards
oh: move current sexp outside of the parent list, backwards
oj: move current sexp inside the next list, making it the first element
ok: move current sexp inside the preceding list, making it the last element
Extending or shrinking the current list or region
> (lispy-slurp) grows the current list or region in the current direction by one sexp
< (lispy-barf) shrinks the current list or region in the current direction by one sexp
Similarly to j and k, these commands maintain
the guarantee of not exiting the parent list, so you can slurp until
the end of the list with e.g. 99>.
Outro
This project is very far from being final, I'm expecting to reach
0.99.0 before getting to 1.0.0. The reason is that the package
aims to build intuition to the point of automation. For each small
step in that direction, every small bug is two steps back, since it
breaks the process of building intuition. So every possible situation
needs to be tested and bugs fixed until the package is finally ironed
out.
While I do appreciate the
stars, actually trying
to do things with lispy and raising
issues would help me a
great deal more. For instance, if you raise an issue like
How can I generate a function call right after the function definition?
I would say to just use 2mcol(:
mark the function name with 2m
clone region with c
move region outside the function body with ol
wrap the region with parens while deactivating it with (
And this would be a sort of FAQ question / recipe already done there.
Or you could raise an issue like:
Why doesn't F work for Racket?
I would say it's because I haven't implemented it yet, since it was
tricky. But I'll get to it, now that I see that there's some interest
in lispy from Racket users.
org-download allows you to drag-and-drop an image from e.g. Firefox
or your own file system to an org-mode buffer. When I originally
wrote it, I was taking a Chemistry course on
edX, which included homework and exams with a
lot of images. Doing the homework in a literate style with org-mode,
I wanted the problems to be completely self-contained, and for that I
needed to save all the images from the website, preferably quickly, to
an org-mode buffer. With a few hacks and some advice and
contributions from more experienced hackers, org-download came to
be.
Today's commits
The main contribution from today's commits was adding the code that
un-aliases images that point to HTML. I see a lot of these from the
people that I'm following on my Twitter,
@_abo_abo. Here's the code:
(defcustomorg-download-img-regex-list'("<img +src=\"""<img +\\(class=\"[^\"]+\"\\)? *src=\"")"This regex is used to unalias links that look like images.The html to which the links points will be searched for theseregexes, one by one, until one succeeds. The found image addresswill be used.":group'org-download)
and in (org-download-image link):
(unless(image-type-from-file-namelink)(with-current-buffer(url-retrieve-synchronouslylinkt)(let((regexesorg-download-img-regex-list)lnk)(while(and(notlnk)regexes)(goto-char(point-min))(when(re-search-forward(popregexes)nilt)(backward-char)(setqlnk(read(current-buffer)))))(iflnk(setqlinklnk)(error"link %s does not point to an image"link)))))
Here, image-type-from-file-name is a built-in from image.el that
decides if link looks like an image file or not.
As you can see, since I intend to use link right after to be
transformed right now, I'm using url-retrieve-synchronously, instead
of the asynchronous url-retrieve. The other one is used by default
to download the actual image.
Note the use of the while / pop combination for traversing a list.
I've seen it used in some places in the Emacs core, and I think it's
pretty neat.
Arranging the regular expressions into a list is necessary to give
priority to some regexes. For instance, the first element of
org-download-img-regex-list should match e.g. an actual referred
image on Twitter, while the second element will match at least a
profile picture in the case when there is no referred image.
Visual demo
I think that I might have over-engineered the custom options of
org-download a bit. Just to keep you motivated enough to figure them
out, here's a link to
a Youtube demo of the
fast clickety-clicking (the mouse usage), that comes after some
clackety-clacking (the customization).
Once you make Ediff bearable, it becomes wonderful. At least this was
my experience. It's a very good tool, but some of the defaults look
very poor to me.
Customizing the customize
First of all, I'll list a macro that I've started to use for setting
custom variables:
This macro I've put together myself after searching though the code
base and not finding something similar; custom-set-variables comes
close to what I want, or maybe custom-initialize-changed. Basically
all I want is a setq that is aware of the custom-set property of a
variable. If you know such a macro, please let me know.
Changing some Ediff options
Now, that I've explained the custom setter (by the way, using
custom-set-variables is absolutely equivalent to csetq), here are
my changes:
Don't use the weird setup with the control panel in a separate
frame. I can manage windows in Emacs much better than my desktop
(Unity or Gnome Shell) can manage the Emacs frames.
Ignore white space. I don't write a lot of Python, so I don't care
about the white space in the diffs. At the same time, I re-format any
LISP code that I edit to my liking, and I want to see only the
important changes in the diff, and not the whitespace nonsense.
(csetq ediff-diff-options "-w")
Changing some Ediff key bindings
This is just the standard stuff that I like to do for each mode that
does not self-insert: assign j to move down, and
k to move up.
When you quit an Ediff session with q, it just leaves the
two diff windows around, instead of restoring the window configuration
from when Ediff was started. Here's the (slightly hacky) code to
restore the old window configuration:
In 80% of the cases, I call Ediff with e (magit-ediff) from
magit-status-mode. This gives me a better overview of the changes,
especially when I want to revert stuff with either a
(ediff-copy-A-to-B) or b (ediff-copy-B-to-A). Probably
with a, since magit seems
to consistently put the @{index} file into the A diff window, and
the current file into the B window.
So if I'm looking at a diff (to which I navigated with
j/k), I can revert it with a. At this
point, the diff region in the current file will become equal to that
of the index file. But it will not be saved yet. If I want to save the
current file in the B window, I can do it with wb
(ediff-save-buffer). After a revert, it's also useful to call
!(ediff-update-diffs), which will remove the zero-length
diffs and update the diff-count in the mode-line.
other methods
ediff-buffers will diff two (different) buffers. If you happen to
have only two windows open with the appropriate buffers, you get them
as defaults and can choose them quickly with RETRET.
ediff-files is similar, but works on files instead. I like to just mark the appropriate files
in dired with m, then they also get auto-selected by ediff-files.
ediff-current-file is useful to see the unsaved changes to the current file.
I rarely use it, since I have a compulsion to save every file every chance I get. There is little hope
to recover from this addiction, unless someone implements an analogue of guru-mode
that annoys you when you try to save a file. Actually, it's not a bad idea - C-x C-s:
Oleh, you have saved this file 40 times in the last hour; the last
save was 0 minutes 34 seconds ago; since then, you've changed 27
bytes of information. Are you sure you want to save [y/N]?
It will save the current buffer, run compile and switch to
compilation buffer.
Compilation customization
Turns out that there's no need to save, since I get away with this:
(setq compilation-ask-about-save nil)
The documentation says:
Non-nil means M-x compile asks which buffers to save before
compiling. Otherwise, it saves all modified buffers without asking.
Select compilation target
A year ago, I wrote a package that looks for a Makefile in the
current directory, parses the available targets and allows you to
choose one with helm. It's
called helm-make. It's almost
the same as M-xcompile (which also has completion for
targets), and it wraps around compile, but I just find it to be more
convenient.
This package is as simple as it sounds, the only customization that
you can do is to set helm-make-do-save to save all open buffers
visiting the Makefile's directory. It also provides, in addition to
the plain helm-make command, a helm-make-projectile command, which
is almost the same, except the Makefile should come not from the
current directory but from projectile-project-root.
Some error navigation to go with your compilation
I've had these bindings for a very long time, just making sure
that you're aware of these commands:
This is the first technical article in this blog, however the main purpose is not to analyse the problem and provide the solutions, but to tell a story of an ordinary person trying to pursuit his vision in a multi-languages environment (Emacs and HTML) that he only knows the basis. Hope you find it is interesting to read and for those who care the solution more than problem-solving approach, please see the last section.
The first time I thought I need an fancy Email is when I sent an quick model update to my colleague; I have a table like this
Conditioning Variable
Dependent Variable
Probability
k >= 50
t >= 50
0.154
k >= 50
t >= 100
0.111
k >= 50
t >= 200
0.078
It was written in org-mode in which I can do the formatting quickly and nicely. But once copied over to Outlook, it looks messy, and the columns does not lineup.
| Conditioning Variable | Dependent Variable | Probability |
|---------------------------------------------------–—|
| k >= 50 | t >= 50 | 0.154 |
| k >= 50 | t >= 100 | 0.111 |
| k >= 50 | t >= 200 | 0.078 |
The correct way is to insert a table in Outlook. First, I have to export the table to a CSV file, than open it in Excel, and finally copy it over to Outlook which will recognised it as a table.
HTML Attachment Solution
I guess the purpose of that email is to give my colleague few numbers, in a way that he can compare and gain a feeling of the model. So the format is really necessary, but the workaround is really tedious.
I have another colleague who is an HTML expert and produced an company CSS style-sheet. He was kindly customised it to match the org-export class, i.e. org-ur, org-table, org-list.
So what I did was to export to org-file as a HTML and attached it in the email so that my colleague can simply click and open it in a browser, which will gives him a nicely formatted table. But people have hundreds email per day and seems to dislike attachments.
Paradise of MIME
I noticed Bernt Hansen pointed out in his famous Org Mode - Organize Your Life In Plain Text! that he use org-mime to sent HTML Email. MIME, standards for Multi-Purpose Internet Mail Extensions, is an extension to plain email and enable user to exchange rich data includes image, table, video etc.
The org-mime can parse the org file into an HTML code, in a way that the email server like Office365 or Gmail can recognise and render it with pre-defined styles.
The default style looks awful: the font, the colour, size, basically nothing is right. Recently I sent about 3-5 emails using this style, I doubt the reader will spent less time in reading and comprehend it, therefore the message is not conveyed.
But the workflow is fascinating: I call org-mime-subtree function, then I just type few email address, no need to switch to system or Outlook, everything is done in Emacs and at the exact point where the main content is generated.
So I was thinking, what if the email is look as good as the attachment? What if I can apply the style to the email, that would be looks fanatic!
I did my research, the org-mime indeed provides feature to let user to change the HTML style, two example are showed on worg. The package first generate the HTML file, and than search-and-replace a certain chunk, for example,
1
2
3
<p>
this is a paragraph
</p>
will becomes something like this, depends on users specification,
1
2
3
<pstyle="blue">
this is a paragraph
</p>
The search-replace mechanics works fine, for a small email. It takes a pair value (element, style), where element can be paragraph, table, list and style can be colour, font, size etc. The problem is this pair is not quick match the standard CSS file,
1
2
3
4
5
6
7
body{font-family:"Helvetica Neue","Lucida Grande","Lucida Sans Unicode",Helvetica,Arial,sans-serif!important;font-size:14px;}body#content{padding-top:70px;}
One can processing the CSS file, and feed the package a long list of pairs. But this approach seems not safe. I quickly skim the CSS file and found something I couldn't understand, for example the body #content block above.
Hack org-mime
I think the most problem-free approach is to follow org-export-html and ensure the generated Email has same style as exported HTML and org-MIME package will eventually implement this, but I don't to wait and decide to hack.
The script is formatted in a nice way, and looks like a textbook C program: it first declares variable and functions, with concise documentation so one can visualise the structure after reading 5-10 minutes. But the implementation is way beyond my knowldge on Emacs-Lisp language. I almost looked up each function that be used, take this snippet for example,
I have no idea of what does it means. You know the feeling when you try to learn an foreign language but took the wrong book that way above your level, and you find there no single word you could understand, and you was like What The Hell? That was my feeling.
The strategy I came up was to build up my Emacs-Lisp vocabulary: try to understand the functions/processes and translate it into a plain English, for example,
with-temp-buffer
create a temporary buffer
insert
insert the string, in this case, called html, at point.
goto-char
move the cursor, which is called point in emacs, to somewhere
point-min
means the begining of a buffer/file
run-hook
run functions that links to org-mime-html-hooks
buffer-string
return a buffer as a string
Now that I understand each words, I need to comprehense it and combine than together to understand the mean of this snippet. I tried to write in a plain English and the first attempt is like this
create temporary buffer, insert the generated html file, than move the cursor to the very start, and than apply other functions that links to org-mime-htmize
I continue this word-sentence-paragraph process and I understand few functions. But it can goes on and on, and the more I learn about Emacs lisp, the future away I digress from my original goal: apply the style sheet to HTML email. I guess this is a common dilemma in working with multi-languages. Usually I follow my interests but this time I choose to focus on achieving the goal.
MIME Solution
It turned out it is a right decision. The concept of "inline-CSS" is mentioned int the script, I googled and found out the solution within 10 minutes. I realised that what I need to do is add a block in beginning of the HTML mail!! BINGO!
1
2
3
4
5
6
<head><style>...</style></head>
;; html email content starts here
Emacs Configuration
Here's the settings:
1
2
3
4
5
6
7
8
9
10
11
12
(require 'org-mime)
(add-hook 'org-mime-html-hook
(lambda ()
(insert
"
<head>
<style>
;; content of the .css file
</style>
</head>"
))
t)
It is very much in the spirit of C++ to pull an object into existence
with e.g. a constructor definition and then poke and prod it with
various method calls. Most of the time, the dot will have the object
name on the left. So why not automatically insert the object name each
time I press the dot? Below, I'll show a code that does exactly that.
The dot command
(defunc++-smart-dot()"Insert a dot or an object name plus dot when appropriate."(interactive)(let(var-name)(if(and(looking-back"^[ \t]*")(setqvar-name(c++-get-recent-var)))(insertvar-name".")(insert"."))))
This one is pretty simple: if we are at the beginning of the line,
optionally with some spaces or (ugh) tabs before the point, then try
to insert the object name along with a dot.
This code doesn't aim to be thorough, it's just a hack that
works reasonably well.
(defconstc++-var-regex"[A-Za-z][A-Za-z0-9_]*""The regex for C++ variable name.")(defunc++-get-recent-var()"Return the closest thing that looks like an object.The search is performed backwards through code."(save-excursion(when(or;; variable dot chain(looking-back(format" \\(%s\\)\\.%s.*\n[\t ]*"c++-var-regexc++-var-regex));; variable constructor init(looking-back(format"[\t ]+\\(%s\\)\\(?:([^)]*)\\)?;[\t\n ]*"c++-var-regex));; variable dot, first on line(re-search-backward(format"^[ \t]*\\(%s\\)\\."c++-var-regex)nilt))(match-string-no-properties1))))
If you're a stickler for performance and you don't want to call format a bunch of times,
you can amend the code like this:
(defalias'c++-get-recent-var(byte-compile`(lambda()(save-excursion(when(or;; variable dot chain(looking-back,(format" \\(%s\\)\\.%s.*\n[\t ]*"c++-var-regexc++-var-regex));; variable constructor init(looking-back,(format"[\t ]+\\(%s\\)\\(?:([^)]*)\\)?;[\t\n ]*"c++-var-regex));; variable dot, first on line(re-search-backward,(format"^[ \t]*\\(%s\\)\\."c++-var-regex)nilt))(match-string-no-properties1)))))"Return the closest thing that looks like an object.The search is performed backwards through code.")
The thing above looks slightly ugly. I'm open to suggestions to make
it look nicer.
I don't usually zoom a lot, typically I do it to size up other
people's functions. Then, I zoom-out a couple times in a row until the
function fits on the screen. Sometimes, I overshoot the zoom-out, and
I have to zoom-in instead. I have text-scale-increase bound to
<f2> g and text-scale-decrease bound to
<f2> l.
Below, I'll demonstrate how to call those two commands interchangeably
with e.g. <f2> g g g l g g l, i.e. omitting the
<f2> prefix.
Define repeatable command
(defundef-rep-command(alist)"Return a lambda that calls the first function of ALIST.It sets the transient map to all functions of ALIST."(lexical-let((keymap(make-sparse-keymap))(func(cdaralist)))(mapc(lambda(x)(define-keykeymap(carx)(cdrx)))alist)(lambda(arg)(interactive"p")(funcallfuncarg)(set-transient-mapkeymapt))))
This is a pretty simple function that takes an alist of keys and
commands, and returns a lambda that calls the first command and sets
the transient map to call the first and other commands. The way the
transient map works, it takes priority over almost all maps, but
disappears as soon as you press something that doesn't belong to it.
And this is it. A nice thing about this setup is that I don't have to
think about the zoom state and quitting it: I'll quit it automatically
as soon as I press anything other than l or g.
Moreover, the command that exited the zoom state will be executed as
usual.
I've been posting a lot about dired lately, and with good cause. A
lot of people say that org-mode is the killer app of Emacs, but
dired should be in that group as well, especially if you count
tramp as part of dired. Below, I'll list a few dired options in my
config that deviate from the defaults.
dired-listing-switches
This is the essence of what dired presents and how it presents it.
A great thing about it is that these are just the ls switches, so
you can look them up with info ls.
These settings make dired skip the confirmation when you copy or
delete a directory that contains other directories. What's the worse
that could happen, right?
One month ago, I officially announcedNames, a package that writes your elisp
namespaces for you. Today, I go into other ways in which Names can help. Think of
these as delicious Easter eggs hidden inside the shabby wood cabin that is the
define-namespace macro (which is built on top of an underground Machiavellic
engine of infinite cogs and spikes, but that's beyond the point).
Names is about turning your code into an actual cohesive package, as opposed to
a collection of functions with a common goal. Since it knows everything about
your namespace, it can use that information to simplify your code. These
features are all implemented as keywords you can pass to the macro, and are
documented inside the names--keyword-list variable.
As a practical example, take this simple snippet from camcorder.el.
The first two lines should be no surprise to you if you've read my introduction
to Names, but next three might. Those three tiny keywords are expressive and
easy to read, and save you a lot of code. The macro above expands to the
following.
(defgroupcamcordernil"Customization group for camcorder.":prefix"camcorder-":group'emacs)(defconstcamcorder-version"0.1""Version of the camcorder package.")(defuncamcorder-version()"Version of the camcorder package."(interactive)(message"camcorder version: 0.1")"0.1");; ...
Package name
This just defines the name of the package, which is also the name of the group.
If you don't provide it, Names will calculate it by taking the namespace (here,
camcorder-) and removing the last character.
Group definition
Most packages have a customization group. Names can define the group for you,
all you need to do is give it the :group keyword and tell it which group is
the parent of your package's defgroup.
The code above is specifying that Names should create a group for this package,
whose parent is the emacs group.
Version numbers
It is considered good practice by many for a package to define its version
number as both a constant and an interactive command. If you don't believe me,
see for yourself:
hit M-x,
type -version,
hit TAB.
By using the :version keyword, which is pretty self explanatory, you get a
constant and a command defined —both named PACKAGE-version— which return
the version you specify.
This concludes the most useful current keywords. The purpose of these
facilitators is not to write less code, writing is easy to automate. The
objective here is the same overarching goal behind Names itself, making the
source code shorter to read and nicer to look at.
To make use of the described features, make sure you require (names
"20150000") in your package. Also in the works are :require and :use, but
I'll let you know when these come out.
Today, I'll continue with the trend of posting a small piece of my
config when I don't have the time to post something more substantial.
Some code
This one looks nice, although it only works on systems with /usr/bin/du,
which actually comprise 100% of the systems that I use:
(defundired-get-size()(interactive)(let((files(dired-get-marked-files)))(with-temp-buffer(apply'call-process"/usr/bin/du"niltnil"-sch"files)(message"Size of all marked files: %s"(progn(re-search-backward"\\(^[ 0-9.,]+[A-Za-z]+\\).*total$")(match-string1))))))
On doing a search, turns out that I got this code from
the wiki at some
point. I can confirm that, unlike some of the other code on the wiki,
this one still works as advertised: you can use it on a directory or
on a series of marked files and directories.
Standard dired marking
In dired, you can:
mark an item with m
unmark an item with DEL
inverse selection with t
unmark everything with U
Compared to this, the selection with the mouse and the control and
shift keys that many file browsers use looks like kindergarten.
This new action, getting the size of marked things, I've bound to z:
This is just a small improvement to make e.g. \\( show up in
regular expressions without the escape chars, but instead fontified
with font-lock-keyword-face. It doesn't affect the underlying code
at all, just makes it look nicer. For the \\| I chose ∨ - the
logical or character.
At first, I wanted to just inline a picture,
but then I thought that htmlize-buffer would be able to handle it.
It didn't, so I just edited a small snippet by hand:
It's really satisfying to see those escape chars vanish as I type in
a capture group in the regex, especially with the help of
lispy-mode.
Here are some relevant tests for the regex support:
In the comments to
my previous post on ansi-term, I discovered
sane-term - a package that
cycles though your terminals in Emacs, as well as implements some of
the tips that I gave. While it's nice and all, and you should check it
out if you're looking for something like that, it's not really for
me. I will describe the system that I'm currently using below.
What is the best list length for cycling?
In my opinion, it's one or two. If it's one, you're not really
cycling, if it's two, it's fine. Anything more than that causes
stress, since you have to check each time if the outcome of the cycle
ended up being the one that you wanted.
That's why I usually have only one *ansi-term* active in my Emacs
session at all times. Here's how it looks like:
(defunterminal()"Switch to terminal. Launch if nonexistent."(interactive)(if(get-buffer"*ansi-term*")(switch-to-buffer"*ansi-term*")(ansi-term"/bin/bash"))(get-buffer-process"*ansi-term*"))(defalias'tt'terminal)
At one point, I had terminal bound to C-t, until I found
a command even better suited for that binding, which was
smex. The actual terminal
command isn't bound right now, I just launch it from smex on very
rare occasions.
How I launch terminal 95% of the time
From dired of course. The shell's natural way of switching the
directory with cd is extremely inefficient compared to dired. So
any time I want to have a shell in a specific directory, I first
navigate there with dired, sometimes combined with
ido-find-file. Then I get my current *ansi-term* and tell it to
switch to the current dired buffer's directory with `
binding:
(define-keydired-mode-map(kbd"`")'dired-open-term)(defundired-open-term()"Open an `ansi-term' that corresponds to current directory."(interactive)(let((current-dir(dired-current-directory)))(term-send-string(terminal)(if(file-remote-pcurrent-dir)(let((v(tramp-dissect-file-namecurrent-dirt)))(format"ssh %s@%s\n"(arefv1)(arefv2)))(format"cd '%s'\n"current-dir)))))
I also have a similar eshell setup, although I have yet to
comprehend why eshell is great and am using *ansi-term* most of the
time instead.
With dired-jump, of course. This command will examine your current
buffer's default-directory and open a dired buffer there. All you
need is:
(require'dired-x)
The dired-jump command will be bound automatically to C-x
C-j. I have it also bound to C-:, since that's more
convenient to press with my keyboard layout.
It's also better in the common situation when I want to jump to a
dired buffer from *ansi-term*. In that situation, C-x
C-j will not work by default, and will call term-line-mode
instead. But it will work once you are in term-line-mode. You can
go back to the default term-char-mode with C-x C-k. To
avoid this nonsense, just bind dired-jump to some binding that's
convenient for you and works from *ansi-term*.
What I do when I need more than one terminal
Then I just name one: since the default one is supposed to be named
*ansi-term*, if I create one named e.g. *jekyll*, it will be
ignored by dired-open-term. This is exactly what I want, since I just
create named terminals for long running processes like jekyll serve.
And I can switch to the named terminals with just ido-switch-buffer.
Here is the very simple code:
On seeing
this Emacs Stack Exchange question,
it occurred to me that if some config code is old for me, it's not old
for the new Emacs users. So I'll share one of the old ido-find-file
hacks that I've been using for ages.
This song is an oldie ...but, uh ... pause
Well, it's an oldie where I come from.
It wouldn't be a LISP if I couldn't generalize the code:
(defunido-find-file-jump(dir)"Return a command that sends DIR to `ido-find-file'."`(lambda()(interactive)(ido-set-current-directory,dir)(setqido-exit'refresh)(exit-minibuffer)))
Note that this is pretty ugly, implementation-wise, since
ido-find-file-jump would be called three times each time you do an
ido related command, like ido-switch-buffer etc.
I would have preferred to do it like this instead:
Note that the mapc statement needs to be evaluated if I dynamically
modify oleh-ido-shortcuts. This isn't a problem for me, but if I
wanted to package a code like this, I would define
oleh-ido-shortcuts like this:
(defcustomoleh-ido-shortcuts'(("~/""~")("~/Dropbox/source/site-lisp/""!")("~/git/lispy/""@"))"A list of directory-shortcut pairs for `ido-find-file'.":set(lambda(symbolvalue)(set-defaultsymbol(mapcar(lambda(x)(if(stringp(carx))(cons(ido-find-file-jump(carx))(cdrx))x))value))))
I tried to make the first call to youtube-dl asynchronous, but it
wasn't working out. So for the current code, there's still about a 2
second delay before the capture buffer appears.
I'm quite busy with a project today, so I can't compose many words.
However, pasting and explaining some code is fine. The basic idea is
creating TODO tasks in certain org-mode files by clicking a link in
Firefox, thanks to
org-mode capture.
org-protocol starter
(require'org-capture)(require'org-protocol)(setqorg-protocol-default-template-key"l")(push'("l""Link"entry(functionorg-handle-link)"* TODO %(org-wash-link)\nAdded: %U\n%(org-link-hooks)\n%?")org-capture-templates)
org-wash-link should clear up some redundancies in the TODO
org-handle-link should open the appropriate file and heading.
org-link-hooks should insert some extra information
org-wash-link
Basically, when I capture a question on
Stack Overflow, I
don't want to see - Stack Overflow - as part of the TODO string,
since the TODO itself is stored in wiki/stack.org/* Questions.
Youtube links will be handled with org-handle-link-youtube
Stack Overflow links will be stored in wiki/stack.org/* Questions
all other links will be stored in ent.org/* Articles
I'll write down org-handle-link-youtube in a later post, since I
would still like to sort out a few kinks with it. The main issue is
that I'm sending two requests to Youtube: one to download the video,
which is fine, since async handles it; and other to get the title of
the video and put it in the heading. And this other request causes a
perceptible delay when capturing.
I've finally wrapped a piece of config that I was using for a while in
a package called headlong.
What does it do?
It provides a macro called headlong-with that modifies minibuffer
completion for the forms within it, making it faster in some
situations. For instance:
(headlong-with(completing-read"Jump to bookmark: "bookmark-alistnilt))
or:
(headlong-with(read-extended-command))
But more importantly, it provides two commands that can use it
efficiently: headlong-bookmark-jump and
headlong-bookmark-jump-other. The second one is basically the same
as the first one, except it opens the bookmark with pop-to-buffer,
i.e. in other window.
How does this completion work?
It's nothing fancy, you will just exit the minibuffer automatically as
soon as there is only one completion candidate left.
So it saves you one keystroke, namely RET.
How much is one keystroke worth? It depends.
If you arrange your bookmarks in a way that I do, with each one
starting with a different letter, it saves you 33% of the total
keystrokes. For example, suppose I have:
Then I can jump to my bookmarked directory named "s: sources" with
two methods:
M-psRET
M-os
The second method looks like it's 33% shorter, but it feels like it's
even more, since pressing RET is harder than it should be
on most keyboards.
Why is this cool?
This is cool because you can implement your bookmarks as efficiently
as you would with just wrapping stuff with a lambda and using
global-set-key, except that you can view and edit the bindings with
bookmark-bmenu-list, and quickly the update bookmark positions with
bookmark-set.
Here's what I get when I call M-xbookmark-bmenu-list:
In the list above:
black bookmarks are files
blue bookmarks are directories
pink bookmarks are functions (you need bookmark+ for them)
Any Emacs package developer worth their salt knows the difference between a
defvar and defcustom. These two comprise the vast majority of variable
definitions in Elisp code, but there's a third child, the defconst. While
regular variables and customizable variables only really differ when it comes to
Emacs' customize system, constants differ in loading logic in a subtle but
important way.
When a defvar is evaluated, the value specified is assigned to the variable
only if the previous value was void (if the variable wasn't defined yet).
Evaluating a defconst, on the other hand, always sets the symbol's value to
the (new) given value. This becomes most relevant when a package is upgraded
after having been loaded in Emacs. Any variables defined by the package will
retain their old values until Emacs is restarted, whereas constants will be
updated to their new values.
In a sense, somewhat ironically, Elisp constants are the least constant of all
variables. Rest assured though, they were given this name for a reason. This
behaviour is indeed suited for storing values that should be constant. Here's a
simple use-case which arose for us in sx.el.
The StackExchange API requires that we provide filters to specify which
information we want to receive. Of course, we were storing these filters in
variables and then using that information while printing the question.
The problem with this approach would only be seen after an upgrade. Lets say we
decide to also display a question's score. Without a doubt, we'd change the
above to something like this.
Now the user upgrades sx.el, tries to run it again, and gets hit on the face
with an error!
sx-browse-filter is still using the old value, so the question score is not
returned by the API. That means .score is nil and number-to-string throws an
error. Changing the defvar to defconst was all we needed to prevent this
problem.
If you're wondering why you've never run into such an issue, that's because
package.el used to have another bug, which would never load new package
versions upon upgrade. That has been fixed in 25.1—packages are now reloaded
on upgrade—so you should ensure your package won't run into new problems by
making proper use of constants.
Simply put, you should defconst whenever there are functions or macros in a
package which rely on the variable having that specific value (kind of like a
proper constant). This way you'll know you never have old values hanging around.
If you haven't been living under a rock, you already know what
Youtube is. It's a repository with videos of
varying degree of usefulness with a terrible media player tacked on.
Instead, I like to watch my videos in
VLC, which comes closest to
providing an Emacs-like experience among video players.
Useful VLC shortcuts
Here is a list of shortcuts that really make me stick with VLC:
f - toggle full-screen
b - toggle audio track
n - toggle subtitle track
] - speed up play by 0.1
[ - slow down play by 0.1
M-right - forward by 15 seconds
M-left - backward by 15 seconds
C-right - forward by 60 seconds
C-left - backward by 60 seconds
M-1 - quarter of video size
M-2 - half of video size
M-3 - full video size
M-4 - double video size
C-h - toggle mouse buttons
So if you're not watching instructional videos at 1.6 speed,
or skipping the Simpsons intro sequence with M-right, you're missing out.
From Youtube to VLC
youtube-dl is an excellent command-line tool
for saving the videos from Youtube. It downloads the highest resolution at a usually
higher speed than Youtube's player buffers. I've discovered it when I had to download
a bunch of lecture videos from edX.
You can install it with:
sudo pip install youtube-dl
One Emacs script to rule them all
I quickly tired of opening a shell, setting the directory, entering
the command, and pasting the link. So I wrote some Elisp code that does it for me.
It's nothing too sophisticated, but I've been using this version for a couple months:
That's it. A new *ansi-term* will open with the task of downloading the video from the link
in the clipboard to ~/Downloads.
I don't have to wait for the download to finish and can immediately open the video from dired.
See the previous post for the description of
dired process-starting setup. I can stack up multiple downloads at once if I wish in different
*ansi-term*s.
This is my script. There are many like it, but this one is mine.
I did an internet search before writing this post. Apparently many
others had the same idea of integrating youtube-dl into Emacs. You
can use mine or any other code to generate a setup that works for you.
For instance, for a while, instead of copy-pasting the URL and calling
youtube-dl I used to just click the
org-mode capture
button in Firefox, and it would automatically create a TODO item,
download the video, and put the link to the downloaded video in the
TODO.
I've dropped this workflow when
the yank bug
surfaced. I don't yet have enough experience of working with Emacs's C
code to fix it. Although, according to this
excellent rant,
fixing the bug is only half of the problem: getting it merged is hard.
I'll see how it goes with my latest tiny patch.
So far it has been ignored, but it is only two days old as of now.
Here are the standard dired functions for starting processes:
! calls dired-do-shell-command
& calls dired-do-async-shell-command
While the second one is usually better than the first one, having the
benefit of not locking up Emacs, it's still not convenient enough for
me. The reason is pretty simple: I want to keep the processes that I
started even when I close Emacs (like opened PDFs or videos). This is
a non-issue for people with months-long emacs-uptime, but for me an
Emacs session lasts on the order of hours, since I mess about with
Elisp a lot. Below, I'll share some of my dired process-related
customizations.
Ignore running processes when closing Emacs
Usually there's nothing wrong with just killing a spawned process,
like an ipython shell or something.
;; add `flet'(require'cl)(defadvicesave-buffers-kill-emacs(aroundno-query-kill-emacsactivate)"Prevent \"Active processes exist\" query on exit."(flet((process-list()))ad-do-it))
Guess programs by file extension
With this setup, usually there's no need to manually type in the
command name.
This is the 15th post on this blog. Thankfully, no heart attacks after
the 13th one or anything. So I'll commemorate it with a post on
dealing with dates in calc.
How much time has passed since I started this blog?
--- Emacs Calculator Mode ---
2: <Sat Dec 20, 2014>
1: <11:56:27am Sat Jan 3, 2015>
.
Subtract with -:
--- Emacs Calculator Mode ---
1: -14.498044
.
This means that, if I want to maintain my one-post-per-day streak, I
still have half of a day to post this. Unfortunately, I have only an
old version of the blog repository on this machine, and the current
one is on a laptop at home. So I'll post this in the evening.
If you're new to calc, you'll probably wonder why you can't enter
negative numbers with -. It can be done with _,
just like in
J. Find
out more in the
info;
the
interactive tutorial
is absolutely excellent.
Prompted by this StackOverflow question,
I wrote down a new package called latex-wrap.
It's only a few hours old, but I like it a lot.
I've grepped the sources of AUCTeX, and it doesn't seem to have this functionality.
Here's how it works
You start with an active region:
Homer
Marge
Bart
Lisa
Maggie
After calling M-xlatex-wrap-region and selecting
enumerate from the list of environments (others being itemize,
center etc.), you get this:
\begin{enumerate}\item Homer
\item Marge
\item Bart
\item Lisa
\item Maggie
\end{enumerate}
Let's mark everything, M-xlatex-wrap-region, and select
center:
\begin{center}\begin{enumerate}\item Homer
\item Marge
\item Bart
\item Lisa
\item Maggie
\end{enumerate}\end{center}
As you can see, I want to make sure that it's possible to conveniently
continue by always placing the point on the end of the last line of
the inserted environment.
How it works with no active region
If (looking-back "^ *") is true, the current line is used as if it was
the region. Otherwise, an empty environment is inserted after the
current line.
Using the code
If you like the idea, you can check out the code at
github and test it out. If
you know of another package that already does this, do let me know,
otherwise I'll post the package on MELPA soon. And if you have some
ideas, or want to add a few environments that I forgot to mention to
the list, just post
an issue, I don't
bite.
There's no reason not to have /bin/bash instead of /bin/sh as the
default choice when you M-xterm.
(setqexplicit-shell-file-name"/bin/bash")
Tip #2
After you close the terminal, you get a useless buffer with no
process. It's probably left there for you to have a history of what
you did. I find it not useful, so here's a way to kill that buffer
automatically:
By default, C-y calls term's own yank, which is different
from Emacs's yank. So, until recently, I was using
S-<insert> to paste stuff into *term*. Here's a
more ergonomic way:
Sometimes new Emacs packages have to fight for their place in the sun,
as all the good bindings and huge keymap areas are already taken by
the older packages. This post will cover some practical problems that
you may encounter when your package needs to be aware of another
active package.
ace-window vs. helm
helm is a wonderful package,
it's my goto-package when I need completion. All of the following
packages use it:
But helm is super-greedy: once you're in the helm-minibuffer,
there's no way out except either a successful completion or a cancel.
But exiting the minibuffer for a short while may be useful. For
instance, you could copy some text and yank it in the
helm-minibuffer. The default minibuffer functions, as well as ido
easily allow it.
I started to investigate into this when I got
issue #15: Does not work with helm in minibuffer
for ace-window. I checked it, and indeed you could not ace-window
out of a helm-minibuffer. In fact, it was not possible to exit with
other-window either. This looked like good news, since I had an
inkling that it used to work at some point. So I checked out a
year-old version of helm and it did work.
magit-bisect to the rescue!
It's very simple:
Check out the master of helm
Call magit-bisect-start and mark HEAD as bad
magit will automatically check out a revision that is halfway
between a bad state and the initial commit. Now I exit Emacs and try
helm again to see if I can exit from the minibuffer.
It's kind of lame to have to exit Emacs, but somehow I don't trust
unload-feature to do the right thing. Anyway, if the thing works,
move to helm's repository and call magit-bisect-good, otherwise
call magit-bisect-bad.
Continue this process until termination. I needed 10 iterations
in this case.
In Emacs, set-transient-map has priority over
overriding-local-map, which was exactly what ace-window was using.
It's also funny that ace-window used to work with helm for a week,
since it was published on Mar 26 and helm switched to
set-transient-map on Apr 2. And I found out that it's not working only
now.
So I've amended ace-window with similar code. The way
set-transient-map works, the last call to it overrides the previous
one, so I thought that it would work out since ace-window is always
called after helm.
It didn't work out, because helm adds the code that calls
set-transient-map to post-command-hook.
In the end, this finally worked:
I thought that enabling lispy-mode for slime-repl-mode and
cider-repl-mode might be a good idea.
Extra navigation options are always good, and the ability to
call raise is just so useful.
But the problem was that SLIME has slime-repl-map-mode minor mode on
that competes for lispy's shortcuts, and CIDER has something similar
as well.
When two minor modes bind the same keys, which one wins?
The answer is the first one on minor-mode-map-alist.
This function I've found on the wiki:
(defunlispy-raise-minor-mode(mode)"Make MODE the first on `minor-mode-map-alist'."(let((x(assqmodeminor-mode-map-alist)))(whenx(setqminor-mode-map-alist(consx(delqmodeminor-mode-map-alist))))))
I didn't want lispy-mode to mess with minor-mode-map-alist too
actively, so I went with this approach: if lispy-mode is called
interactively, i.e. via a key binding or M-x, put it ahead:
It's still a work in progress, currently only lispy's RET
properly yields to call cider-repl-return and slime-repl-return
respectively. I'll see if there are more key bindings that need to
yield. Happy coding in the New Year!
I've discovered a nice Emacs-related blog -
Content AND Presentation.
Unlike The Axis of Eval, the blog
that I mentioned yesterday, this one is pretty self-contained, i.e. it
doesn't link to many external information sources. So I thought that
it's perfect for trying EWW to read it.
Not so fast! Configure first.
For modes that don't require to self-insert (and even for some that
do), I like to navigate with
h/j/k/l as arrows. These
arrows are on the home row, so navigation becomes a pleasant and
relaxed experience. But for EWW I decided to add a little twist: if
the point is in the first column, j/k will move
down/up not by one line, but by one paragraph. This is great for
concentration: I navigate to a paragraph, read it, navigate to the
next one. The point shows me the paragraph that I'm reading,
everything before the point I've already read. So every time I press
j, I get a tiny warm feeling of accomplishment.
I can also pair this with v, which I bind to recenter the
current line to the top.
As you see, I was careful to re-assign eww-back-url and
eww-view-source - the commands that I've displaced from l
and v. And here's the implementation of the arrows:
And while I was at it, I've added ace-link support for EWW. It was
pretty easy, basically the same routine as for help-mode and
info-mode. The only hard part was to figure out what part of the
code keeps binding shr-save-contents to o, my preferred
binding for ace-link. Turns out it was eww-link-keymap, and not
shr-map. So here's the new key binding code:
(defunace-link-setup-default()"Setup the defualt shortcuts."(eval-after-load"info"'(define-keyInfo-mode-map"o"'ace-link-info))(eval-after-load"help-mode"'(define-keyhelp-mode-map"o"'ace-link-help))(eval-after-load"eww"'(progn(define-keyeww-link-keymap"o"'ace-link-eww)(define-keyeww-mode-map"o"'ace-link-eww))))
If you haven't tried ace-link
before, you can get it from MELPA. The minimal configuration that you
need is this:
I'm using it from my git folder since when I want to edit a project, I
like to use smex's
jump-to-definition.
And use-package is useful
for the cases when my ~/git/ happens not to contain ace-link. In
that case, instead of getting a debugger error on starting Emacs, I
get only a polite "Could not load package ace-link" in my *Messages*
buffer.
Two months ago, I listed a few big things I was looking forward to for Emacs 25.
I knew I was being unrealistically optimistic to mention concurrency in Elisp,
but there was a point behind it. It bothered me a lot that I had to get up and
go for a coffee whenever I upgraded more than a few packages, and asynchronous
upgrades were the only way I saw of fixing that. This Christmas, as promised and
well ahead of schedule, I've implemented asynchronous execution into Paradox,
thanks to the fantastic async library.
What this means is that whenever you hit x on Paradox's Package Menu, be it an
upgrade, installation, or deletion, if you've installed the async library,
you'll have the option to perform the operation in the background. Then you can
get back to work and forget about it. Paradox will message you once it's done,
and that's it.
You can customize whether or not you want this feature with the
paradox-execute-asynchronously variable (the default is to ask you each time).
It is still considered a beta feature, so you won't find it on Melpa-stable yet.
You can get it on Melpa or wait a couple of weeks for it to be marked stable.
As a bonus, there's also a new command paradox-upgrade-packages, inspired by
this Emacs.SE question, which upgrades everything without the usual manual
labor.
Recently, I happened upon a wonderful blog called
The Axis of Eval. I knew that I'd
love it just when I read the name. (or emacs pales in comparison,
but I couldn't just sit on my hands for months or years while thinking
up a perfect blog name. If you think of a blog name as awesome as "The
Axis of Eval" and are willing to let me use it, I'll probably make the
switch.
The blog did not disappoint, featuring gems like this:
In the Lisp world, new languages are built by combining large,
battle-tested building blocks, and polishing or updating them when
needed, instead of starting over from toothpicks and double-sided
duct tape. A large Lisp like Common Lisp is like a toolchain of
decades-old tools that have proven their worth, and have been
codified in standards, folklore, and implementations.
The only thing in the way of extracting information and enjoyment from
this blog was the horrendous theme of black background, white
foreground and magenta links. Plus the RSS was kind of quirky, and I
couldn't just feed all of it into
Elfeed.
Greasemonkey to the rescue!
In the previous post I've mentioned that, in addition to using
the best editor, I'm using
the best browser. Well,
this particular best browser has an extension called
Greasemonkey
that allows you to automatically run your own JavaScript on certain
websites.
I'm not very proficient in JavaScript, the following code I just found
by searching around. The part to note is the @include - the pattern
of website names for which this script should be run automatically.
I can barely stand to look at it. How could you take Scheme and turn
it into this monstrosity? Such a shame. But it works, so I guess
everyone should learn JavaScript. All hail the mighty
HypnoToad JavaScript!
Currently, I'm using two methods for completing Elisp: company-mode
and helm-lisp-completion-at-point. The latter is the cannon, the big
gun: it always gets the job done, but I don't want to shoot at
sparrows with it. So I only bring it out for hairy cases, like for
stuff that starts with LaTeX-. Hence, the company-mode. But too
often have I typed region- only to find 7 candidates staring at me,
4 of them useless. Which prompted me to look for an additional
completion method.
I had the whole abbrev thing in the back of my mind until I saw a link
to the post
Abbrevs for the most frequent elisp symbols.
That's when I decided to act.
That post eventually links to a pastebin, where 1600 abbrevs are defined.
With my handy best extension
for best browser I've opened
the paste in Emacsbest editor by just clicking the edit button
in the RAW Paste Data section.
I had to M-xemacs-lisp-mode, since the file opened in
text-mode. And boy, it's big. In lispy-mode, I usually use
99j to navigate 99 sexps down and therefore to the end of
the list. Well, for this file even 999j wasn't enough.
I quickly tired of deleting one-by-one the each individual useless abbrev.
I mean:
ek -> echo-keystrokes,
when is that ever going to be useful? So I wrote this throw-away code:
After switching to a two-pane window layout, with point in the
pastebin buffer, calling foobar would count the amount of the abbrev
matches in
my most frequent elisp buffer.
If it was less than 5, the abbrev was auto-deleted, otherwise the
decision was up to me, as holding C-. would no longer
delete. In the end, there were only 56 abbrevs left out of 1600.
In one of the earlier posts,
I was discussing the implementation of an Emacs Lisp lexer for
Pygments. Here, I'll show how to install the
update and get nicely highlighted code in a pdf via the
minted LaTeX package.
The install
Assuming that you are on a Debian-related system:
sudo apt-get install mercurial
mkdir ~/git && cd ~/git
hg clone https://bitbucket.org/abo-abo/pygments-main
cd pygments-main
make mapfiles
sudo python setup.py install
And, of course, I'm assuming that you already have TeX Live installed.
I'm not too sophisticated about it, so I just install everything:
sudo apt-get install texlive-full
The result
So here I took some code from
a previous post and
copy-pasted it into minty.org file.
And here's the result of the PDF export (C-c C-e lo):
minty.pdf.
The red tape
org-mode had trouble exporting on my laptop until I did this:
cd /usr/bin/
sudo ln -s /usr/local/texlive/2013/bin/x86_64-linux/pdflatex
Also be mindful of the -shell-escape flag to pdflatex:
Do you use Emacs for writing the LaTeX, Markdown, or org documents? Do you have a set of specific settings only for writing? In this article I will share my experience of configuring a writing mode in Emacs that make it the most efficient writing tool for me.
I try to write as concise as possible and I use word count as a benchmark. Counting the words does not sounds like a trivial task in my cases because I have a habit to comment, even for general writing. I may comment out the whole paragraph, and leave a note aside about why, which are kept as it will be helpful in edit/review. These comments and notes should not be counted since the reader can't see them.
Addition to comments, there is a full list that does not count for technical articles, like source code, tables, figure captions etc. Some people may add reference section to the list as well.
org-wc provides the org-wc-subtree function that know what to count and what not to count. Also, org-wc-display will loop though all sections and overlay the number of words to each section headline. It is particularly useful when I need to know which sections needs to trim down and which to add more.
One of my daily achievement is to complete a writing challenge, which is about either to have write about 500 words or 45 minutes, whichever comes first. It is like a racing game for me, knowing the time or number of words is important. Tracking time is simple in Org-mode but words is problematic: I have to call the org-wc-subtree function manually. I raised a issues on GitHub and guided to nanowrimo mode, which updates the word counts while I am typing and shows it on mode-line.
It works out of box for me. The number of words is adjacent to the time I spent, which make it is very convenient to compare. Also, it calculate the average number of words per minute. It use this number to predict how long I need to achieve my daily goals (which is 500 words).
The picture above shows that I spent 30 minutes editing and there are 254 words in this section.
Variable-width Font
I have a little OCD about font since university. I use Time News Rome for formal report and any other serif font for general writing because they make paragraph and text easier to read.
There was a time my friend passed me a PDF file and asked me to review it. The problem was it was in Arif font (I think) which looks terrible, and also writing became unpleasant. This experience makes me to think what is the best font for writing.
I did some research and come across the concept of variable-width font. As a programmer, I use Adobe's Source Code Pro font as default which means I face monospaced font all day. For a monospaced font, each character has same space.
While for variable-doth font, each cahracter takes width corresponding to it's shape. For example, the length of "i" is about 1 of 4th of "w". Needless to say, variable-width font is more close the nature of hand-writing. Emacs has a built-in variable-pitch-mode that could change the font.
But will it make any different to my writing? I am not sure at this moment, but I would like to have a special font that I solely use in writing. The link between the font and my write mind will gradually become firm, and eventually increase my productivity in writing.
Sentence Highlight
Writing requires thinking and concentration. People have their own tips that help them to stay focus and get writing done, it may relates to a place, time or tools.
I tired many tips, like mediate before write, drink coffee, cut off internet but none of them works very well, the effects seems random. One problem I have in writing is that I jump between the sections quite often.
I tried to highlight the one sentence at a time so that I can focus on the one I am writing. I found hl-sentence package does exactly what I want. Also, I followed the author's suggestion and tweak the configuration to blur the other sentences to reduce the noise.
The current setting has two folder and helps me in a way that I can focus naturally: I don't need to force myself not looking other sentence.
The sentence highlight feature also has an big impact on my writing process by making the editing easier. One thing I want to achieve is to have proper length for each sentence/paragraph: If it is too short, I will merge it. If it is too long, I will break up into short sentences. The highlights give me a sense of the length visually which I used to get by reading or counting. To check how many sentences exactly for each paragraph, I move the cursor to end of a sentence by M-e, and then count how many flashes I have to reach the end of a paragraph.
Wrap-up
I am fairly happy about the nanowrimo, hl-sentence and variable-pitch mode and the powerful Emacs. Thanks to all the authors who wrote the scripts, because of their quality work, many things work out of box and I am able to have an seamless integration to the current workflow. It has became more efficient and productive, and makes me believe the Emacs is the best writing tool for me.
Which program do you use for writing? which feature do you like most?
It all started with a heated discussion with the author of
yasnippet over some minor
nonsense. In the end, we agreed to disagree, but not before he suggested:
So I hereby challenge you to create this stripped down, no-crap,
version of yasnippet. Dub it " tiny is not yasnippet " after your
grandiose views and in the glorious unix tradition of recursive
acronyms
The Thought Process
Well, doing exactly that would probably be lame, but I really loved
the acronym. Somewhere around that time I saw some post about using
eval-and-replace, i.e. inserting some Elisp in your non-Elisp buffer
and then replacing that code in-place with the result of the eval.
Here's the type of code that I was playing around with:
(mapcar(lambda(x)(*xx))(number-sequence17))
Then I realized that the code should probably produce a string.
Here's a more refined version:
Loops are a useful thing to have, they are a blind spot of
yasnippet, and looping is exactly what the code above does. The
parameters for this loop expansion are:
integer range start: 1
integer range end: 7
separator to join the expressions: ";\n"
Elisp expression to transform the linear range: (* x x)
format expression for the result: "hex: 0x%x"
So ideally, in order to have a package called tiny, I'd like to
keep only the parameters and throw away everything else.
The Result
Here's the final result of the shortening, and what tiny-expand would produce:
As you see, it's pretty compact, with only two characters which are not
actually the parameters of the template:
m signifies the start of the template. I think this way is much
better than something like having to mark the template body with a
region before expanding. tiny-expand should be called from the end
of the snippet, so there's no need to mark the end position.
| signifies the end of the Elisp expression and the start of the format string.
It can be omitted if your format string starts with a %.
Note also the use of shortened Elisp. You can still use the full thing
if you want. Or just use only the closing parens to resolve the
ambiguities.
The Demos
Here are some more snippets, you can click on them to see what they
expand to. You can also find them and more in the comments section of
the source code:
m10
0 1 2 3 4 5 6 7 8 9 10
m5 10
5 6 7 8 9 10
m5,10
5,6,7,8,9,10
m5 10*xx
25 36 49 64 81 100
m5 10*xx%x
19 24 31 40 51 64
m5 10*xx|0x%x
0x19 0x24 0x31 0x40 0x51 0x64
m25+x?a%c
a b c d e f g h i j k l m n o p q r s t u v w x y z
m25+x?A%c
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
0 and 0 and 0
2 and 1 and 1
4 and 4 and 2
6 and 9 and 3
8 and 16 and 4
10 and 25 and 5
12 and 36 and 6
14 and 49 and 7
16 and 64 and 8
18 and 81 and 9
20 and 100 and 10
* TODO http://emacsrocks.com/e01.html
* TODO http://emacsrocks.com/e02.html
* TODO http://emacsrocks.com/e03.html
* TODO http://emacsrocks.com/e04.html
* TODO http://emacsrocks.com/e05.html
* TODO http://emacsrocks.com/e06.html
* TODO http://emacsrocks.com/e07.html
* TODO http://emacsrocks.com/e08.html
* TODO http://emacsrocks.com/e09.html
* TODO http://emacsrocks.com/e10.html
* TODO http://emacsrocks.com/e11.html
* TODO http://emacsrocks.com/e12.html
* TODO http://emacsrocks.com/e13.html
* TODO http://emacsrocks.com/e14.html
m\n8|* TODO Wash dog %(+ x 2) \nDEADLINE: <%(date "Jan 1" (* x 5))>
* TODO Wash dog 2
DEADLINE: <2015-01-01 Thu>
* TODO Wash dog 3
DEADLINE: <2015-01-06 Tue>
* TODO Wash dog 4
DEADLINE: <2015-01-11 Sun>
* TODO Wash dog 5
DEADLINE: <2015-01-16 Fri>
* TODO Wash dog 6
DEADLINE: <2015-01-21 Wed>
* TODO Wash dog 7
DEADLINE: <2015-01-26 Mon>
* TODO Wash dog 8
DEADLINE: <2015-01-31 Sat>
* TODO Wash dog 9
DEADLINE: <2015-02-05 Thu>
* TODO Wash dog 10
DEADLINE: <2015-02-10 Tue>
You can expand them one-by-one to see what they do. As you can see,
Ruby-style interpolation is available in the format string. There's
also one special function called date that you can use there. It
takes the start date as a string ("Jan 1" in the example) and an
integer shift and prints an org-style date.
Then optional separator that defaults to a single space.
Then mandatory range end.
Then optional Lisp expr, that defaults to identity.
Then optional format-style string, that defaults to %d. You have
to separate it with | if the format string does not start with
%. You can also Ruby-style interpolation here, e.g. %(* x x).
With the point at the end of the snippet, M-xtiny-expand.
The Summary
In the end, tiny lives up to the name,
implementing only one snippet that can be used in a variety of ways.
Man, I just love toggles: the light switches, the
f - full-screen key in vlc, and the clicky pens (ooh, those
are the best). So I try to model some of my Emacs key bindings as
toggles.
Let me just quantify the two features that make a good toggle:
only two states: on and off
the state is visible at a glance
One could argue that with undo most editing commands become toggles.
But they're not, since each time you call undo, you mess with
Emacs's undo state. And the undo state isn't visible at a glance, so
both requirements for a good toggle aren't fulfilled.
I'll demonstrate the two editing commands that I use every day,
capitalize-word-toggle and upcase-word-toggle, that are good
toggles.
I may not have mentioned this before, but you should for the most part
ignore the key bindings mentioned on this blog. I'm actually using
them, they work for me because of my non-standard layout, but you
should assign what works for you.
Anyway, capitalize-word-toggle clearly has a state that's visible at
a glance: the first char of the current symbol. Also, there are only
two possible states: the char can either be upper-case or lower-case.
Hence, I can toggle this state with C-z for fun and profit.
Maybe some background on how this command is useful for me. I write a bunch of C++,
and the code features a lot of lines like this:
Triangulationtriangulation;// duh
So when I'm using auto-complete, it often eagerly expands to
Triangulation when I want triangulation, and the other way around.
So capitalize-word-toggle is super-useful there.
upcase-word-toggle's state becomes binary after you call it once,
since initially the thing at point could have mixed case.
But afterwards, it's either all lowercase or all uppercase.
So again, a clearly visible binary state is a good thing.
This command works either on (bounds-of-thing-at-point 'symbol) or
on the active region. Since region-active-p is deactivated after you
call the command once, there's some machinery to remember the state
and to toggle when called again.
A quick search brought me to this page:
Write your own lexer -- Pygments.
Turns out that the Pygments development takes place on
Bitbucket, so I had
to start an account there. I shortly cloned the repository:
__all__=['SchemeLexer','CommonLispLexer','HyLexer','RacketLexer','NewLispLexer','EmacsLispLexer']classEmacsLispLexer(RegexLexer):""" An ELisp lexer, parsing a stream and outputting the tokens needed to highlight elisp code. """name='ELisp'aliases=['emacs','elisp']filenames=['*.el']mimetypes=['text/x-elisp']flags=re.MULTILINE# the rest of the code was copied from CommonLispLexer for now
Apparently, infrastructure-wise, I only need to know two commands.
The first one needs to be run just once, so that Pygments is aware of the new lexer:
$ cd ~/git/pygments-main && make mapfiles
The second command is to (re-)generate /tmp/example.html:
$ cp ~/git/emacs/lisp/vc/ediff.el \
~/git/pygments-main/tests/examplefiles/
$ ./pygmentize -O full -f html -o /tmp/example.html \
tests/examplefiles/ediff.el
I would repeat the last line with each update to the code, and then
refresh the page in Firefox to see the result.
The Elisp
To finalize the lexer, the following tasks ensued:
get a list of built-in macros
get a list of special forms
get a list of built-in functions
In the process, I've added two more lists:
a list of built-in functions that are highlighted with font-lock-keyword-face:
To generate the other three lists, I started off writing things in
*scratch*, but after a while my compulsion to C-x C-s
kicked in and I've saved the work to research.el. At least,
thanks to @bbatsov,
I'm not C-x C-s-ing that much since I've added this:
This way I can cycle the buffers with the same shortcut that invokes
save-and-switch-buffer. The defaults are C-s and
C-r, in case you didn't know.
The C
Getting the list of built-in C functions and special forms, obviously
involved browsing the C source code. In case you don't (yet) have the Emacs sources,
they're here:
$ git clone git://git.savannah.gnu.org/emacs.git
I switched to the ./src directory and called M-xfind-name-dired with
*.c to build a list of all the sources.
Then I ran the following code from research.el:
You can read up on the special forms in the
SICP.
There's no node for them, so just use isearch.
The Result
You can see it here: ediff.html, as well as on
the rest of the site, since I've switched it on everywhere.
The Impact
Unfortunately this won't have impact on the Github source code
highlighter, since
Github dropped Pygments
recently.
But people that use the static blog generator
Jekyll or the LaTeX package
minted (that's the package that
org-mode's PDF Export uses by default) will be able to get better
Elisp highlighting. In fact, this blog is already using the new highlighter.
Do you know what the
most frequently used
Emacs commands are? I can confirm by my own experience that they are
next-line and previous-line (in that order).
So why do M-u - upcase-word, M-l - downcase-word, and
M-c - capitalize-word have such terrible synergy with Emacs's best commands?
No, upcase-word, this is not what I had in mind:
Learn basic keystroke commands
Overview of Emacs features at gnu.org
M-u
Learn basic keySTROKE commands
Overview of Emacs features at gnu.org
If you say:
But how did you get the cursor in such a crazy position in the first
place? You should have used M-b/M-f.
Well, I got there with previous-line - one of the best Emacs commands!
Resolve the *-word malarkey with defadvice
Here are some simple advice commands that I've just rolled:
before upcase-word is called, execute the body of upcase-word-advice
unless we are at the beginning of the word
backward-word once to move to the beginning of the word
I'm intentionally not using the newest advice system here, since not everyone
has yet upgraded to Emacs 24.4.
In fact, I saw this gem today at Stack Overflow:
Expand region is one of those packages that deserves to be built-in. It's so
simple and useful it makes Emacs worthwhile all by itself. Fundamentally, the
entire package boils down to a single command, expand-region, which
incrementally increases the selected region by semantic units. So deciding where
to bind this command is an important decision.
The Readme suggests C-= and I've seen recurring recommendations for S-SPC,
which I used for a while, but neither felt quite right for me. After marking a
small region, the most common operation for me is to copy it, but it's very
awkward for my fingers to hit S-SPCM-w. With that in mind, in one of my
brightest epiphanies, I defined the following keybind.
(global-set-key(kbd"M-2")#'er/expand-region)
Not only does M-2 use the same modifier as M-w, but it's right above the
latter. I couldn't hope for a better key. Besides, after years of switching tabs
in Firefox, my fingers are sniper rifles when it comes to hitting M-2, making
this combo extra speedy. The downside is that you forgo using Meta for prefix
arguments, but I'm fine with that.
When I select large regions, I'm probably killing instead of copying. While
M-2C-w is not terribly natural, it's also not awkward, so the key is good
enough here. Finally, M-2 also fits well with my multiple-cursors keybinds,
but that's a topic for another post.
I was very surprised to find the
lispy build broken after I pushed
some minor update, like a change to README.md. I mean, how in the
world would a few words in README.md break the Elisp tests? Upon
investigation, it turned out that only one test was broken 1. This
one:
The culprit was an update in clojure-mode's indentation.
The previous behavior:
(->>[12345](map sqr)(filter odd?))
is now replaced with:
(->>[12345](map sqr)(filter odd?))
Thankfully, the Travis CI in combination
with cask is keeping me up to date.
Apparently, there were some
heated discussions
accompanying the change, and there was some reverting going on.
Anyway, it looks to me that both approaches have merit: the first one
is more logical, since ->> is an operation akin to Elisp's
with-current-buffer, where the first argument is different from the
others, while the second one is more aesthetically pleasing. Fine
with me either way, I'm not complaining:)
Also, the key sequence in the test is pretty ancient. These days I'd
probably use:
2(->>C-fd<j<skwAM. I've
recently done a more complex Elisp refactoring screencast, check it
out here. Later on, I
plan to do more Emacs-related screencasts (not just lispy-related)
on my channel.
If you haven't tried lispy yet, you're missing out -
doing this refactor operation feels like you're doing the 15-number puzzle:
And that's fun in my book. But let me get back to the short overview
of the Emacs testing tools that lead me to this post, mainly cask.
Cask is a project management tool for Emacs Lisp to automate the
package development cycle; development, dependencies, testing,
building, packaging and more.
Yes, please, I'd like to do that! But after the exciting intro
sentence, there's very little followup documentation-wise. It took me
ages to figure out how cask can actually give me some tangible
benefits, since I thought that package.el is enough to maintain my
own config (it still is).
tangible benefits of cask
I'd like to be sure that my packages work across recent Emacs
versions. I'm using the
bleeding edge myself, but
people who download my packages from MELPA might be using something
older, like emacs-24.3.
So I want to run my tests on both versions. Also, even for just one
version, the tests need to be run in a minimum environment, i.e. with
only the dependencies loaded, so that my personal configuration does
not interfere with the tests.
This is where cask actually shines: it can bootstrap a whole new
.emacs.d, separate from your own, just for running tests. It can do
it on your machine as well as on Travis CI.
As you can see, the Makefile has two separate testing targets: an
interactive one (compile) and a non-interactive one (test).
There's actually some validity to this, since it happened once that
the same tests we failing in non-interactive mode, but passing in
interactive mode. Also, compile obviously compiles, testing for
compilation warnings/errors. I can change the Emacs version at the
top, although I don't have to do it too often.
So each time I push a change to github, Travis CI will
install emacs24
install cask
install the packages from MELPA:
helm
ace-jump-mode
noflet
iedit
multiple-cursors
cider
slime
geiser
projectile
s
highlight
load Emacs with these packages
load lispy-test.el and run it
show up green if make test returned 0
Seems a bit wasteful, but it's the Cloud - what can you do?
upon even further investigation, the test itself was broken for almost a year, since lispy-with-clojure should have been used instead of lispy-with, but cider was changing the indentation of ->> also for emacs-lisp-mode, so things were kind of working out ↩
When you press DEL (also known as backspace) in a helm
buffer, and there isn't any input to delete, it only errors at you
with:
Text is read only
Why not make it do something useful instead, for instance close helm?
Easy:
(require'helm)(defunhelm-backspace()"Forward to `backward-delete-char'.On error (read-only), quit without selecting."(interactive)(condition-casenil(backward-delete-char1)(error(helm-keyboard-quit))))(define-keyhelm-map(kbd"DEL")'helm-backspace)
My name is Oleh and I've been using Emacs for about 3 years now. I
think that it's an awesome editor, and I've accumulated some know-how
to make it even better (at least for me and people who think like me).
Sharing is caring, so here we go.
ace-window update
On this weekend I've made a major update to my package
ace-window that allows it to
be used as a library. Luckily the change went smoothly, as there are
no complaints in the github issues so far. In case you don't know
what the package does in the first place, a short blurb follows.
ace-window's "what and why"
I'm sure you're aware of the other-window command. While it's great for
two windows, it quickly loses its value when there are more windows:
you need to call it many times, and since it's not easily predictable,
you have to check each time if you're in the window that you wanted.
Another approach is to use windmove-left, windmove-up etc. These
are fast and predictable. Their disadvantage is that they need 4 key
bindings. The default ones are shift+arrows, which are hard to reach.
This package aims to take the speed and predictability of windmove
and pack it into a single key binding, similar to other-window. To
achieve this, I'm using the excellent
ace-jump-mode.
The learning skill becomes more and more important nowdays because there are just so much to learn, either from daily job or personal interest. But have you ever thought about the way you learn or how good is your learning skill? In this article, I want to share my experience in learning how to build this blog using Jekyll and how I exam the way I learn via data and statistical analysis. It is worth reading if you:
want to improve your learning skill,
are trying to learn Jekyll or want to build a personal website,
are interested in quantified-self project.
Why I Learn Jekyll/Blog
All my high-school classmate known that I am really bad at writing. Things becomes worse when I was in university studying mathematics. I start to void writing at all the cost: I deliberately volunteed the do the coding or maths bit and let others do the writing. Everybody in the group seems like me.
During my study in Warwick University, I fancy the Statistics and really enjoy telling people the relationship I between all sort of facts. Then I come across The Guardian's Data Journalism program which is a term in use since 2009/2010, to describe a journalistic process based on analyzing and filtering large data sets for the purpose of creating a news story1. I found it is really cool! I tried to analyst a new dataset from World Bank and want to write an article but when I sit down my mind just completely blank. I managed to write a few paragraph but it was really awaful. That was the first time that I thought I wish I have a proper writing skill. I won't bother to learn because I planed to go back to China soon but I decided to work in the UK at the last minute.
Last week, I was doing a statistical consutlantcy project. I was do the analysis, writing code in the morning then try to write it up in the afternoon. I realise it pretty easy for me to do the analysis and coding, but it was really a pain to write it up, but I do enjoy it. So I decided to build a personal blog so that I can practise my writing skill and at the same time, to promote the usage of statistics in daily life.
I have no prior knowledge of building a website and spent couple of evenings/weekends sitting in front of computer at library/caffee and try to learn. This project accros two months and takes me about 20 hours in total to finally build this blog. Motivation really is crucial to learning. What make this learning expeirence really unique is that I have data about my learning.
Process Raw Data
I have a very good habit: I record every single tasks I do in terms of how much time I spent and what I did. Take this task for example,
PM (strcture)
SCHEDULED: <2014-12-11 Thu 17:45>
:LOGBOOK:
CLOCK: [2014-12-11 Thu 17:41]--[2014-12-11 Thu 18:28] => 0:47
:END:
:PROPERTIES:
:Effort: 0:45
:END:
[2014-12-10 Wed 22:28]
wait for 4 minutes
- [ ] go though all the headlines,
- [ ] group into few categories, like 1) learn, 2) apply, 3) improve etc.
- [ ] start to edit
This note includes all the info I need to know:
I want to do it On Wed and estimate it takes 45 minutes to compltete
I planed to do on Thurs,
I started at Thur 17:41, 4 minutes before the scheduled time,
It takes me 47 minutes to do the job, 2 minutes longer then expected.
what i did, the main body
I gathered all the notes that are relevant to this project and accumulate the time I spent for each sub-tasks. It can be summaries as a table:
Table 1: Clock summary at [2014-12-10 Wed 21:42]
Headline
Time
Total time
17:22
TODO blog
17:22
DONE Jekyll official guide on github…
1:22
workflow
1:45
TODO how to change the look of html
2:11
org-jeykll workflow (final)
1:26
DONE blog, does not looks good on…
0:10
discovery jeykll template
1:57
DONE Jeykll disaster
1:00
NEXT tweak jkyll (add social content…
1:17
DONE Jekyll (disqus and google…
0:25
Jekll,
2:23
Jekyll general search
0:53
NEXT intro to jekyll
0:28
Jekyll code highlight
1:03
DONE Jekyll - non-doing action
1:02
It tells that I spent 17 hours and 22 minutes in total on this project. The first question is: what does this 17 hours mean to me? I spent 500hours plus in playing video games between 2013-2014 and I really don't get nothing out of it. I really enjoy this learning experience because I have a definitive goal I want to achieve, then I put effort and I can get some results.
The rest of the table tells me tasks I did in chronological order. The first task was to read the official Jekyll guide and I spent 1 hour 22 minutes. This table looks really awfully and gives you an insight of what a real world data looks like. I don't really know what to do with it. So I have to refers to the way I learn as a child.
From our education system, the way we are learning is that we start on a basic level, we study the topic, apply it, make mistakes, correct it, and continue this process to the next level. It gives me an idea to define "levels". So I skim the whole project, from beginning to end, and grouped the data into five categories:
Basis
the foundation of Jekyll, website and HTML language,
Features
he extended features provided by Jekyll, for example, add social network link, add a discussion/comments,
Workflow
integrate the publishing process to my current workflow and try to automate as much process as possible,
Try
try to impelement something new that for my own needs, or try other people ideas,
Fix
fix problems along the way I build up the website
Analysis 1 - Mixture of levels
There is a linear increasing level, and one dependence the previos level. Ideally, the most cost-effective approach for a person to learn, is to focus on step, master it, and then goes to the next. This is the way of our education system. But it won't be the case of a self-study project, which is most likely to be interest-driven. it wold be very interesting to see that how I was jumping between these five levels. I sort out the timeline for each tasks I did and plot the time with levels.
Figure 1: timeline view
The x-axis is the time line in minutes for this project and y-axis is the level I defined. I can tell which level I am in a studying time. For example, for the first 200 minutes I was studying the basis knowledge, then I spent 35 minutes on Feature and so on.
I am very surprised that I went to workflow level so early, in about 20% of this project. It is actually make a lot sense because this project across 2 months and having a workflow that suitable for me really accelerate this project because It takes no time to pick up after leaving this project for few days.
There was a time that the website is broken and I cannot figure out why. It turned out I missed spelled an configuration file.
Finally I spent some time on googling about how other people use Jekyll and tried quit a few and never goes back to lower level.
Analysis 2 - Time distribution
It would be also interesting to see how long I spent on each levels, i.e.
Figure 2: time distritbuoin
Level
Time (Min)
Percentage
Basis
184
0.18
Feature
276
0.26
Workflow
191
0.18
Fix
70
0.07
Try
321
0.31
The pie chart should be read in clocking-wise direction and it is ordered by levels. I spent 3 hours in Basis level and to be honest I understand much about Jekyll. It is interesting to see that I spent 8% more time on the expanded features than the basic knowledge, i.e. using it rather study the knowledge.
Thanks to all the volunteers that working on connect Jekyll and Org mode. I only spent 3 hours in setting up the workflow that includes write article in Org mode (plain text), convert it to HTML web page, and then upload to my blog. Jekyll is a sophisticated and well tested software that it can be configured easier and I didn't running into any problems.
The rest 5.4 hours was the most inefficient in this project. At that time I was keen to include Table of Content and Code Syntax Highlight in my blog. I did a lot further research for solutions and tried a lot. But none fits well with the foundation I have built up. Some is buggy and create more problems. If you really need these feature, see this blog.
Analysis 3 - Benefits
The simple question: is it worth spending 18 hours building a website side? In timewise, I have spent 20 hours on writing, but this number does not count, because I can write without a blog at all. But I feel that having this blog promote me writing because:
the writings has their destination. It is not a digital file in my computer that I will forget in few weeks or a page in a notebook I hardly look back with potentially lose. All my writings will be in this single website that has a unique address that everybody can visit it whenever or whatever they are.
the aim for writing has been changed. It is not only to express myself, like a daily, but to share an journal to other. I have to consider the reader's feel, will they like it? will they understand it? So, the writing becomes more proactive, more thinking on human relationship, and thus more fun.
Writing is linked to this website, which is linked to quantifies-self project and Emacs/org-mode, which then linked back to statistics and programming, which are the two main passion of me. Writing is not something that occurs to my mind one day and then I swear I will master it, but something in my passion network and extended my passion to another area.
Conclusion
[2014-12-19 Fri 13:21]
First, quantify my time on learning is
let-alist is the best thing to happen to associative lists since the
invention of the cons cell. This little macro lets you easily access the
contents of an alist, concisely and efficiently, without having to specify them
preemptively. It comes built-in with 25.1, and is also available on GNU Elpa for
older Emacsen.
If you've ever had to process the output of a web API, you've certainly had to
deal with endless alists returned by json-read. I'll spare you the rant and go
straight to the example.
Here's a very simplified version of a function of the SX package, before let-alist.
How much nicer is that? let-alist detects all those symbols that start with a
., and wraps the body in a let form essentially identical to the one above.
The resulting code is much nicer to write, and the byte-compiled result is
exactly as efficient as the manually written version. (If it's not
byte-compiled, there will be a performance difference, though it should be
small.)
And just to make things nicer, you can use this snippet to highlight those .
symbols as if they were keywords.
Due to popular demand, let-alist now does nested alists. The example above
shows how you can use .owner.display_name to access the value of
display_name inside the value of owner.
A little over a month ago, I released a package called Names, designed for
mitigating Emacs' namespace issue. Before I even had a chance to announced it,
it made a bit of a splash on r/emacs, which I've taken to mean that people are
interested. I've been holding off on this post until I had a couple of
Names-using packages under my belt, so I could actually speak from experience as
opposed to expectation, and that's finally the case.
Names aims to provide an implementation of namespaces in Emacs-Lisp with four
guiding principles:
Practical
Actually useful and easy to grasp.
Complete
Support any function, macro, or special-form available in
emacs-lisp, even the ones defined by you or a third party.
Robust
No-surprises, well-tested, and with clearly stated limitations.
Debuggable
Support edebug, find-function/ variable/ face,
eval-defun, and eval-last-sexp (also known as C-x C-e).
Integration with other developing tools are under way.
Why a namespace package?
The lack of actual namespaces in elisp is a recurring topic in the Emacs
community. The discussion is long and beyond the scope of this post, so I'll
refer those interested to Nic Ferrier's great essay on the subject and to a
concise explanation by Jon Snader as well.
In short, Emacs takes the approach of prefixing every symbol name with the name
of the package. This successfully avoids name clashes between packages, but it
quickly leads to code that's repetitive and annoying to write. The links above
mentions many shortcomings of this approach, but this code-cluttering repetition
of the package name has always bothered me the most. It makes the code longer
and more tiresome to read
See this example from package.el. The word “package” is repeated 7 times in
a 10-line function.
It doesn't change this approach, nor does it try to revolutionize Emacs or
reinvent packages. Names fully adheres to Emacs conventions and is completely
invisible to the end-user, it simply gives you (the developer) a convenient
way of writing code that adheres to this standard.
Here's what the code above would look like inside a namespace.
define-namespace is a macro. At compilation, it expands to generate the exact
same code as the original, thus being completely invisible to the user.
How reliable is it?
I'll forgive you for being sceptical, wrapping your entire package in a macro
sounds risky. Rest assured, Names is very smart in what it does, and it
strives to avoid surprises.
Furthermore, it features a thorough test suite which leverages off other
well-tested packages to the respectable sum of 244 tests. Of course, the number
by itself has no objective meaning, but it shows that I'm commited to making it
as reliable and robust as possible. To prove I mean business, the last twopackages I pushed to Melpa already make use of it. And if that's not enough, I
just got news that shell-switcher has joined the party and seems to be passing
all tests.
In terms of availability, it's on GNU Elpa, so it's a safe dependency for any
Emacs starting with 24.1.
And how do I use it?
If you learn better by example, you can have a quick look at this short dummy
package available on the repo or, for real world packages, see camcorder.el or
aggressive-indent. In any case, it's important to check the Readme for the most
up-to-date instructions but here's the gist of it:
List names as a dependency. A typical header will contain
;; Package-Requires: ((names "0.5") (emacs "24"))
Write all code inside the define-namespace macro, preceded by an
;;;###autoload cookie. The first argument of the macro is the prefix which
will be applied to all definitions. require and provide statements go
outside the macro.
;; `require' statements come first.(require'whatever);;;###autoload(define-namespaceexample-;;; Code goes here)(provide'example);;; example.el ends here
Inside the macro, instead of using ;;;###autoload cookies, use the
:autoload keyword.
And that's pretty much it. Just write your code without prefixing every symbol
with the package name, and Names will take care of that for you.
1. Every definition gets namespaced
Any definitions inside will have the prefix prepended to the
symbol given. So the code
2. Function calls and variables are namespaced if defined
This is best explained by example. This code:
(define-namespacefoo-(defvarvarinfinite)(defuninfinite(x)(infinitex))(infinite2);; Local function call(something-elset);; External function call(::infinitevar);; External function callinfinite;; Variable)
expands to this code:
(defvarfoo-myvarinfinite)(defunfoo-infinite(x)(foo-infinitex))(foo-infinite2);; Local function call(something-elset);; External function call(infinitefoo-var);; External function call.infinite;; Variable.
Note how:
The infinite symbol gets namespaced only as a function name, not when it's
used as a variable. That's because Names knows that foo-infinite is not a
defined variable.
The symbol ::infinite is not namespaced, because it had been protected with
::.
something-else is not namespaced, because it is not a locally
defined function, so it must be external.
3. Quoted forms are not namespaced.
Whenever a form is not meant for evaluation, it is left completely untouched.
The most significant example of this are lists and symbols quoted with a simple
quote (e.g. 'foo). These are regarded as data, not code, so you'll have to
write the prefix explicitly inside these quoted forms.
Some examples of the opposite:
Symbols quoted with a function quote (e.g. #'foo) are regarded as function
names, and are namespaced as explained in item 2. Remember you should use the
function quote for functions anyway.
Comma forms inside a back-tick form (e.g. `(nothing ,@(function) ,variable))
are meant for evaluation and so will be namespaced.
But is it all worth it?
Absolutely! Like I said, I've already written two packages using Names, and I
had a blast! Of course, my opinion is biased. But I can say with all honesty
that it's an absolute delight to not need to worry about all those prefixes.
I invite people to try it out. If you do, make sure you (require 'names-dev)
in your init file. This will enable integration with edebug, find-function,
eval-defun, and eval-last-sexp. I've already got news that shell-switcher
has made the conversion, and alchemist seems to be on the way.
What’s PGP, and what can we do with it? George walks us through
creating and uploading a key, encrypting and signing emails, git
commits, and files, and introduces the web of trust.
When writing prose, I find auto-completion to be more of a distraction
then an aid. However, my field of research involves much
repetition of some annoyingly long words, such as “thermalization” or
“distinguishability”.
Instead of completely disabling auto-complete, I've found it useful to dial it
down a notch, while still letting it trigger on these monstrosities.
(setqac-auto-start3)(setqcompany-minimum-prefix-length3)(defunendless/config-prose-completion()"Make auto-complete less agressive in this buffer."(setq-localcompany-minimum-prefix-length6)(setq-localac-auto-start6))(add-hook'text-mode-hook#'endless/config-prose-completion)
This will offer completions based on previously typed text, but only when I
stumble while typing a long word.
Do you use any kind of tab completion for writing prose? It seems
company-mode has an Ispell back-end, has anyone ever tried that?
Continuing on this cheerful series, we now go into rectangles. The release notes
are pretty self-explanatory on this one, so a few gifs should be enough to
convey what's needed.
* Rectangle editing
** Rectangle Mark mode can have corners past EOL or in the middle of a TAB.
* C-x C-x in rectangle-mark-mode now cycles through the four corners.
* `string-rectangle' provides on-the-fly preview of the result.
Now that the previous post has leveled the playing field, we go into slightly
more advanced debugging features. First we go deeper into Edebug navigation
commands, and then we discuss when Edebug just won't do and explain the power of
Emacs' built-in debugger.
Edebug
Stepping through a function with n and finishing with c is just a subset of
what Edebug can do. You can find the full list of commands on Edebug's manual
page (under “Modes”, “Jumping”, and “Misc”), but here are the most useful ones:
q
Quit current execution, occasionally more useful than c, C and G.
h
Move point somewhere else, then hit h. Everything in between will be
“skipped” (executed, but not displayed), and Edebug continues from
there. Highly useful in very long functions.
o
Move out of the containing sexp. Tremendously useful in loops.
i
Go “inside” the function called at point. In other words, if point is
immediately before a function call, i will instrument the function as
if you had called C-u C-M-x on it and leave you inside this new
Edebug session.
You also have the possibility of setting breakpoints, which I may go into later.
But the keys above are enough to do what Edebug is best at, interactive
investigation when you just don't know where the problem is.
debug and debug-on-entry
As shiny as it is, Edebug isn't exactly clean. Its instrumentation, though
invisible, is not free of side-effects. It is not surprising then, that some
bugs simply vanish when you Edebug them. Fortunately, that's not our only
option. Emacs also offers the debug function, which is powerful and
straightforward.
Just add (debug) anywhere inside a function. Next time the function is
executed, Emacs will present you with a backtrace of the code state at that
point. Alternatively, you invoke the command debug-on-entry and give it the
name of a function.
Now, you might be forgiven to think there's nothing special about that. Any
programming language worth its salt can spew out backtraces on demand. But
that's not the end of it. Though this backtrace display might not be as pretty
as Edebug, it's every bit as interactive.
There's a gif here (linked separately to avoid annoyance) which shows what you
can do by just hitting d to step deeper into the structure, and then c to
quickly get out. You can also jump around a bit with j, and evaluate
expressions under an environment as if you were inside the function with e.
These are all much easier to use than to explain, so give it a shot yourself.
Write (debug) inside a function of yours and start jumping around in there.
You should get the hang of things quickly enough.
Running into errors is not only a consequence of tinkering with your
editor, it is the only road to graduating in Emacs. Therefore, it
stands to reason that Emacs would contain the most impressive
debugging utilities know to mankind, Edebug.
Learning to Edebug is the right decision for anyone who doesn't know
how to Edebug.
It's easy as pie, you have no excuse not to learn it.
It helps you solve your own problems. Not that the community isn't
helpful, it just allows you to help yourself—which will eventually
allow you to help the community.
If you're a newcomer, it will teach you more about elisp than most
tutorials out there.
If you're anything other than a newcomer, then it's about time and
you should be ashamed of yourself.
A quick first stop
Before delving into Edebug, you should be aware of toggle-debug-on-error and
toggle-debug-on-quit (which happen to be in our toggle-map). Though not as
powerful as the alternative, they're a quick n' dirty way to get a backtrace of
your problem.
How to Edebug a function
Go to where the function is defined. You can usually do that with
C-h f (which calls describe function) or just
M-x find-function.
Press C-u C-M-x. This turns on Edebug for that function.
Now, just invoke the function (or some other command that calls
that function).
The next time the Edebugged function gets called, Emacs will display
its source code and will take you through it step-by-step. Press
n to continue to the next step or
c to stop fooling around and skip to the end.
M-x find-function RET package-menu--generate RET
C-u C-M-x
M-x paradox-list-packages RET
Once a function has been instrumented, it will trigger Edebug every time it is
called. To undo this effect (to remove instrumentation) simply visit its
definition again and hit C-M-x without the prefix.q
Magnar Sveen's blog, What the emacs.d?!, was the greatest inspiration behind
Endless Parentheses, and these 5 extra minutes of content from him cheered up my
entire day.
It doesn't take a psychic to guess the <kbd> tag will be useful when
writing about Emacs. When Jorge Navarro asked for the best way to do this, over
at Emacs.SE, I thought I'd share the snippet I use.
This simple command will ask for a key sequence (just like C-h
k would) and will insert it for you. If you want to write the
sequence manually, just hit RET when prompted. C-c k feels
like a nice binding for it.
(eval-after-load'ox-html;; If you prefer to use ~ for <code> tags. Replace "code" with;; "verbatim" here, and replace "~" with "=" below.'(push'(code."<kbd>%s</kbd>")org-html-text-markup-alist))(define-keyorg-mode-map"\C-ck"#'endless/insert-key)(defunendless/insert-key(key)"Ask for a key then insert its description.
Will work on both org-mode and any mode that accepts plain html."(interactive"kType key sequence: ")(let*((orgp(derived-mode-p'org-mode))(tag(iforgp"~%s~""<kbd>%s</kbd>")))(if(null(equalkey"\C-m"))(insert(formattag(help-key-descriptionkeynil)));; If you just hit RET.(insert(formattag""))(forward-char(iforgp-1-6)))))
It should work in both org-mode and html-like modes.
Update <2014-11-30 Sun>
A big kudos to u/abo-abo, for suggesting the use of ~.
Since the previous post on Emacs 25 has been well received, I've decided to
start a series on the topic. It's time to get off the rocking chair, step off
the porch and onto the sidewalk, and start walking towards that ominous “25” on
the horizon. Unlike the birthday post, we're not here to discuss possibilities,
but to meet and greet every new feature that's already implemented in Emacs
master.
This time, we stick to the style of the blog. Short posts listing a single
feature, or two at most. It's going to be a long walk, so we might as well take
it one step at a time.
When [in] query-replace […], typing M-p will now show
previous replacements as "FROM SEP TO", where FROM and TO are the original text
and its replacement, and SEP is an arrow string defined by the new variable
query-replace-from-to-separator. […]
This is actually rather cute.
The arrow is intangible, so it really doesn't get in the way. You can edit
either sides of the prompt, switch them around for a kind of “undo” effect, and
you can even delete the arrow to turn the prompt into a regular FROM prompt.
If you keep cycling back through all the previous replacements, you will
eventually be offered replacements in the old format (just TO and then just
FROM).
The sharp quote (or function quote, or simply #') is an abbreviation
for the function form. It is essentially a version of quote (or ')
which enables byte-compilation, but its actual usefulness has changed
throughout the years.
A little over two decades ago, it was used to quote lambda forms.
You see, '(lambda (x) (* x x)) was just a raw list to the
byte-compiler, but #'(lambda (x) (* x x)) was an actual function
that could be compiled. Now-a-days—or rather, now-a-decades—the
lambda form sharp-quotes itself, meaning a plain (lambda (x) (* x
x))is identical to the #' version. In fact, you should never
quote your lambdas with either quotes.
On the other hand, just as you'd expect the sharp quote to become
redundant for the elisp programmer, a new use arises for it. The
compiler throws a warning whenever it notices you've used an undefined
function, say (not-defined "oops"), but it can't do the same for
something like (mapcar 'not-defined some-list) because it doesn't
know that symbol is the name of a function. The sharp quote is a way
of conveying that information to the compiler, so if it runs into
(mapcar #'not-defined some-list), it can throw a warning
accordingly.
So it is always good practice to sharp quote every symbol that is the
name of a function, whether it's going into a mapcar, an apply, a
funcall, or anything else. Adhering to this actually unearthed a
small bug in one of my packages.
And of course, we can make things more convenient.
(defunendless/sharp()"Insert #' unless in a string or comment."(interactive)(call-interactively#'self-insert-command)(let((ppss(syntax-ppss)))(unless(or(eltppss3)(eltppss4)(eq(char-after)?'))(insert"'"))))(define-keyemacs-lisp-mode-map"#"#'endless/sharp)
thoughtbot NYC
1st floor of the WeWork at Bryant Park
54 W. 40th St.
New York, NY
Happy Holidays, folks! Let’s get paranoid!
This month we’re focusing on security. George Brocklehurst will be
giving an introductory talk/workshop on PGP, and afterward we’ll be
hosting a key signing party! As ever, thoughtbot will be providing
pizza and beer.
George will be introducing PGP: creating a key, uploading it, signing
keys, the web of trust, and sending encrypted mail.
We may also have a brief lightning talk on integrating PGP with Gnus.
For the key signing party, please remember to bring:
A couple forms of photo identification
Pencil and paper, a laptop, or a camera, to record others’ key
fingerprints
Your key’s fingerprint, if you’ve already set one up
Don’t worry if you don’t already have a key; George’s talk should get
you all set up. If you’d like to create your key and try out PGP
beforehand, though, check out Caleb Thompson’s terrific blog post,
PGP and You.
Finally, if you’re on meetup.com, please remember to RSVP! The
building’s security is a little paranoid, and everything will be much
easier if your name is on the guest list.
The amazing Emacs org-mode has myriad uses and features including
outlining, note-taking, table management, and publishing to HTML and
LaTeX. It’s also a fabulous tool to help make your research
reproducible. From Wikipedia:
Org-mode allows authors to include their datasets and all programs
that were run to generate figures and text. Included files can be
used to facilitate version control. While there are limits to this
approach—including data generation and software versioning—it
represents a big advance in reproducibility and one which has not
yet been widely adopted.
capitalize-word and downcase-word are godsends. Like many other
commands, they've spoiled me to the point that I can no longer write
prose without Emacs. There's just one peeve that bothers me quite often.
Let's say I want to join the following two sentences, and point is at the “B”.
Languages are fleeting. But Emacs is forever.
You could do DEL DEL , M-l, but that's so long! When I
downcase the word “But”, it's perfectly obvious that I'll want to get rid of
that full stop, so why can't M-l do that for me?
(global-set-key"\M-c"'endless/capitalize)(global-set-key"\M-l"'endless/downcase)(global-set-key"\M-u"'endless/upcase)(defunendless/convert-punctuation(rgrp)"Look for regexp RG around point, and replace with RP.
Only applies to text-mode."(let((f"\\(%s\\)\\(%s\\)")(space"?:[[:blank:]\n\r]*"));; We obviously don't want to do this in prog-mode.(if(and(derived-mode-p'text-mode)(or(looking-at(formatfspacerg))(looking-back(formatfrgspace))))(replace-matchrpnilnilnil1))))(defunendless/capitalize()"Capitalize region or word.
Also converts commas to full stops, and kills
extraneous space at beginning of line."(interactive)(endless/convert-punctuation","".")(if(use-region-p)(call-interactively'capitalize-region);; A single space at the start of a line:(when(looking-at"^\\s-\\b");; get rid of it!(delete-char1))(call-interactively'subword-capitalize)))(defunendless/downcase()"Downcase region or word.
Also converts full stops to commas."(interactive)(endless/convert-punctuation"\\."",")(if(use-region-p)(call-interactively'downcase-region)(call-interactively'subword-downcase)))(defunendless/upcase()"Upcase region or word."(interactive)(if(use-region-p)(call-interactively'upcase-region)(call-interactively'subword-upcase)))
The snippet above will automatically convert between commas and full-stops when
you're (un)capitalizing prose. It comes up on every single writing session for
me.
Emacs turned 24.4 this Monday. Admittedly, it's not a hugely significant number,
but every birthday is a big deal. The cries of joy and celebrationechoedthroughout the websphere. And now, while Emacs sleeps off an alcohol-induced
headache, as the janitor sweeps off the inch-thick confetti covering the floor,
we gather ourselves and look forward. Just before the horizon, a prominently
ominous, silvery, unavoidable “25” is within sight. And it brings changes.
Last week, I exposed my favorite perks about the version that is now upon us.
Today, I bring to you my expectations. Features I'd be overjoyed to find in the
“News” file. Some of these are guaranteed, others are hopeful ramblings, but
they're all at least implementable. Most importantly, they all have something in
common: their absence, in my opinion, holds back the Emacs community from
growing even faster.
Git
No, I'm not talking about git support, anyone knows Emacs is better at git than
git itself. I'm referring to moving the codebase from bzr to git. At the
moment, this is planned to happen on November 11, and a cheerful day that will be.
The bzr version control system is dying; by most measures it is already
moribund. The dev list has flat-lined, most of Canonical's in-house projects have
abandoned bzr for git, and one of its senior developers has written a remarkably
candid
assessment of why bzr failed[.]
I'm not an expert on bzr, but I've noticed it discourages some new contributors.
Ultimately, adopting a system people already tend to be familiar with is a big
step in the right direction.
Dynamic library loading
Dynamically loading libraries is a big deal, and it's closer than ever. Some
very inspiring results has been reported in the last month and, while there are
still some obstacles under discussion, it's looking almost in shape for release.
Of course, not every package needs this—in fact, the vast majority doesn't.
But this would be a huge boon to a number of packages which, so far, have had to
resort to sub-processes or resign to elisp.
Concurrency in Elisp
This is not a new idea in any way but it's become increasingly urgent with the
growth of the package repositories. I'll be honest, I haven't seen much talk
about this feature. Still, sometime next year, if there's no sign whatsoever of
concurrency being a thing, I'll write a hack for it work with package.el.
Here's why I think that's important.
Consider this situation a little more than a year ago. I noticed my
laptop was taking a while to display the *Packages* buffer.
“No doubt,” I thought to myself, “1477 packages is a bit much for this
laptop. No worries though, 5 seconds of waiting isn't that bad.”.
Fast-forward fifteen months. I invoke M-xlist-packages,
wait 20 seconds, and the buffer I'm given crosses the 3000 lines threshold.
That's partially aggravated by the fact that package.el displays separate
entries for each repository a package can be found in. But even if we eliminate
these redundancies, since July 2013 one thousand unique new packages have been
released. That is an impressive rate of over 2 new packages every single day.
Now consider that this rate shows absolutely no sign of diminishing. Worse yet,
consider that the rate in which packages are updated on Melpa is almost two
orders of magnitude higher. Where will we be by the time Emacs 25 is released?
Gradually and surely, the time it takes to upgrade one's Emacs packages will
surpass that of updating one's Operating System. And what are we to do in the
mean time? Play Minesweeper?
We need asynchronicity, now more than ever, because the *Packages* buffer
plays one of the most fundamental roles in the expansion of the community. We
need a package manager that checks for new releases in the background while we
navigate the previously cached list, and one that performs updates in the
background without forcing us to go for a coffee.
A Better Package Menu
This is an extension to the previous topic. Sluggishness is not the only
consequence of an ever-increasing package menu. Navigating the menu in its
current form is akin to exploring the Amazon rain-forest. There's plenty of
dazzling richness to be found, but the sheer size and roughness are
overwhelming—not mention the piranhas.
Emacs 24.4 introduces keyword filtering to the menu, and Paradox offers a couple
other forms of filtering, but they're still very crude. We need a navigable
tree of categories—which would to facilitate discovery for times when
isearch is a bit too narrow. And we need a simple and honest search box, even
if its only purpose is to invoke something similar to occur.
A More Robust Customize Interface
The customize interface is powerful and beautiful. Its major achievement, in my
opinion, is to make it Emacs customization accessible even for those who know no
elisp at all. When correctly configured, I could see my grandmother using it.
Still, it is for that same reason that we must not admit it to have recurring
bugs. A sizeable part of its user base might never be aware that they're running
into a bug, they'll just think “Emacs is complicated”. So it doesn't help, you
see, that it has a couple of issues so deep-seated that even experienced lispers
might be led to think they're doing something wrong.
I could go on about this for several posts. Plain and simple: Emacs doesn't
have namespaces, so we have to prepend the package name or prefix to every
single defined symbol. Here's a snippet of what that looks like:
I'm sure you see how that's annoying to write and read. Not to mention every
symbol is permanently available in the global namespace.
There have been talks of implementing a system similar to common-lisp packages,
which takes a lot of adaptation to how symbols are interned. While we wait, I
wrote a namespacing engine in the form of a macro. Which turns the above into
something like this.
One way or the other, some form of solution should make its way into Emacs. And
I'll fight for that to happen before 25.
Conclusion
That's my list of most immediate improvements. Did I miss something big? Do you
feel insulted and disagree aggressively, or do you feel enlightened and praise
such accuracy?
aggressive-indent is quite something. I’ve only just released it and it seems
to have been very well received. As such, it’s only fair that I invest a bit more
time into it.
The original version was really just a hack that was born as an answer on
Emacs.SE. It worked phenomenally well on emacs-lisp-mode (to my delight), but
it lagged a bit on c-like modes.
The new version, which is already on Melpa, is much smarter and more optimised.
It should work quite well on any mode where automatic indentation makes sense
(python users, voice your suggestions).
For more than a decade now, Emacs has come with a built-in Tetris
clone, originally written by XEmacs’ Glynn Clements. Just run M-x
tetris any time you want to play. For anyone too busy to waste time
playing Tetris, earlier this year I wrote an autotetris-mode that will
play the Emacs game automatically.
Load the source, autotetris-mode.el and M-x autotetris. It will
start the built-in Tetris but make all the moves itself. It works best
when byte compiled.
At the time I had read an article and was interested in trying
my hand at my own Tetris AI. Like most things Emacs, the built-in
Tetris game is very hackable. It’s also pretty simple and easy to
understand. Rather than write my own I chose to build upon this one.
Heuristics
It’s not a particularly strong AI. It doesn’t pay attention to the
next piece in queue, it doesn’t know the game’s basic shapes, and it
doesn’t try to maximize the score (clearing multiple rows at once).
The goal is to continue running for as long as possible. But since
it’s able to get to the point where the game is so fast that the AI is
unable to move pieces fast enough (it’s rate limited like a human
player), that means it’s good enough.
When a new piece appears at the top of the screen, the AI, in memory,
tries placing it in all possible positions and all possible
orientations. For each of these positions it runs a heuristic on the
resulting game state, summing five metrics. Each metric is scaled by a
hand-tuned weight to adjust its relative priority. Smaller is better,
so the position with the lowest score is selected.
Number of Holes
A hole is any open space that has a solid block above it, even if that
hole is accessible without passing through a solid block. Count these
holes.
Maximum Height
Add the height of the tallest column. Column height includes any holes
in the column. The game ends when a column touches the top of the
screen (or something like that), so this should be kept in check.
Mean Height
Add the mean height of all columns. The higher this is, the closer we
are to losing the game. Since each row will have at least one hole,
this will be a similar measure to the hole count.
Height Disparity
Add the difference between the shortest column height and the tallest
column height. If this number is large it means we’re not making
effective use of the playing area. It also discourages the AI from
getting into that annoying situation we all remember: when you
really need a 4x1 piece that never seems to come. Those are the
brief moments when I truly believe the version I’m playing has to be
rigged.
Surface Roughness
Take the root mean square of the column heights. A rougher surface
leaves fewer options when placing pieces. This measure will be similar
to the disparity measurement.
Emacs-specific Details
With a position selected, the AI sends player inputs at a limited rate
to the game itself, moving the piece into place. This is done by
calling tetris-move-right, tetris-move-left, and
tetris-rotate-next, which, in the normal game, are bound to the
arrow keys.
The built-in tetris-mode isn’t quite designed for this kind of
extension, so it needs a little bit of help. I defined two pieces of
advice to create hooks. These hooks alert my AI to two specific events
in the game: the game start and a fresh, new piece.
I talked before about the problems with global state.
Fortunately, tetris-mode doesn’t store any game state in global
variables. It stores everything in buffer-local variables, which can
be exploited for use in the AI. To perform the “in memory” heuristic
checks, it creates a copy of the game state and manipulates the copy.
The copy is made by way of clone-buffer on the *Tetris* buffer.
The tetris-mode functions all work equally as well on the clone, so I
can use the existing game rules to properly place the next piece in
each available position. The game’s own rules take care of clearing
rows and checking for collisions for me. I wrote an
autotetris-save-excursion function to handle the messy details.
(defmacroautotetris-save-excursion(&restbody)"Restore tetris game state after BODY completes."(declare(indentdefun))`(with-current-buffertetris-buffer-name(let((autotetris-saved(clone-buffer"*Tetris-saved*")))(unwind-protect(with-current-bufferautotetris-saved(kill-local-variable'kill-buffer-hook),@body)(kill-bufferautotetris-saved)))))
The kill-buffer-hook variable is also cloned, but I don’t want
tetris-mode to respond to the clone being killed, so I clear out the
hook.
That’s basically all there is to it! While watching it feels like it’s
making dumb mistakes, not placing pieces in optimal positions, but it
recovers well from these situations almost every time, so it must know
what it’s doing. Currently it’s a better player than me, which is my
rule-of-thumb for calling an AI successful.
I won’t repeat myself on the usefulness of prefix arguments, though it
would hardly be an overstatement. Killing 7 lines of text in two
keystrokes is a bliss most people will never know.
Today's post was prompted by this question on Emacs Stack Exchange. Itsjeyd has
grown tired of using 3 keystrokes to kill a line (C-a C-k
C-k) and asks for an alternative. The straightforward answer is to use
kill-whole-line instead, but then you either need another keybind,
C-S-backspace, or you need to lose the regular kill-line
functionality.
The solution I've found for myself is to use prefix arguments.
You see, killing half a line is a useful feature, but slaughtering three and a
half lines make very little sense. So it stands to reason to have kill-line
meticulously murder everything in its sight when given a prefix argument.
(defmacrobol-with-prefix(function)"Define a new function which calls FUNCTION.
Except it moves to beginning of line before calling FUNCTION when
called with a prefix argument. The FUNCTION still receives the
prefix argument."(let((name(intern(format"endless/%s-BOL"function))))`(progn(defun,name(p),(format"Call `%s', but move to BOL when called with a prefix argument."function)(interactive"P")(whenp(forward-line0))(call-interactively',function))',name)))
With this little macro, C-k still kills from point, but
C-3 C-k swallows three whole lines. As a bonus, we get the
kill-whole-line behavior by doing C-1 C-k.
Are there any other functions which might benefit from this macro?
Our final post of the series starts with a sober note, but swiftly
moves to a happy ending. Just as we cherish each improvement to our
favorite packages, so must we honor the dead who served us and give
the newborn a chance to thrive. These are the packages marked
obsolete, followed by the new packages you didn’t even know you wanted.
Obsolete packages
First, some short comments on the obituary.
longlines.el; use visual-line-mode.
This is sad, but don’t weep yet. I have a feeling a dead will rise
from this grave on Emacs 25 (stay tuned).
iswitchb.el; use icomplete-mode.
This news flowed around the Emacsphere some months ago. Between
icomplete-mode and ido-mode, iswitchb was no longer necessary.
terminal.el; use term.el instead.
They were redundant, and now they’re identical.
meese.el.
sup-mouse.el.
the old version of todo-mode.el (renamed to otodo-mode.el).
xesam.el (owing to the cancellation of the XESAM project).
yow.el; use fortune.el or cookie1.el instead.
Five packages I’ve never ever ever heard of. But I’m sure somebody
will miss them.
New Modes and Packages in Emacs 24.4
Now, the interesting part.
New package eww.el provides a built-in web browser.
This requires Emacs to have been compiled with libxml2 support.
I remember a lot of talk on eww’s speed and visual accuracy. It’s
not always that I salute a package whose role is already filled by
several others, but this one looks promising.
New package nadvice.el offers lighter-weight advice facilities.
It is layered as: add-function and remove-function, which can be used to add/remove code on
any function-carrying place, such as process filters or <foo>-function hooks. advice-add and advice-remove to add/remove a piece of advice on a named
function, much like defadvice does.
Perhaps the advice interface needed a light-weight alternative. I
never really noticed, and this package (while nice) feels rather
underwhelming. Here’s a usage example.
New package frameset.el provides a set of operations to save a frameset
(the state of all or a subset of the existing frames and windows, somewhat
similar to a frame configuration), both in-session and persistently, and
restore it at some point in the future.
Now we’re talking! Can you believe Emacs had no way of saving
window/frame configurations between sessions? I ran into this problem
before, and so did this user at Emacs.SO. Before, all you could do was
save it within a single session.
Thanks to frameset.el, not only can you save configurations
manually, but desktop-save-mode hooks into that automatically!
New package filenotify.el provides an interface for file system
notifications. It requires that Emacs be compiled with one of the
low-level libraries gfilenotify.c, inotify.c or w32notify.c.
I can imagine some nice uses for this, but we’ll have to see what
comes of it. For what it’s worth, auto-revert-mode already makes use
of this interface when it’s available, making it much less demanding.
New minor mode superword-mode, which overrides the default word motion
commands to treat "symbol_words" as a single word, similar to what
subword-mode does.
Bozhidar has more to say on this than I do. I’m a subword-mode
worshipper, so I won’t be using this mode.
New minor modes prettify-symbols-mode and global-prettify-symbols-mode
display specified symbols as composed characters. E.g., in Emacs Lisp mode,
this replaces the string "lambda" with the Greek lambda character.
And we go off with a bang! By default, in emacs-lisp-mode, this will
only turn lambda into λ, but that’s enough reason to turn it on.
Following yesterday's list of the Sweetest New Features in 24.4, today
I go through the ones that strike me as most useful, either to me or to
the Emacs environment itself. This is the part where I had to cut the
most. Emacs 24.4 exhibits a downpour of usability improvements, and it
almost feels unjust to not list them all. Nonetheless, I've narrowed
them down to 8.
The commands eval-expression (M-:), eval-last-sexp (C-x C-e),
and eval-print-last-sexp (C-j in Lisp Interaction mode) can take a
zero prefix argument. This disables truncation of lists in the output,
equivalent to setting (eval-expression-)print-length and
(eval-expression-)print-level to nil. Additionally, it causes integers
to be printed in other formats (octal, hexadecimal, and character).
Fantastic! The default behaviour of truncating lists proves frustrating
every so often. My solution had been to set print-length to nil, but
that's also not ideal as you don't always want to see everything either.
Being able to quickly toggle between the two is just perfect.
This used to be a very popular third party extension for dired. It's
always nice to see good ideas getting promoted to built-in.
More packages look for ~/.emacs.d/<foo> additionally to ~/.<foo>.
I won't list each of the affected files here, suffice to say that they
add up to 17 and include some very popular packages, such as places
and ido. You could always do this yourself by configuring a score of
different variables, but it's nice to see Emacs taking initiative
against home clutter.
New macro define-alternatives can be used to define generic commands.
I had to write up some code to see what this was about and it
certainly piqued my interest. define-alternatives is a command which
defines a function with several possible implementations. The user is
then asked to choose the implementation he prefers upon invoking the command
for the first time.
For instance, say I'm writing up a major-mode for the Julia language.
I want to offer a compilation command, but I don't known whether the
user will prefer synchronous or asynchronous compilation. So I use
define alternatives.
(define-alternativesjulia-mode-compile;; These keywords are used on the defcustom;; for `julia-mode-compile-alternatives'.:groupjulia-mode:version"1.0")(setqjulia-mode-compile-alternatives'(("Compile synchronously".julia-mode-compile-synchronously)("Compile asynchronously".julia-mode-compile-asynchronously)))
The first time the user invokes M-xjulia-mode-compile, they will be asked to choose an implementation.
At last, Emacs 24.4 is getting released next Monday (the 20th). To
commemorate, I took a stroll through the “News” file and picked a
few new features that spoke to me the most. I had to cut dozens of
interesting items for it to fit the format of this blog. And I still
had to divide it into 3 parts.
Today, I’ll just list my reaction to some sweet new features. Worthy
of mention: back in December 2013 Mickey made a huge post also listing
new features, and Bozhidar has been doing a series of great posts on
the subject.
Favourite New Features
New [Tramp] connection method "adb", which allows to access Android.
Woohoo!
New hook eval-expression-minibuffer-setup-hook run by eval-expression on entering the minibuffer.
You can enable ElDoc inside the eval-expression minibuffer.
This is awesome! Try it now! Evaluate the following, then call
M-: and type something like (message.
The Messages buffer is created in messages-buffer-mode, a new
major mode, with read-only status. Any code that might create the
Messages buffer should call the function messages-buffer to do so
and set up the mode.
New option load-prefer-newer affects how the load function chooses the file to load.
Hopefully, someday, t will be the default value of this. For the
moment, make sure you add (setq load-prefer-newer t) so you’re never
accidentally using outdated compiled files.
New library subr-x.el with miscellaneous small utility functions: hash-table-keys, hash-table-values, string-blank-p, string-empty-p, string-join, string-reverse, string-trim-left, string-trim-right, string-trim, string-remove-prefix, string-remove-suffix
Get in the habit of using these functions. They greatly reduce the need
for the s- package, not that there’s anything wrong with it but less
package dependencies lead to a cleaner environment.
When you have more than one browser installed, you must choose which
one's the default. The OS will then proceed to open all URLs in the
default browser, even if it happens to be closed while another one
happens to be running.
At least in Emacs we can fix that behavior. The following code
configures Emacs to use whatever browser happens to be open right now,
instead of always defaulting to the same one.
;;; We don't need to `require' because this function is only;;; ever called by `browse-url' itself. But if you have;;; problems, try uncommenting this:;; (require 'browse-url)(defunendless/browse-url-best-browser(url&rest_)"Use a running browser or start the preferred one."(setqurl(browse-url-encode-urlurl))(let((process-environment(browse-url-process-environment))(command(endless/decide-browser)))(start-process(concatcommand" "url)nilcommandurl)));; Use this for all links. If you actually don't want some;; links to be viewed externally, change this line here.(setqbrowse-url-browser-function'((".".endless/browse-url-best-browser)))
Here we configure our preferences through the endless/browser-list
variable. Its value is a list of cons cells, each representing a
browser. The car is a regexp to match the name of the browser's
process (used to determine whether the browser is running), and the cdr
is the name of the executable.
(defcustomendless/browser-list'(("xulrunner\\|conkeror"."conkeror.sh")("xulrunner\\|conkeror"."conkeror")"conkeror.exe";; This works if it's in your $PATH"luakit"("chrome$"."google-chrome-stable")"chromium""chromium-browser"("firefox\\|mozilla"."firefox");; This works regardless of your $PATH("firefox.exe"."c:/Program Files (x86)/Mozilla Firefox/firefox.exe"))"List of browsers by order of preference.
Each element is a cons cell (REGEXP . EXEC-FILE).
If REGEXP matches the name of a currently running process and if
EXEC-FILE a valid executable, EXEC-FILE will be used to open the
given URL.
An element can also be a string, in this case, it is used as both
the REGEXP and the EXEC-FILE.
It is safe to have items referring to not-installed browsers,
they are gracefully ignored.":type'(repeat(choicestring(consregexpfile))))
And this is the function responsible for checking the running
processes and finding a browser in there.
(defunendless/decide-browser()"Decide best browser to use based on `endless/browser-list'."(let((process-list(mapcar#'endless/process-name(list-system-processes)))(browser-listendless/browser-list)browserout);; Find the first browser on the list that is open.(while(andbrowser-list(nullout))(setqbrowser(carbrowser-list))(if(and(cl-member(car-or-selfbrowser)process-list:test'string-match)(executable-find(cdr-or-selfbrowser)))(setqout(cdr-or-selfbrowser))(setqbrowser-list(cdrbrowser-list))));; Use the one we found, or the first one available.(orout(endless/first-existing-browser))))(defunendless/first-existing-browser()"Return the first installed browser in `endless/browser-list'."(require'cl-lib)(cdr-or-self(car(cl-member-if(lambda(x)(executable-find(cdr-or-selfx)))endless/browser-list))))(defunendless/process-name(proc)(cdr(assoc'comm(process-attributesproc))))(defuncar-or-self(x)"If X is a list, return the car. Otherwise, return X."(or(car-safex)x))(defuncdr-or-self(x)"If X is a list, return the cdr. Otherwise, return X."(or(cdr-safex)x))
I've tested it on Windows and a couple of Linux distros. Could a Mac
user test it for me as well?
WeWork NoMad
3rd floor
79 Madison Ave.
New York, NY 10016
As usual, we’ll be starting at 6:30 with pizza and beer.
Evan Misshula will be presenting Org-mode for Reproducible Research:
The amazing Emacs org-mode has myriad uses and features including
outlining, note-taking, table management, and publishing to HTML and
LaTeX. It’s also a fabulous tool to help make your research
reproducible. From Wikipedia:
Org-mode allows authors to include their datasets and all programs
that were run to generate figures and text. Included files can be
used to facilitate version control. While there are limits to this
approach—including data generation and software versioning—it
represents a big advance in reproducibility and one which has not
yet been widely adopted.
The site goes well and the curtains are raised. The
Emacs.StackExchange beta has left the privacy of its dressing room and
ventures onto the public stage. Even if you have nothing in your mind to
ask, go pay it a visit and have a browse through the questions. The
quality of content is impressive, and you’re sure to learn something
of interest.
I’ve already brought here several lessons I would never have learnt if
not for the beta. Smart Dired, Longlines mode, and Aggressive
Auto-indentation are now essential parts of my init file. Of course,
I’ll keep bringing more, they come up faster than I can write about them.
Not to mention, the community is friendly and intelligent. Not that
it’s any surprise in the Emacs ecosystem.
Keyboard macros are a powerful Emacs feature. A keyboard macro is
simply a recording of a sequence of key sequences that can be played
back and repeated. In this meetup we will first go over the
different keyboard macro commands Emacs has to offer. In the second
half of the meetup we will solve a couple repetitive text
manipulation tasks using macros.
If you’d like to get some directed practice using keyboard macros,
Jacob hosts a collection of sample exercises. We also mirror those
exercises here in .zip or .tar.gz formats, if you’d prefer.
The launcher-map wouldn't be half of what it is if not for the run
macro. With it, we can easily turn Emacs into a quick 'n dirty app
launcher.
(defmacrorun(exec)"Return a named function that runs EXEC."(let((func-name(intern(concat"endless/run-"exec))))`(progn(defun,func-name(),(format"Run the %s executable."exec)(interactive)(start-process""nil,exec))#',func-name)))(define-keylauncher-map"m"(run"Mathematica"))(define-keylauncher-map"k"(run"keepass"))(define-keylauncher-map"v"(run"steam")); Vapor =P
We could use a lambda instead of the macro, but the macro defines a
nicely documented function.
While we're at it, let's do something similar for websites.
(defmacrobrowse(url)"Return a named function that calls `browse-url' on URL."(let((func-name(intern(concat"endless/browse-"url))))`(progn(defun,func-name(),(format"Browse to the url %s."url)(interactive)(browse-url,url))#',func-name)))(define-keylauncher-map"t"(browse"http://twitter.com/"))(define-keylauncher-map"?";; See also SX.el.(browse"http://emacs.stackexchange.com/"))(define-keylauncher-map"r"(browse"http://www.reddit.com/r/emacs/"))(define-keylauncher-map"w"(browse"http://www.emacswiki.org/"))(define-keylauncher-map"+"(browse"https://plus.google.com/communities/114815898697665598016"))
Emacs.SE has truly revived my init file. Every day, a new snippet gets
added. I started this series only 6 days ago, and it’s already on its
fourth episode.
Today's question was asked by me, and answered by Tikhon Jelvis,
Francesco, and Sacha Chua. As usual, this link might not work for you
until the beta goes public.
Question
I’ve long been keeping my LaTeX documents under version control.
Still, the effectiveness of this initiative would be greatly improved
if I could bring myself to adopt a “one-sentence-per-line” approach.
It would facilitate managing people’s conributions and reverting old
changes.
If you’re wondering what’s so difficult about that, here is what such
a document would look like. (Long sentences are to be avoided, but in
scientific writing you don’t always have a choice).
A quite short sentence here.
New sentence on the same paragraph, with more text with more text with more text with more text.
Some more text, still on the same paragraph.
This is another very very very very very very very very very very very very very very very very very very very very long sentence.
The lack of readability above is evident. Having text extend beyond 80
columns is trouble enough, but intertwining long and short lines just
makes my brain weep in despair.
The official solution is to activate visual-line-mode and add window
margins (or just decrease window width). That causes the lines to wrap
(visually, not in the file) and increases readability by an order of
magnitude.
A quite short sentence here.
New sentence on the same paragraph, with more text with more
text with more text with more text.
Some more text, still on the same paragraph.
This is another very very very very very very very very very
very very very very very very very very very very very long
sentence.
But alas, it comes at a cost. If this text were indented by a few
spaces (quite common in LaTeX), the outcome looks like it went through
a blender. And don’t get me started on broken equations.
A quite short sentence here.
New sentence on the same paragraph, with more text with
more text with more text with more text.
Some more text, still on the same paragraph.
This is another very very very very very very very very
very very very very very very very very very very very very
long sentence.
\begin{equation}
H = equation * lines + sometimes - \need{to}{be /
long}
\end{equation}
Solution
tdsh points out in the comments, the adaptive-wrap package fixes
the indentation issue for visual-line-mode.
The solution reached is a hack to make longlines-mode act more
intelligently on LaTeX buffers. This minor mode is similar to the
combination of visual-line-mode and window margins mentioned
above. It makes lines wrap at fill-column, intead of window-width.
The advantage to longlines-mode is that it’s implemented in elisp
(while window margins are done in C code), so we can hack it to our
hearts’ content. Bafflingly, this little gold nugget has been marked
obsolete on Emacs 24.4.
The hack below does 3 things:
It enables “soft” wrapping of text. That is, the text is wrapped to
fill-column in the buffer, but that does not reflect in the file.
It fixes the indentation problem. So wrapping an indented line follows
the line’s indentation. So environments with indented text (like itemize, or
theorem) are much more readable.
It even prevents wrapping of equations!
The code is a bit heavy, so I’ve added comments describing the lines
which were actually changed.
(require'longlinesnilt)(add-hook'LaTeX-mode-hook#'longlines-mode)(defunlonglines-encode-region(begend&optional_buffer)"Replace each soft newline between BEG and END with exactly one space.
Hard newlines are left intact. The optional argument BUFFER exists for
compatibility with `format-alist', and is ignored."(save-excursion(let((reg-max(maxbegend))(mod(buffer-modified-p)))(goto-char(minbegend));; Changed this line to "swallow" indendation when decoding.(while(search-forward-regexp" *\\(\n\\) *"reg-maxt)(let((pos(match-beginning1)))(unless(get-text-propertypos'hard)(goto-char(match-end0)); This line too(insert-and-inherit" ")(replace-match"":fixedcase:literal); This line too(remove-text-propertiespos(1+pos)'hard))))(set-buffer-modified-pmod)end)))(defunlonglines-wrap-line()"If the current line needs to be wrapped, wrap it and return nil.
If wrapping is performed, point remains on the line. If the line does
not need to be wrapped, move point to the next line and return t."(if(and(bound-and-true-platex-extra-mode)(null(latex/do-auto-fill-p)))(progn(forward-line1)t);; The conditional above was added for latex equations. It relies;; on the latex-extra package (on Melpa).(if(and(longlines-set-breakpoint);; Make sure we don't break comments.(null(nth4(parse-partial-sexp(line-beginning-position)(point)))))(progn;; This `let' and the `when' below add indentation to the;; wrapped line.(let((indent(save-excursion(back-to-indentation)(current-column))))(insert-before-markers-and-inherit?\n)(backward-char1)(delete-char-1)(forward-char1)(when(>indent0)(save-excursion(insert(make-stringindent?)))(setqlonglines-wrap-point(+longlines-wrap-pointindent))))nil)(if(longlines-merge-lines-p)(progn(end-of-line)(if(or(prog1(bolp)(forward-char1))(eolp))(progn(delete-char-1)(if(>longlines-wrap-point(point))(setqlonglines-wrap-point(1-longlines-wrap-point))))(insert-before-markers-and-inherit?\s)(backward-char1)(delete-char-1)(forward-char1);; This removes whitespace added for indentation.(while(eq(char-after)?)(delete-char1)(setqlonglines-wrap-point(1-longlines-wrap-point))))nil)(forward-line1)t))))
Update <2014-10-07 Tue>
Fixed handling of comments. Now comments don’t get wrapped either.
WeWork NoMad
3rd floor
79 Madison Ave.
New York, NY 10016
As usual, we’ll be starting at 6:30 with pizza and beer.
Jacob O’Donnell will be giving a talk
and hosting a workshop on advanced features of keyboard macros. Bring
your laptops!
Keyboard macros are a powerful Emacs feature. A keyboard macro is
simply a recording of a sequence of key sequences that can be played
back and repeated. In this meetup we will first go over the
different keyboard macro commands Emacs has to offer. In the second
half of the meetup we will solve a couple repetitive text
manipulation tasks using macros.
electric-indent-mode is enough to keep your code nicely
aligned when all you do is type. However, once you start shifting
blocks around, transposing lines, or slurping and barfing sexps,
indentation is bound to go wrong.
So long as you have Melpa configured, you can install it with.
M-x package-install RET aggressive-indent
Then simply turn it on and you’ll never have unindented code again.
(global-aggressive-indent-mode)
This will activate aggressive-indent-mode on every non-text buffer. If you're
a little shy, you can also turn it on only for specific major modes by using
hooks.
Here’s some code to try it on. After you’ve evaluated the above, open
a new “.el” file, paste the following code, and type something
before the opening parentheses.
Kaushalmodi uses #+INCLUDE: statements with line specifications in his
org files. Here is an example similar to his. 14 and 80 are the first
and last line of a class declaration in the source file, so they’re
quite obvious for a human to identify.
The problem here is that whenever “my-class.sv” is edited those
line numbers are likely to become outdated. So you would have to go
through each org file which might include “my-class.sv” and update
the numbers.
The Solution
Unfortunately, org-mode doesn’t have a flexible way of declaring
include statements. You either specify the line numbers or you don’t.
The solution was to add :range-begin and :range-end keywords to
the statement
goes through each #+INCLUDE: statement in the buffer,
checks if it has :range-begin and/or :range-end keywords and
takes their arguments as regular expressions,
visits the relevant file and searches for these regular expressions,
checks what the line numbers are now,
and updates them accordingly in the org buffer.
This function can then be assigned to a key, added to
before-save-hook, or added to one of org-mode’s bajillion available
hooks. Finally, to go the extra mile, we make the behaviour
customizable per file extension through a defcustom.
(add-hook'before-save-hook#'endless/update-includes)(defunendless/update-includes(&restignore)"Update the line numbers of #+INCLUDE:s in current buffer.
Only looks at INCLUDEs that have either :range-begin or :range-end.
This function does nothing if not in org-mode, so you can safely
add it to `before-save-hook'."(interactive)(when(derived-mode-p'org-mode)(save-excursion(goto-char(point-min))(while(search-forward-regexp"^\\s-*#\\+INCLUDE: *\"\\([^\"]+\\)\".*:range-\\(begin\\|end\\)"nil'noerror)(let*((file(expand-file-name(match-string-no-properties1)))linesbeginend)(forward-line0)(when(looking-at"^.*:range-begin *\"\\([^\"]+\\)\"")(setqbegin(match-string-no-properties1)))(when(looking-at"^.*:range-end *\"\\([^\"]+\\)\"")(setqend(match-string-no-properties1)))(setqlines(endless/decide-line-rangefilebeginend))(whenlines(if(looking-at".*:lines *\"\\([-0-9]+\\)\"")(replace-matchlines:fixedcase:literalnil1)(goto-char(line-end-position))(insert" :lines \""lines"\""))))))))(defunendless/decide-line-range(filebeginend)"Visit FILE and decide which lines to include.
BEGIN and END are regexps which define the line range to use."(let(lr)(save-match-data(with-temp-buffer(insert-filefile)(goto-char(point-min))(if(nullbegin)(setql"")(search-forward-regexpbegin)(setql(line-number-at-pos(match-beginning0))))(if(nullend)(setqr"")(search-forward-regexpend)(setqr(1+(line-number-at-pos(match-end0)))))(format"%s-%s"lr)))))
The emacs.stackexchange beta has only just started and interesting
topics have already begun to pop up. Partially to share snippets I
find nifty, and partially to help promote the beta, I’ll be posting
here questions and answers I find engaging.
I’m kicking off this series with a question of my own, answered by none
other than Sebastian Wiesner. (Did you know he has a blog)? The beta
is still private, so you might not be able to visit the links above,
but I’ll summ it up here.
Question
For a while now I’ve noticed my dired patterns are somewhat
predictable.
Whenever I visit the directory of one of my papers, I go straight to
the “master.tex” file.
When I open the root of an android project, it’s almost always to
visit the the “AndroidManifest.xml”.
Finally, when I go to a directory where I’m developing an emacs
package, 80% of the time I’m headed for the package's main source
file.
See the pattern? For all these directories, the first file I visit is
very predictable, so I wanted dired to focus that file automatically
for me. So I would only need to hit RET—lazyness
to the extreme.
Solution
Sebastian looks into the save-place package’s source code, and finds
out about dired-initial-position-hook, which is run exactly when we
need. Visiting a file whose name we already know is then a trivial
matter. Note I’ve edited his code a bit.
(defcustomendless/important-files'("master.tex""AndroidManifest.xml""init.org")"List of files which dired should focus by default.":type'(repeatstring))(defunmy-dired-goto-important-file()"Go to an important file in the current dired buffer."(let((candidatesendless/important-files)(matchednil))(while(andcandidates(nullmatched))(setqmatched(dired-goto-file(expand-file-name(popcandidates)))))(unlessmatched(endless/goto-elisp-file))))(add-hook'dired-initial-position-hook;; Append so we run after `save-place'#'my-dired-goto-important-file'append)
Making dired focus the source file of elisp packages is tad bit
tricker because the file’s name is not fixed. The solution I found
(still borrowing some of Wiesner’s code) was to look for a file whose
name matched the current directory’s name, and if it that doesn’t
exist just focus any “.el” file. This works well for me because my
directory structure typically looks like “~/Git/paradox/paradox.el”.
(defunendless/goto-elisp-file()"Go to a file with .el extension.
If more than one exists, go to the one with the same name as this
directory. This is enough to catch most package source files."(let*((files(endless/dired-file-list))(dirname(file-name-base(directory-file-namedefault-directory)))(target(concatdirname".el")))(unless(dired-goto-file(expand-file-nametarget))(setqtarget(car-safe(cl-member-if(lambda(x)(string-match"\\.el$"x))files)))(whentarget(dired-goto-file(expand-file-nametarget))))))(defunendless/dired-file-list()"List of files in this dired buffer."(save-excursion(let(files)(goto-char(point-min))(while(not(eobp))(let((filename(dired-get-filenamenil'no-error)))(whenfilename(pushfilenamefiles)))(forward-line1))files)))
If you read r/emacs or follow some Emacs bloggers on twitter, you’ll
have noticed a proposal for an Emacs stack exchange site was launched
a couple of weeks ago. The commitment rate of the proposal was nearly
record-breaking, and now it has entered private beta.
If I understand correctly, the private beta will only last a couple of
weeks, but members are allowed to send out invites. So, if you missed
the commitment stage but you’d like to help, drop a comment here.
Even if you don’t think you are expert enough to give good answers,
asking questions and voting is a great help already. The focus during
this stage should be difficult and interesting questions. So don’t
go easy!
This stage sets the tone of the entire website.
For reasons which ellude me, the org-in-src-block-p function is
disappointingly inconsistent for me. Given its major role in
narrow-or-widen-dwim, this frequently led to org-edit-src-code not
getting called when point was inside a code block.
I’ve edited the narrow-or-widen-dwim to not need that function. Go
copy it again if you were using it.
I keep a couple of subdirectories inside my “.emacs.d/” for the
purpose of organization, so rgrep'ing is tremendously useful for
finding where a variable is being changed. By default, that will also
search inside “elpa”/, which slows the search and pollutes the
results page.
We can make grep more selective and, while we're at it, enable line
truncation.
Typography is a long and blurry road, of which I know very little.
There are, still, some simple lessons that take you a long a way.
Round (Unicode) quotation marks is one of them.
LaTeX already does that for you, but many other prose environments
don’t. This snippet inserts “round” quotes for you, bind it to your
text-modes.
(define-keyorg-mode-map"\""#'endless/round-quotes)(eval-after-load'markdown-mode'(define-keymarkdown-mode-map"\""#'endless/round-quotes))(defunendless/round-quotes(italicize)"Insert “” and leave point in the middle.
With prefix argument ITALICIZE, insert /“”/ instead
\(meant for org-mode).
Inside a code-block, just call `self-insert-command'."(interactive"P")(if(and(derived-mode-p'org-mode)(org-in-block-p'("src""latex""html")))(call-interactively#'self-insert-command)(if(looking-at"”[/=_\\*]?")(goto-char(match-end0))(whenitalicize(if(derived-mode-p'markdown-mode)(insert"__")(insert"//"))(forward-char-1))(insert"“”")(forward-char-1))))
Following on our series of mnemonic keymaps, we arrive on the
launcher-map. Where the toggle-map was designed for toggling values
and minor-modes we only use every once in a while, the launcher-map
runs those standalone features of Emacs that also don't always see the
light of day.
To remember these keybinds, think of “Emacs launch calc” for instance.
(define-prefix-command'launcher-map);; `C-x l' is `count-lines-page' by default. If you;; use that, you can try s-l or <C-return>.(define-keyctl-x-map"l"'launcher-map)(global-set-key(kbd"s-l")'launcher-map)(define-keylauncher-map"p"#'paradox-list-packages)(define-keylauncher-map"c"#'calc)(define-keylauncher-map"d"#'ediff-buffers)(define-keylauncher-map"f"#'find-dired)(define-keylauncher-map"g"#'lgrep)(define-keylauncher-map"G"#'rgrep)(define-keylauncher-map"h"#'man); Help(define-keylauncher-map"i"#'package-install-from-buffer)(define-keylauncher-map"n"#'endless/visit-notifications)(define-keylauncher-map"s"#'shell)
calc, man, list-packages… I use these features, on average, once or
twice every other moon. An intuitive keymap is exactly what I need to always
remember their keybinds and not have to M-x their whole name.
Update <30 Nov 2015>
Changed the n key from nethack to endless/visit-notifications, as per the
new post.
On today's episode, we again find ourselves admiring the cl package
and focus on a very little-known feature of Emacs, psetq. Short for
“parallel-setq”, it takes the same syntax as setq, but saves all
values before assigning.
For instance, if one needs to swap the variables old and new,
assuming you've required cl-lib, you can simply do the following.
(cl-psetqoldnewnewold)
And, of course, it has also been generalized to psetf for assigning
generalized places.
Update <2014-09-09 Tue>
Truth be told, psetq is more useful when your demands are more
sophisticated. When all you need is to swap two variables (or places),
the nice and quick solution is cl-rotatef.
While the official elisp reference does contain some style tips, it's
hard to deny that the community needed a more comprehensive and (at
the same time) concise style guide. Mostly, new Emacs developers have
been left to learn the ropes by reading other source files. As practical
as that might be, it's dangerous in its propensity to propagate bad
habits.
To medicate this, Bozhidar Batsov, whose blog I'm sure you follow,
puts up a community-driven style guide for Emacs lisp.
If you write any sort of Emacs package, make sure you pay it a visit
and see if you can think of anything that's missing. I'm doing my part
over there, let's get that plane off the ground.
Once you've come to terms with power of setf, it is time to meet its
older sister, cl-letf. As the name implies, letf is to setf like
let is to setq, but, once again, that is only the tip of the
iceberg.
To get started, let's have a variable and require the feature.
(require'cl-lib)(setqmy-list'(012345))
When temporarily assigning a value to a variable, letf is (mostly)
identical to let.
(cl-letf((my-list"not actually a list"))(message"%s"my-list));; ==> "not actually a list"
They differs slightly in behaviour when you don't provide a value.
let binds the variable to nil and restores it upon exit (as I'm sure
you know), while letf preserves the current value and restores it
upon exit.
Of course, that's not where letf shines. It's when you use place
expressions that it simply blows let out of the water. Let's take a
simple example using our list.
If you're having trouble reading past the three consecutive
parentheses, this snippet simply takes our list and temporarily
changes its first and fourth elements to 1000 and 200,
respectively.
“Alright, but how often is that actually useful?”
It all depends on your creativity. This stackoverflow question
highlights a common issue that came up with the release of Emacs 24.3.
flet (short for function-let) was a macro which locally (and
dynamically) replaced the function definition associated to given
symbols. This is extremely useful for error testing, but if you try to
use this macro now you'll get following the message.
Warning: `flet' is an obsolete macro (as of 24.3); use either `cl-flet' or `cl-letf'.
Unfortunately, cl-flet is not identical to the original
flet—it's lexical, not dynamic.
For instance, the url-retrieve-synchronously usually prints a
message on the echo area. Now say you want to prevent that, you can
temporarily rebind the message function to do nothing.
Now do M-x silent-retrieve, and see that it works.
Unfortunately, if you try to be a good coder and replace the obsolete
flet with the recommended cl-flet, it won't work!
letf as a replacement for flet
The solution I usually see floating around is to employ Nic Ferrier's
fantastic noflet package. But I've never seen anyone mention the
built-in option, even though Emacs itself tells you to use it: “use
either `cl-flet' or `cl-letf'”.
You see, (symbol-function SYMBOL) is a valid place expression. Which
means you can bind it dynamically using cl-letf. Evaluate the
following defun and call M-x new-silent-retrieve.
It works! No messaging! All that we've done was tell letf to
temporarily replace the message function with the format function
(which does the same thing without echoing). We could also use
ignore instead of format (the latter just happens to have the same
return value as message).
setf is a modest name for a macro that does much more work than it
gets paid for. Quoting the doc page:
This is a generalized version of `setq'; the PLACEs may be symbolic
references such as (car x) or (aref x i), as well as plain symbols.
For example, (setf (cadr x) y) is equivalent to (setcar (cdr x) y).
The return value is the last VAL in the list.
Upon reading that, you'd be excused for thinking something mediocre
such as “That's neat!” or “I'll try to remember that”. That
understatement of a doc page merely touches on the power that lies
beneath the surface.
Let's start with an example from the manual, depending on your Emacs
version you may need to (require 'cl) first.
(let((world"world"))(setf(substringworld24)"o"));; world is now "wood"
Now say you want to replace the entire contents of a buffer, instead
of the manual erase-buffer then insert, you could do
(setf(buffer-string)"replacement")
Let's get slightly more practical. Do you know which function changes
the buffer being displayed by a given window? How about changing the
height of a window? You don't need to!
Have you ever stopped to consider what is the string you type most
often?
In my case the answer is two-fold: “~/” and “.emacs.d/”.
Minibuffer completion helps, but we can always make things better.
The following snippet intelligently inserts both of these with a
single key, and it's smart enough to know which one you want.
Besides its use in writing configurations, it's great for going
straight to one of these directories inside the find-file prompt.
(global-set-key(kbd"M-ç")#'endless/~-or-emacs.d)(defunendless/~-or-emacs.d()"Insert '~/', then '.emacs.d/'."(interactive)(if(looking-back"~/")(insert".emacs.d/")(insert"~/")))
The ç key is right under my right pinky, but find
one that fits your need.
To those who were interested in gmail-message-mode. Thanks to patjak and
oantolin, there is now a solution for using it in Firefox as well. See
the original post for the instructions.
Perry Metzger has been using Emacs as his text editor since early
September, 1983—nearly 31 years. Over much of that time, it
has also been his primary way to read email, compile programs, and
perform a variety of other tasks.
Why would anyone use a single program for that long? This talk is
partially intended to answer that question.
Emacs remains one of the most important user interfaces (and text
editors) for computer professionals almost 40 years after it was
created. The talk is intended to be part history, part philosophy,
and part speculation on the future. It will also teach Emacs fans
how to explain to their skeptical friends why it is still a good
idea to learn a tool from the terminal era that requires
memorization of dozens of control sequences in an age of GUIs and
smart phones.
Perry is (among other things) a programmer, a computer security
consultant, and a doctoral student at the University of
Pennsylvania. When he started using Emacs in 1983, it was still
written in TECO, and portions of his Emacs init file date
to 1985. He expects to still be using Emacs for decades to come.
One little-know feature of org-mode is that you can define new types
of links with the aptly named org-add-link-type. The applications of
this virtue are many. One might, for instance, write links which
search an entire code base for an expression.
(org-add-link-type"grep"'endless/follow-grep-link)(defunendless/follow-grep-link(regexp)"Run `rgrep' with REGEXP as argument."(grep-compute-defaults)(rgrepregexp"*"(expand-file-name"./")))
Then, when you click on something like the following link in an
org-mode buffer, you'll be taken to a list of results.
** TODO Refactor [[grep:OldClassName][OldClassName]] into NewClassName
LINK header arguments
As /u/blue1_ points out, for links that are simple URL substitutions
you can also use #+LINK headers.
#+LINK: isbn http://www.amazon.com/dp/%s
Tag Searches
For another use case, the following code defines links which search
your headlines for specific tags.
(org-add-link-type"tag"'endless/follow-tag-link)(defunendless/follow-tag-link(tag)"Display a list of TODO headlines with tag TAG.
With prefix argument, also display headlines without a TODO keyword."(org-tags-view(nullcurrent-prefix-arg)tag))
Trying out the myriad of Emacs mail clients is no less than an
odyssey. I am proud to say I dove into this sea of protocols and
credentials and emerged from it a better man, albeit empty handed.
I do not blame the clients available. Gnus and Mew, in particular, are
both Herculean beasts of coding prowess. It was I who failed,
regrettably, to fit them into my workflow.
Thus I designed my own solution.
gmail-message-mode combines all the features of Gmail's web interface,
with the editing prowess we all love about Emacs. How does it work?
First of all, it is not a mail client.
In you browser, when composing a message, you invoke a hotkey that sends you to Emacs.
In Emacs, you can use plain text or the full power of Markdown to write your email.
Hit C-x # to finish your edits and gmail-message-mode seamlessly
converts the message back to Gmail's format (html).
The installation process is as follows.
Install the mode
Good old (package-install 'gmail-message-mode) will do. If you'd
like to install manually, see the Readme.
Configure your browser
Chrome and Firefox require an extra add-on for editing text-fields in
Emacs. Conkeror works out of the box.
Last week, Bin Chen shared his workflow for merging Github pull
requests. Other than the use of Firefox instead of Conkeror, it was
identical to mine. Now I gladly come to admit that Alexander Yakushev
has outsmarted us both by fixing up magit-gh-pulls-mode, a package
originally written by Yann Hodique which does it all from within
Magit.
It bothers me very slightly that it immediately queries for
pull-requests when I first call magit-status, even if I’m not
interested in pull-requests at the moment. So we change the suggested
setup in order to fix that.
(when(fboundp'magit-gh-pulls-mode)(eval-after-load'magit'(define-keymagit-mode-map"#gg"#'endless/load-gh-pulls-mode))(defunendless/load-gh-pulls-mode()"Start `magit-gh-pulls-mode' only after a manual request."(interactive)(require'magit-gh-pulls)(add-hook'magit-mode-hook#'turn-on-magit-gh-pulls)(magit-gh-pulls-mode1)(magit-gh-pulls-reload)))
With this setup it’ll only activate magit-gh-pulls-mode after you
try to update the list of pull-requests.
My laptop is so old it is almost getting out-performed by my smartphone,
the principal consequence being that its keyboard has started to give
up on life. Occasionally, a key won't register when I hit it, and
occasionally I'll get paranoid and hit a key twice without knowing it
actually had registered the first time.
Emacs' keystroke echoing is a fantastic aid in this regard, informing
me whether the key worked or not when I'm typing a complex sequenced command. I only need to make it faster so
that I don't have to wait for it.
WeWork NoMad
3rd floor
79 Madison Ave.
New York, NY 10016
Notice that this month we’ll be trying out a new location (where we
can come in through the front door!)
As usual, we’ll be starting at 6:30 with pizza and beer.
Perry Metzger will be presenting on 31 Years of Emacs:
Perry Metzger has been using Emacs as his text editor since early
September, 1983 – nearly 31 years. Over much of that time, it has
also been his primary way to read email, compile programs, and
perform a variety of other tasks.
Why would anyone use a single program for that long? This talk is
partially intended to answer that question.
Emacs remains one of the most important user interfaces (and text
editors) for computer professionals almost 40 years after it was
created. The talk is intended to be part history, part philosophy,
and part speculation on the future. It will also teach Emacs fans
how to explain to their skeptical friends why it is still a good
idea to learn a tool from the terminal era that requires
memorization of dozens of control sequences in an age of GUIs and
smart phones.
Perry is (among other things) a programmer, a computer security
consultant, and a doctoral student at the University of
Pennsylvania. When he started using Emacs in 1983, it was still
written in TECO, and portions of his Emacs init file date
to 1985. He expects to still be using Emacs for decades to come.
Take a minute now and be honest with yourself. Do you like the shift
key, or do you just put up with it? Perhaps it's just because my hands
are the size of basketballs, but I've always found
Shift hard to reach.
Key-chord has allowed me to ban my shift use at least when it comes to
inputting symbols.
(key-chord-define-global"0o"")");; Sadly, "1q" is impossible to hit on my keyboard.(key-chord-define-global"1q""!")(key-chord-define-global"2w""@")(key-chord-define-global"3e""#")(key-chord-define-global"4r""$")(key-chord-define-global"5t""%")(key-chord-define-global"6y""^")(key-chord-define-global"6t""^")(key-chord-define-global"7y""&")(key-chord-define-global"8u""*")(key-chord-define-global"9i""(")(key-chord-define-global"-p""_")(key-chord-defineemacs-lisp-mode-map"7y""&optional ")(key-chord-mode1)
Anyone know of a way to get rid of shift for capitalization?
Narrowing is one of those features you won’t even hear about in a more
mundane editor, but Emacs has an entire keymap for it. While I
wouldn’t want to be without this feature, I’m all for simplification.
(defunnarrow-or-widen-dwim(p)"Widen if buffer is narrowed, narrow-dwim otherwise.
Dwim means: region, org-src-block, org-subtree, or
defun, whichever applies first. Narrowing to
org-src-block actually calls `org-edit-src-code'.
With prefix P, don't widen, just narrow even if buffer
is already narrowed."(interactive"P")(declare(interactive-only))(cond((and(buffer-narrowed-p)(notp))(widen))((region-active-p)(narrow-to-region(region-beginning)(region-end)))((derived-mode-p'org-mode);; `org-edit-src-code' is not a real narrowing;; command. Remove this first conditional if;; you don't want it.(cond((ignore-errors(org-edit-src-code)t)(delete-other-windows))((ignore-errors(org-narrow-to-block)t))(t(org-narrow-to-subtree))))((derived-mode-p'latex-mode)(LaTeX-narrow-to-environment))(t(narrow-to-defun))))(define-keyendless/toggle-map"n"#'narrow-or-widen-dwim);; This line actually replaces Emacs' entire narrowing;; keymap, that's how much I like this command. Only;; copy it if that's what you want.(define-keyctl-x-map"n"#'narrow-or-widen-dwim)(add-hook'LaTeX-mode-hook(lambda()(define-keyLaTeX-mode-map"\C-xn"nil)))
If you’re the kind of person who knows how to use narrow-to-page,
this command might not be for you. Meanwhile, for us mortals, it more
than fits the bill.
Update 05 Sep 2014
Sacha Chua’s comment below gave me a glimpse of inspiration.
I’ve never liked org’s default keybind for editing a source code
block, C-c ', and have been looking for a better
one. But if you meditate on it for a minute, the effect of
C-c ' is a narrow command with some bells
attached. So it fits perfectly into narrow-or-widen-dwim (updated
above).
Now that I’m no longer using C-c ' to edit code
blocks, I also need a better key to finish editing code blocks, and
C-x C-s makes perfect sense.
I’ve updated it to no longer use the org-in-src-block-p function and
just try calling org-edit-src-code instead. org-in-src-block-p
has proven quite unreliable.
Update 05 Dec 2015
Now it should also work with any type of org-block supported by
org-narrow-to-block.
Update 07 Dec 2015
Use LaTeX-mode-hook instead of eval-after-load, as suggested by Omar in the
comments.
hungry-delete-mode is what I like to call a “free feature” —it
asks nothing of you. In contrast, most other worthwhile features
charge you a price. The cheapest novelties charge some mental effort on
your part to fit into your workflow, the moderate ones ask for some space on
your ever-diminishing set of free keys, and a few narcisistic gems demand
that you adopt a completely new state of mind (paredit, I'm looking at you).
hungry-delete-mode doesn't steal a key and takes absolutely no
effort to master. It isn't big and flashy, but it is one of the
features I miss the most when I'm not in Emacs.
Plain and simple, it makes backspace and
C-d erase all consecutive white space in a given
direction (instead of just one). Use it everywhere.
Comments have been made with regards to this breaking
delete-selection-mode. Well, while the weak complained, Michael
Fogleman took it upon himself to fix it and make the world a better
place.
Jon Snader over at Irreal mentioned that Trey Harris shared on G+ how
to use a fallback font for Unicode Symbols. I won't repeat what they
said here, you've probably seen it already (and if you haven't then
go). I merely come to offer my own solution to this predicament, which
doesn't require any package installation and is pretty configurable to boot.
I ran into this a few months ago, while optimising my jabber+gtalk
setup, and the answer turns out to be remarkably simple. After
installing Symbola on your system, it's a 1-line solution.
If you just glanced over that, go back and read the first
function's name. Now try to say it 3 times quickly.
The documentation of set-fontset-font doesn't even remotely allude
to the fact it can be used for setting a fallback fonts. Makes me
wonder its intended purpose.
The :size 20 argument (which you're free to remove) increases the
size of this font only. It's useful because some complex unicode
symbols are hard to see in small font.
The nil would allow you to restrict the range of glyphs affected
by this font. Setting it to nil makes it the fallback font.
I love intuitive keymaps. Some are so perfect, you just can’t avoid
mouthing the words every time you hit that blissful combo. A wizard
murmuring an incantation under his breath as his fingers draw the
arcane patterns.
The following keymap toggles some options which tend to be useful
throughout a session.
(define-prefix-command'endless/toggle-map);; The manual recommends C-c for user keys, but C-x t is;; always free, whereas C-c t is used by some modes.(define-keyctl-x-map"t"'endless/toggle-map)(define-keyendless/toggle-map"c"#'column-number-mode)(define-keyendless/toggle-map"d"#'toggle-debug-on-error)(define-keyendless/toggle-map"e"#'toggle-debug-on-error)(define-keyendless/toggle-map"f"#'auto-fill-mode)(define-keyendless/toggle-map"l"#'toggle-truncate-lines)(define-keyendless/toggle-map"q"#'toggle-debug-on-quit)(define-keyendless/toggle-map"t"#'endless/toggle-theme);;; Generalized version of `read-only-mode'.(define-keyendless/toggle-map"r"#'dired-toggle-read-only)(autoload'dired-toggle-read-only"dired"nilt)(define-keyendless/toggle-map"w"#'whitespace-mode)
There are eight keys being defined there, most of which aren’t even
used every day, but I know I’ll never forget a single one. That is the
beauty of mnemonics.
Doesn’t “Emacs, toggle column” just roll off your tongue as you’re
typing C-x t c? I feel like I’m commanding the
strands of reality, but that could just be my D&D past taking the
better of me.
Also note: The manual recommends C-c for user keys, but I like using C-x for global keys and using C-c for mode-specific keys.
Hopefully I'm not the only one who obsessively tries to abide by the
high codes of good practice and writes descriptive documentation on
every last defun or defcustom. If so, I can't be the only one who
has grown tired of typing ` then ' inside Emacs doc strings.
(define-keyemacs-lisp-mode-map"\C-cm""`'\C-b")
The m key feels natural to me here, because I also
use C-c m in LaTeX for quoting math, but you might
find some other key more intuitive.
Emacs is an intelligent editor in more ways than one. Instead of
hardcoding its movement functions (such as end-of-defun or
forward-paragraph), they're implemented in terms of delimiters which
are configured by each major-mode. It works beautifully for functions,
but it feels like a stone in my shoe for paragraphs.
Say what you will, but a paragraph delimiter is a blank line.
Whichever mode I'm in, that's where I expect to go when I hit
M-e and it annoys me to no end when some
major-mode changes that.
(global-set-key(kbd"M-a")'endless/backward-paragraph)(global-set-key(kbd"M-e")'endless/forward-paragraph)(defunendless/forward-paragraph(&optionaln)"Advance just past next blank line."(interactive"p")(let((para-commands'(endless/forward-paragraphendless/backward-paragraph)));; Only push mark if it's not active and we're not;; repeating.(or(use-region-p)(not(memberthis-commandpara-commands))(memberlast-commandpara-commands)(push-mark));; The actual movement.(dotimes(_(absn))(if(>n0)(skip-chars-forward"\n[:blank:]")(skip-chars-backward"\n[:blank:]"))(if(search-forward-regexp"\n[[:blank:]]*\n[[:blank:]]*"nilt(cl-signumn))(goto-char(match-end0))(goto-char(if(>n0)(point-max)(point-min)))))))(defunendless/backward-paragraph(&optionaln)"Go back up to previous blank line."(interactive"p")(endless/forward-paragraph(-n)))
By no means am I the first to be bothered by this. Both Xah Lee and
Magnar Sveen, for instance, also complain about this in html-mode
(I'm more bothered by LaTeX-mode). Had I the bravery to delve into the mailing lists, I'd no doubt
find posts on this that are older than me.
Does paragraph navigation ever get on your nerves?
I've been using 24.4 for months now, yet only today I came to
realise the *Messages* buffer has been granted its own major-mode. The
practical difference is that the buffer is now read-only and has a
non-writing-oriented key-map.
God-only knows how many times I've fruitlessly hit q while trying to
quit the buffer, so the change is welcome.
Ray will show us his implementation of turtle graphics in emacs and
take us under the shell to see how it works. Turtle graphics are a
way of drawing pictures in which one provides a series of directions
such as “move forward” or “turn left” to steer the pen. (The name
comes from the fact that the original implementation involved a
robotic turtle as output device controlled by the computer.) In
addition to providing a useful and entertaining program, this talk
will also discuss techniques of pbm graphics and the use of a buffer
for drawing which are of general use when doing graphics in emacs.
This version addresses a previous annoyance in which you'd have to
zoom-in every time you open a page (because the mobile pages aren't
quite mobile-friendly). This was addressed by implementing a
(optional) built-in viewer which remembers your zoom level.
It also implements theme choosing through the settings menu.
I am not a fantastic typist. My speed is acceptable, but I make a great
deal of mistakes. The following snippet has turned me into the Messi
of keyboards.
Whenever I make a typo:
Hit C-x C-i, instead of erasing the mistake;
Select the appropriate correction (thanks to Ispell);
Sleep easier at night knowing I'll never see that mistake again (thanks to abbrev).
(define-keyctl-x-map"\C-i"#'endless/ispell-word-then-abbrev)(defunendless/simple-get-word()(car-safe(save-excursion(ispell-get-wordnil))))(defunendless/ispell-word-then-abbrev(p)"Call `ispell-word', then create an abbrev for it.
With prefix P, create local abbrev. Otherwise it will
be global.
If there's nothing wrong with the word at point, keep
looking for a typo until the beginning of buffer. You can
skip typos you don't want to fix with `SPC', and you can
abort completely with `C-g'."(interactive"P")(let(befaft)(save-excursion(while(if(setqbef(endless/simple-get-word));; Word was corrected or used quit.(if(ispell-wordnil'quiet)nil; End the loop.;; Also end if we reach `bob'.(not(bobp)));; If there's no word at point, keep looking;; until `bob'.(not(bobp)))(backward-word)(backward-char))(setqaft(endless/simple-get-word)))(if(andaftbef(not(equalaftbef)))(let((aft(downcaseaft))(bef(downcasebef)))(define-abbrev(ifplocal-abbrev-tableglobal-abbrev-table)befaft)(message"\"%s\" now expands to \"%s\" %sally"befaft(ifp"loc""glob")))(user-error"No typo at or before point"))))(setqsave-abbrevs'silently)(setq-defaultabbrev-modet)
In my innocence, I had initially assigned this command to a
3-key sequence. It took me less then a month to realise it needed a
shorter bind, I'd been using it 30 times a day.
A gentleman (or woman) by the name of baam_waak recently asked on
/r/emacs how can one make sure a package gets installed in case
requireing it fails. Sadly, this predicament has no perfect
solution, for you require features, not packages. Fortunately, it has
plenty of solutions that are good enough, since features commonly
share the name of their package.
Motivated by the question, I saw it fit to add this
functionality to paradox's arsenal.
(defunparadox-require(feature&optionalfilenamenoerrorpackagerefresh)"A replacement for `require' which also installs the feature if it is absent.
- If FEATURE is present, `require' it and return t.
- If FEATURE is not present, install PACKAGE with `package-install'.
If PACKAGE is nil, assume FEATURE is the package name.
After installation, `require' FEATURE.
FILENAME is passed to `require'.
If NOERROR is non-nil, don't complain if the feature couldn't be
installed, just return nil.
By default, the current package database (stored in
`package-archive-contents') is only updated if it is empty.
Passing a non-nil REFRESH argument forces this update."(or(requirefeaturefilenamet)(let((package(orpackage(if(stringpfeature)(internfeature)feature))))(require'package)(unless(andpackage-archive-contents(nullrefresh))(package-refresh-contents))(and(condition-casee(package-installpackage)(error(ifnoerrornil(error(cadre)))))(requirefeaturefilenamenoerror)))))
Just use (paradox-require 'dash) instead of (require 'dash) and
the package will be installed if necessary.
A lesson for the less-informed: while using ido to switch buffers, you
can kill buffers with C-k.
For some reason, though, you can't bury them. C-b
seems like an obvious choice, but find your personal preference.
(add-hook'ido-setup-hook(defunendless/define-ido-bury-key()(define-keyido-completion-map(kbd"C-b")'endless/ido-bury-buffer-at-head)))(defunendless/ido-bury-buffer-at-head()"Bury the buffer at the head of `ido-matches'."(interactive)(let((enable-recursive-minibufferst)(buf(ido-name(carido-matches)))(nextbuf(cadrido-matches)))(when(get-bufferbuf);; If next match names a buffer use the buffer object;;; buffer name may be changed by packages such as;; uniquify.(when(andnextbuf(get-buffernextbuf))(setqnextbuf(get-buffernextbuf)))(bury-bufferbuf)(if(bufferpnextbuf)(setqnextbuf(buffer-namenextbuf)))(setqido-default-itemnextbufido-text-initido-textido-exit'refresh)(exit-minibuffer))))
Don't be overly impressed by the apparent robustness of the code,
it's merely an adaptation of ido-kill-buffer-at-head.
Learning numeric prefixes is a vital step in the road to Emacs
mastery, and it's probably one of the editor's least appreciated
features. It took me more than a year to get the hang of doing
C-3 C-k instead of C-a C-space C-n C-n C-n C-w.
Still, having three different ways of invoking this marvel is
one-too-many for my taste, specially when they take up premium
keyboard space. M-9 isn't just a nice key, it's a
key I can hit while drunk in the dark and wearing boxing gloves —it
deserves an equally important command.
You're not a true Emacs package developer until you run
checkdoc through your packages. You'll be amazed at the
number of style errors it finds. Unfortunately, not enough
adepts are aware of this tool, perhaps because it's more of
an icing spatula than a carving knife.
Just visit your package file and run M-xcheckdoc.
It runs through the buffer and pinpoints style and spelling
errors in your docstrings and comments. It won't change the
taste of your chocolate cake, but it will give the icing
that lovely silky smoothness.
Update 29 Jun 2014
As Grant Retke and lunaryorn were kind to point out in the comments,
Flycheck does checkdoc for you.
When I decided to share my limited wisdom, I realized I’d
need something better than a 4-year-old, thrown together,
init file. The words “init dot org” had crossed my ears
through one of Sacha Chua’s previous videos; however, due to
63m of water above my head, internet searches weren’t quite
operating at top efficiency.
Without a second thought, and armed with nothing but its
name (init.org), I took to writing —I thought at the
moment— the simplest way of implementing such a feature.
;;; init.el(defvarendless/init.org-message-depth3"What depth of init.org headers to message at startup.")(with-temp-buffer(insert-file"~/.emacs.d/init.org")(goto-char(point-min))(search-forward"\n* init.el")(while(not(eobp))(forward-line1)(cond;; Report Headers((looking-at(format"\\*\\{2,%s\\} +.*$"endless/init.org-message-depth))(message"%s"(match-string0)));; Evaluate Code Blocks((looking-at"^#\\+BEGIN_SRC +emacs-lisp *$")(let((l(match-end0)))(search-forward"\n#+END_SRC")(eval-regionl(match-beginning0))));; Finish on the next level-1 header((looking-at"^\\* ")(goto-char(point-max))))))
Once back within the range of cell towers, a quick search
revealed it could have been slightly shorter.
Nonetheless, I stuck with my guns and kept the first version.
Primarily for stubbornness, but also a few other reasons:
Fine grained control
I can choose exactly what gets
evaluated. In this case, anything inside the
init.el header. This is important because this blog
is my init file, and I don’t want to evaluate everything
I post.
Lots of messaging
It calls message on each header it
finds (up to a configurable level). Whenever something goes
wrong, forget about restarting with --debug-init, the
messages buffer tells me exactly where it happened.
It doesn’t (require 'org)
This might sound silly, but
that inconspicuous line forms one of the most
time-consuming statements you could possibly write. I’d
challenge anyone to find a 14-character statement that
takes longer than that (other than an empty loop or a
sleep command, of course).
As part of an ongoing process of de-Googling, we’re moving the RSS feed off of Feedburner.
This means that if you want to continue receiving updates from this site, you’ll need to change the address in your RSS reader to http://emacsnyc.org/atom.xml.
At Sonian we have a small team
building interactive visualizations where companies can better understand their
massive amounts of email data. Building this kind of program is non-trivial:
data needs to be synchronized between servers and complex interfaces through
user interactions while at the same time "feeling right" to the user. Like other
software disciplines it's a combination of engineering and art.
As engineers we want to write our programs so that they have as few bugs as
possible and are easily extendable, and as a company delivering value to
customers we want stable features delivered quickly. Reconciling these
objectives is a universal problem which is made easier in software engineering
through the use of patterns and tools.
One pattern used to help with building contemporary web applications is
MVC, conceived in
the 1970s as a pattern for separating concerns in software programs, still
popular today. Backbone is a well-known implementation of MVC where you build
programs with object-oriented constructs: models and collections
encapsulate data and views respond to user interactions and mutate the
browser DOM. Objects communicate with each other through method calls and
asynchronous events. We started
this project using Backbone simply because that's what we were more comfortable
with, having using it extensively (and arguably successfully) since 2011.
Scaling Backbone
The complexity of the software systems we are asked to develop is increasing,
yet there are basic limits on our ability to cope with this complexity.
As we progressed through requirements, we had built collapsible panes, modals,
popups with dynamic and navigatable content, full-text search filters, data
sorting and pagination, and so on. Every one of these parts had to work in
concert so that the UI represented the right context for the sequence of actions
a user applied in any of multiple available interconnected interactive D3.js
visualizations.
The code behind the features turned complex fast and we reached a point where a
fair portion of effort was spent on maintenance. This situation is certainly
familiar to most.
Graphs of stateful objects
Contemporary event-based MVC architectures can be viewed as a graph of
stateful nodes communicating asynchronously.
In Backbone, particularly, despite following best practices and using
complementary tools like Marionette, it's likely
that as the codebase evolves it ends up with:
Singular models dispatching events handled by multiple views
View A handling events from View B and vice-versa
Dispatched events causing chain/cycle reactions
An "event bus" where a dispatched event is handled by multiple objects
Example typical backbone architecture.
This kind of construction makes it hard to reason about the program. In addition
to global mutable state there
are events being dispatched likely causing multiple objects to change their
state, which in their turn may dispatch even more events, possibly in ways you
didn't anticipate, making the program harder to extend and debug.
Programs that use state in a haphazard way are very difficult to understand.
For example, if the state is visible throughout the whole program, then it can
be assigned anywhere. The only way to reason is to consider the whole program
at once.
While it's still entirely possible to build complex and useful programs using
this kind of architecture (people have been doing it for years), maybe there's a
better way to write software where we don't spend so much effort juggling with
complexity unrelated to the actual problem at hand.
Enter React
An alternative tool for helping build large-scale web applications is React.
Open-sourced by Facebook, React is a "library for creating user interfaces", and
it works by having the programmer express how their UI should look like at any
point in time and having React automatically manage UI updates when the
underlying data changes, obviating the need for explicit event observation or
manual DOM mutation.
(...) programmers did not know how harmful complexity is, and secondly they
did not know either, how much complexity can usually be avoided if you give
your mind to it.
The basic unit of construction is the component, which is essentially a
function: composable by definition. Components are lightweight, in-memory
representation of the DOM, and as their input changes React diffs these
representations and batch applies the minimal set of changes required on the
real DOM. The best part is that when using React you don't have to think about
any of this: just build components! React does the hard work under the hood.
This is in our opinion a much less leaky abstraction for synchronizing program
state with its visual representation than any of the current solutions. It
allows you to build programs using functions and data structures rather than
framework-specific constructs or markup, and it makes entangling logic from your
problem domain with observables a thing from the past.
The React Way
React draws a lot of inspiration from functional programming. It promotes
immutability through the clear separation of props and state and encourages
writing functions (components) that are free from side-effects and always return
the same values (DOM representation) given the same inputs (props).
The functional model of computation combined with immutable data and one-way
dataflow makes it much easier to reason about your program, greatly facilitating
extension and debugging.
Our Approach
We got to know React late 2013 through the Clojure community, which quickly
adopted it as a solution for the
DOM problem, and
our transition to it started by replacing a complicated composite, nested
Backbone view that showed related word clouds side by side and allowed the user
to perform interactions with either individual or groups of words. The existing
implementation involved a lot of state and several events. It was hard to
understand and had noticeable lag when the word clouds were too big. It had been
a few weeks since we wanted to start using React, so we took this opportunity to
see if it could better handle complexity. After literally one day of learning we
started the transition, and after two days we had replaced the view with a React
component with less code and more features. Its performance with even bigger
word clouds was smooth even though we didn't write a single line thinking about
performance and it integrated nicely with every other part of the Backbone
application.
We considered that the experiment validated the power of React, and since then
we replaced almost all of the Backbone code: models and collections are now pure
functions returning immutable data, namespaced by regular JavaScript objects and
views are now React components. Note that we could have continued using Backbone
models and collections, because React doesn't care about what your data looks
like. It doesn't tell you how to talk to servers, or what routing construct to
use, so you're free to choose the tools and patterns you find best. In our case
it looks like the following:
A single mutable reference to immutable application state (a database,
conceptually)
"State manager" with state-changing transactions that are passed down to
components as props
The application state is an immutable hash map provided by
mori, which we use to represent every
piece of data. Transactions are pure functions that return transformations of
the current application state, which the state manager uses to transition to new
states. With state and its possible transitions isolated to a single place, the
rest of the application is essentially just logic for displaying data. This
approach gives us a
model of time
for free, since the state manager has access to the application state history.
With time, application-wide undo and redo is trivial.
Wins
Adding features needs much less changing of previous code to account for new
behaviors. Components are naturally decoupled, meaning code diffs are mainly
additions.
Codebase reductions. Not as dramatic as folks on
Prismatic
have seen, since we haven't transitioned to ClojureScript yet.
Less time fighting MVC frameworks and libraries to make things work. React has
only one major "concept" (the component), which makes it a very simple
abstraction in contrast.
Performance improvements achieved through adding a single, generally short,
idiomatic function. Conversely, performance improvements in MVC systems are
achieved by minimizing DOM interactions through improvised caching and reduced
usage of data-binding in key areas via,
ad-hoc,
non-idiomatic
code changes that generally require extensive comments in order to be
explained and justified.
Future plans
Sonian already uses ClojureScript and om in other internal projects and we'll be
transitioning to that soon.
Conclusion
React is a powerful tool that we found to be objectively better than
contemporary MVC frameworks. It provides a simpler abstraction with the
collateral benefit of delivering faster performance. It's helping us build UIs
that would be nearly impossible to build using current alternatives under our
current constraints.
GNU Emacs is seven years older than Unicode. Support for Unicode had
to be added relatively late in Emacs’ existence. This means Emacs has
existed longer without Unicode support (16 years) than with it (14
years). Despite this, Emacs has excellent Unicode support. It feels as
if it was there the whole time.
However, as a natural result of Unicode covering all sorts of edge
cases for every known human language, there are pitfalls and
complications. As a user of Emacs, you’re not particularly affected
by these, but extension developers might run into trouble while
handling Emacs character-oriented data structures: strings and
buffers.
In this article I’ll go over Elisp’s Unicode surprises. I’ve been
caught by some of these myself. In fact, as a result of writing this
article, I’ve discovered subtle encoding bugs in some of my own
extensions. None of these pitfalls are Emacs’ fault. They’re just the
result of complexities of natural language.
Unicode and Code Points
First, there are excellent materials online for learning Unicode. I
recommend starting with UTF-8 and Unicode FAQ for Unix/Linux.
There’s no reason for me to repeat all this information here, but I’ll
attempt to quickly summarize it.
Unicode maps code points (integers) to specific characters, along
with a standard name. As of this writing, Unicode defines over 110,000
characters. For backwards compatibility, the first 128 code points are
mapped to ASCII. This trend continues for other character standards,
like Latin-1.
In Emacs, Unicode characters are entered into a buffer with C-x 8
RET (insert-char). You can enter either the official name of the
character (e.g. “GREEK SMALL LETTER PI” for π) or the hexadecimal code
point. Outside of Emacs it depends on the application, but C-S-u
followed by the hexadecimal code works for most of the applications I
care about.
Encodings
The Unicode standard also describes several methods for encoding
sequences of code points into sequences of bytes. Obviously a selection
of 110,000 characters cannot be encoded with one byte per letter, so
these are multibyte encodings. The two most popular encodings are
probably UTF-8 and UTF-16.
UTF-8 was designed to be backwards compatible with ASCII, Unix, and
existing C APIs (null-terminated C strings). The first 128 code points
are encoded directly as a single byte. Every other character is
encoded with two to six bytes, with the highest bit of each byte set
to 1. This ensures that no part of a multibyte character will be
interpreted as ASCII, nor will it contain a null (0). The latter means
that C programs and C APIs can handle UTF-8 strings with few or no
changes. Most importantly, every ASCII encoded file is automatically a
UTF-8 encoded file.
UTF-16 encodes all the characters from the Basic Multilingual Plane
(BMP) with two bytes. Even the original ASCII characters get two bytes
(16 bits). The BMP covers virtually all modern languages and is
generally all you’ll ever practically need. However, this doesn’t
include the important TROPICAL DRINK or PILE OF POO
characters from the supplemental (“astral”) plane. If you need to use
these characters in UTF-16, you’re going to run into problems:
characters outside the BMP don’t fit in two bytes. To accommodate
these characters, UTF-16 uses surrogate pairs: these characters are
encoded with two 16-bit units.
Because of this last point, UTF-16 offers no practical advantages
over UTF-8. Its existence was probably a big mistake. You
can’t do constant-time character lookup because you have to scan for
surrogate pairs. It’s not backwards compatible and cannot be stored in
null-terminated strings. In both Java and JavaScript, it leads to the
awkward situation where the “length” of a string is not the number of
characters, code points, or even bytes. Worst of all, it has serious
security implications. New applications should avoid it
whenever possible.
Emacs and UTF-8
Emacs internally stores all text as UTF-8. This was an excellent
choice! When text leaves Emacs, such as writing to a file or to a
process, Emacs automatically converts it to the coding system
configured for that particular file or process. When it accepts text
from a file or process, it either converts it to UTF-8 or preserves it
as raw bytes.
There are two modes for this in Emacs: unibyte and multibyte. Unibyte
strings/buffers are just raw bytes. They have constant access O(1)
time but can only hold single-byte values. The byte-code compiler
outputs unibyte strings.
Multibyte strings/buffers hold UTF-8 encoded code points. Character
access is O(n) because the string/buffer has to be scanned to count
characters.
The actual encoding is rarely relevant because there’s little way (and
need) to access it directly. Emacs automatically converts text as
needed when it leaves Emacs and arrives in Emacs, so there’s no need
to know the internal encoding. If you really want to see it anyway,
you can use string-as-unibyte to get a copy of a string with the
exact same bytes, but as a byte-string.
(string-as-unibyte"π");; => "\317\200"
This can be reversed with string-as-multibyte), to change a unibyte
string holding UTF-8 encoded text back into a multibyte string. Note
that these functions are different than string-to-unibyte and
string-to-multibyte, which will attempt a conversion rather than
preserving the raw bytes.
The length and buffer-size functions always count characters in
multibyte and bytes in unibyte. Being UTF-8, there are no surrogate
pairs to worry about here. The string-bytes and position-bytes
functions return byte information for both multibyte and unibyte.
To specify a Unicode character in a string literal without using the
character directly, use \uXXXX. The XXXX is the hexadecimal code
point for the character and is always 4 digits long. For characters
outside the BMP, which won’t fit in four digits, use a capital U with
eight digits: \UXXXXXXXX.
"\u03C0";; => "π""\U0001F4A9";; => "💩" (PILE OF POO)
Finally, Emacs extends Unicode with 256 additional “characters”
representing raw bytes. This allows raw bytes to be embedded
distinctly within UTF-8 sequences. For example, it’s used to
distinguish the code point U+0041 from the raw byte #x41. As far as I
can tell, this isn’t used very often.
Combining Characters
Some Unicode characters are defined as combining characters. These
characters modify the non-combining character that appears before it,
typically with accents or diacritical marks.
For example, the word “naïve” can be written as six characters as
"nai\u0308ve". The fourth character, U+0308 (COMBINING DIAERESIS),
is a combining character that changes the “i” (U+0069 LATIN SMALL
LETTER I) into an umlaut character.
The most commonly accented characters have a code of their own. These
are called precomposed characters. This includes ï (U+00EF LATIN
SMALL LETTER I WITH DIAERESIS). This means “naïve” can also be written
as five characters as "na\u00EFve".
Normalization
So what happens when comparing two different representations of the
same text? They’re not equal.
(string="nai\u0308ve""na\u00EFve");; => nil
To deal with situations like this, the Unicode standard defines four
different kinds of normalization. The two most important ones are NFC
(composition) and NFD (decomposition). The former uses precomposed
characters whenever possible and the latter breaks them apart. The
functions ucs-normalize-NFC-string and ucs-normalize-NFD-string
perform this operation.
Pitfall #1: Proper string comparison requires normalization. It
doesn’t matter which normalization you use (though NFD should be
slightly faster), you just need to use it consistently. Unfortunately
this can get tricky when using equal to compare complex data
structures with multiple strings.
(string=(ucs-normalize-NFD-string"nai\u0308ve")(ucs-normalize-NFD-string"na\u00EFve"));; => t
Emacs itself fails to do this. It doesn’t normalize strings before
interning them, which is probably a mistake. This means you can have
differently defined variables and functions with the same canonical
name.
(eq(intern"nai\u0308ve")(intern"na\u00EFve"));; => nil(defunprint-résumé()"NFC-normalized form."(print"I'm going to sabotage your team."))(defunprint-résumé()"NFD-normalized form."(print"I'd be a great asset to your team."))(print-résumé);; => "I'm going to sabotage your team."
String Width
There are three ways to quantify multibyte text. These are often the
same value, but in some circumstances they can each be different.
length: number of characters, including combining characters
bytes: number of bytes in its UTF-8 encoding
width: number of columns it would occupy in the current buffer
Most of the time, one character is one column (a width of one). Some
characters, like combining characters, consume no columns. Many Asian
characters consume two columns (U+4000, 䀀). Tabs consume tab-width
columns, usually 8.
Generally, a string should have the same width regardless of which
whether it’s NFD or NFC. However, due to bugs and incomplete Unicode
support, this isn’t strictly true. For example, some combining
characters, such as U+20DD ⃝, won’t combine correctly in Emacs nor in
other applications.
Pitfall #2: Always measure text by width, not length, when laying
out a buffer. Width is measured with the string-width function.
This comes up when laying out tables in a buffer. The number of
characters that fit in a column depends on what those characters are.
Fortunately I accidentally got this right in Elfeed because
I used the format function for layout. The %s directive operates
on width, as would be expected. However, this has the side effect that
the output of may format change depending on the current buffer!
Pitfall #3: Be mindful of the current buffer when using the format
function.
Regular expressions operate on code points. This means combining
characters are counted separately and the match may change depending
on how characters are composed. To avoid this, you might want to
consider NFC normalization before performing some kinds of regular
expressions.
;; Like string= from before:(string-match-p"na\u00EFve""nai\u0308ve");; => nil;; The . only matches part of the composition(string-match-p"na.ve""nai\u0308ve");; => nil
Pitfall #5: Be mindful of combining characters when using regular
expressions. Prefer NFC normalization when dealing with regular
expressions.
Another potential problem is ranges, though this is quite uncommon.
Ranges of characters can be expressed in inside brackets, e.g.
[a-zA-Z]. If the range begins or ends with a decomposed combining
character you won’t get the proper range because its parts are
considered separately by the regular expression engine.
It’s especially important to keep all of this in mind when
sanitizing untrusted input, such as when using Emacs as a web server.
An attacker might use a denormalized or strange grapheme cluster to
bypass a filter.
Interacting with the World
Here’s a mistake I’ve made twice now. Emacs uses UTF-8 internally,
regardless of whatever encoding the original text came in. Pitfall #6:
When working with bytes of text, the counts may be different than
the original source of the text.
For example, HTTP/1.1 introduced persistent connections. Before this,
a client connects to a server and asks for content. The server sends
the content and then closes the connection to signal the end of the
data. In HTTP/1.1, when Connection: close isn’t specified, the
server will instead send a Content-Length header indicating the
length of the content in bytes. The connection can then be re-used for
more requests, or, more importantly, pipelining requests.
The main problem is that HTTP headers usually have a different
encoding than the content body. Emacs is not prepared to handle
multiple encodings from a single source, so the only correct way to
talk HTTP with a network process is raw. My mistake was allowing Emacs
to do the UTF-8 conversion, then measuring the length of the content
in its UTF-8 encoding. This just happens to work fine about 99.9% of
the time since clients tend to speak UTF-8, or something like it,
anyway, but it’s not correct.
Further Reading
A lot of this investigation was inspired by JavaScript’s and other
languages’ Unicode shortcomings.
WeWork Soho West
8th floor lounge
69 Charlton St.
New York, NY 10014
As usual, we’ll be starting at 6:30 with pizza and beer!
Ray Puzio will be presenting on Emacs Mutant Anime Turtles: Impersonating Logo with Emacs Lisp:
Ray will show us his implementation of turtle graphics in emacs and take us under the shell to see how it works. Turtle graphics are a way of drawing pictures in which one provides a series of directions such as “move forward” or “turn left” to steer the pen. (The name comes from the fact that the original implementation involved a robotic turtle as output device controlled by the computer.) In addition to providing a useful and entertaining program, this talk will also discuss techniques of pbm graphics and the use of a buffer for drawing which are of general use when doing graphics in emacs.
Games are a great way to get started writing programs in any
language. In Emacs Lisp, they’re even better—you use the same exact
techniques to extend Emacs, configuring it to do what you want. In
this presentation, I livecode tic-tac-toe. You’ll see how to create
a basic major mode, make functions, store state, and set
keybindings.
Bailey will provide a lightning talk on evil-mode, aptly named after
the editor of the beast. He’ll show why evil-mode can make Vim
veterans feel at home while exposing them to the power available in
Emacs.
WeWork Soho West
8th floor lounge
69 Charlton St.
New York, NY 10014
As usual, we’ll be starting at 6:30 with pizza and beer!
Zachary Kanfer will be giving a talk on writing games with Emacs:
Games are a great way to get started writing programs in any language. In Emacs Lisp, they’re even better—you use the same exact techniques to extend Emacs, configuring it to do what you want. In this presentation, I livecode tic-tac-toe. You’ll see how to create a basic major mode, make functions, store state, and set keybindings.
Bailey Ling will be giving a lightning talk on evil-mode:
Bailey will provide a lightning talk on evil-mode, aptly named after the editor of the beast. He’ll show why evil-mode can make Vim veterans feel at home while exposing them to the power available in Emacs.
Evan will provide an introduction to incremental analysis through
IPython moderated by the editing features of Emacs. A brief
introduction to integrating with Org-mode will also be
given. Advantages over the R console and Bash shell will be
discussed.
Evan has made his slides (in org-mode!) available.
With a little effort, Emacs can be a powerful, multi-language IDE
with code completion, documentation lookup, and more. Taking Python
as our language, we will show how to use the the auto-complete, epc,
and jedi.el libraries to create a semantically rich editing
experience. We’ll discuss how these libraries work together and how
to customize them for your own setup.
For many years Richard Stallman (RMS) prohibited a foreign function
interface (FFI) in GNU Emacs. An FFI is an API for dynamically calling
native libraries at run-time, like the Java Native Interface (JNI). He
was concerned that people might use it to make proprietary extensions
to the popular editor. This was the same (paranoid) justification for
rejecting a package manager in Emacs for many years, that someone
might use it to distribute proprietary packages.
Fortunately, times have changed. RMS reevaluated his
stances on FFI and on package managers. Today Emacs comes with
a package manager (package.el), and there are multiple package
repositories with no proprietary packages in sight. Though, outside of
some unaccepted patches, no significant progress has been
made to add an FFI.
A few weeks ago I did something about that by writing a package that
adds an FFI. It requires no patches or any other changes to Emacs
itself. Instead, it drives a subprocess running libffi,
passing arguments and return values back and forth through a pipe, in
the spirit of EmacSQL. It’s not as efficient as a built-in
API, but it could potentially be distributed through an ELPA
repository.
The API is modeled loosely after Julia’s elegant FFI. A call
interface (CIF) doesn’t need to be prepared ahead of time. Provide all
the necessary information at the call site and the library takes care
of building and caching CIFs and handles for you.
API Examples
The core function for the FFI is ffi-call. Here’s an example that
calls the system’s srand() and then rand().
;; seed with 0(ffi-callnil"srand"[:void:uint32]0);; => :void(ffi-callnil"rand"[:sint32]);; => 1102520059
The first two arguments are similar to the first two arguments of
dlsym(). For ffi-call, the first argument is the library shared
object name. The back-end automatically takes care of obtaining a
handle on the library with dlopen(). In this case we’re accessing a
function that’s already in the main program, so we pass nil. This is
identical to passing NULL to dlsym(). In this FFI, nil always
corresponds to NULL.
The second argument is the function name, just like dlsym()’s second
argument.
The third argument is the function signature. It’s a vector of
keywords declaring the return value type followed by the types of each
argument. In this example, srand() returns nothing (void) and
accepts a single 32-bit unsigned argument, so the signature is
[:void :uint32].
The remaining arguments are the native function arguments. I can keep
making the second FFI call (“rand”) to retrieve different numbers,
using the first FFI call (“srand”) to reset the sequence.
Using a Library
Here’s another example, loading libm and calling cos.
The first time a library is used, the back-end creates a handle for it
with dlopen(). Further calls will reuse the handle, trying to be as
efficient as possible. Handles are never closed.
Pointers
Here are a couple of examples that use pointers. As stated before, nil
is used to pass a NULL pointer. Like the underlying libffi, the FFI
doesn’t care what kind of pointer you’re passing, just that it’s a
pointer, so it’s declared with :pointer.
Strings are automatically copied to the subprocess, their lifetime
tied to the lifetime of the Elisp string (note: this detail is still
unimplemented). When used as arguments, they become pointers.
Pointers can be handled as values on the Elisp side. They’re
represented as symbols whose name is an address. In the above example,
0x7fffc13ceb29 is one of these symbols. I would have preferred to
use a plain integer to represent pointers, but, because Elisp integers
are tagged, they’re guaranteed not to be wide enough for this. I
plan to add pointer operators to do pointer arithmetic on these
special pointer values.
The function ffi-get-string is used to retrieve the null-terminated
string referenced by a pointer. If the string returned by getenv()
needed to be freed (it doesn’t and shouldn’t), the FFI caller would
need to be careful to call free() as another FFI call.
How It Works: The Stack Machine
My goal is to keep the back-end as simple as possible. All resource
management is handled by Emacs, tied to garbage collection.
For example, the pointer returned by dlopen() isn’t stored anywhere
in the subprocess. It’s passed to Emacs and managed there. To call a
function using the handle, the pointer is transmitted back to the
subprocess.
To keep it simple, the back-end is just a stack machine with a simple
human-readable bytecode. You can see the instruction set by looking at
the big switch statement in ffi-glue.cc. For example, to push a
signed 2-byte integer 237 onto the stack, send a j followed by an
ASCII representation of the number (terminated by a space if needed):
j237.
As usual, my assumption is that the Elisp printer and reader is faster
than any possible serialization I could implement within Elisp itself.
This also nicely sidesteps the byte-order issue.
The function signature is declared by pushing zeros of the
return/argument types onto the stack, with a special void “value” used
to communicate void. Once it’s all set up, the C instruction is
called, collapsing the signature into a CIF handle: a pointer for the
Elisp side to manage.
Pointers to raw strings of bytes are pushed onto the stack with the
M instruction. It pops the top integer on the stack to get the byte
count, reads that number of bytes from input into a buffer,
null-terminates the buffer in case it’s used as a string, and finally
puts a pointer to that buffer on the stack.
Calling functions is just a matter of pushing all the needed
information onto the stack, invoking libffi to magically call the
function, then popping the result off the stack. Popping a value
transmits it to Elisp.
Stack Machine Example
Here’s a concise example that calls cos(1.2) (assuming libm.so is
already linked). The actual Elisp-generated FFI bytecode doesn’t plan
things quite this way — particularly because it needs to keep track
of the various pointers involved — but this example keeps it simple.
d1.2d0d0w1Cp0w3McosSco
You can run this example manually by executing the ffi-glue program
and pasting in that line as standard input. The result will be
printed.
d1.2 : Push a double, 1.2, onto the stack. This will be the
function argument.
d0d0 : Push a couple of zero doubles onto the stack. This is our
function signature. It takes a double and returns a double.
w1 : Push an unsigned 32-bit 1 onto the stack. Instructions that
use integers accept unsigned 32-bit integers. This 1 indicates that
our function accepts one argument.
C : create a CIF. The integer 1 and the two 0 doubles are
consumed and a pointer to a CIF is put on the stack. Elisp would
normally pop this off and save it for future use, but we’re going
to leave it there (and ultimately leak it in the example).
p0 : Push a NULL onto the stack. p means push a pointer and 0
is a NULL pointer. This is our library handle. We’re assuming cos
will be in the main program.
w3Mcos : Put a pointer to the string “cos” into the stack. First
push on the number 3 (string length), then M to read from input,
then pass three bytes: “cos”. In our example, this buffer will be
leaked because we lose the buffer pointer.
S : Call dlsym() on the string and handle on top of the stack.
This consumes the top two values (NULL and “cos”), and pushes a
function handle on top of the stack. At this point the stack has
three values: 1.2, the CIF, and the function handle.
c : Call the function pointed to by the top of the stack. This
consumes the top pointer, the CIF below it, and the CIF indicates
how many more values to consume: just one in this case, since the
function takes one argument. The function’s return value is pushed
on the stack. If the function is void, the special void “value”
is pushed on the stack.
o : Pop the top stack value, sending it to Emacs. This is
what would be returned by ffi-call.
Before I got the Elisp side of things going, I was testing out the
back-end by writing lots of little programs like this by hand.
A Safe FFI
While using an FFI through a pipe is slow compared to a built-in FFI,
there is a distinct advantage. The FFI can never crash Emacs! Normally,
making calls to an FFI is unsafe. It allows the programmer to
violate normal language constraints. If the programmer misuses the
FFI, the whole process may crash or become corrupt. This will lose any
state held behind foreign interface, but Emacs will be safe.
In my package, the handle for the FFI Emacs subprocess is called the
context. A context is automatically established and bound to the
ffi-context global variable as needed. This context keeps track of
CIFs, string buffers, handles, and any other resources held by the
subprocess. If the subprocess dies, the context becomes meaningless
since the pointers it holds are dead.
Limitations
This FFI package is about 80% complete. It occasionally leaks memory
in the subprocess, it’s overly-sensitive to mis-typing, it doesn’t
manage stdin/stdout, it can’t inspect/modify structs, and it can’t set
up closures.
The last point, closures, would require some changes to the
interprocess communication. The purpose here would be to allow foreign
functions to call Elisp functions. The subprocess would need to be
able to initiate activity with Elisp.
Manipulating structs is complex, and even libffi has limited support
for working with them. It allows structs to be declared, but leaves
alignment and access up to the user to sort out. That’s where the
previously-mentioned pointer arithmetic comes into play.
Currently stdin, stdout, and stderr are problems, especially when I
was trying to write a test GTK application with Elisp. Any command
line junkie knows that GTK (and Qt) applications are ridiculously
noisy. It spews hundreds of lines of warnings and notifications as
part of its normal operation. This noise interferes with FFI
communication with Emacs. I need to figure out how to separate this
and get standard input/output/error to/from Emacs through separate
channels.
Like libffi, there are no guarantees about variadic function calls. It
should generally Just Work, but you can’t rely on it.
The whole thing will not work as well in 32-bit Emacs, where integers
are limited to a tiny 29 bits. For example, those rand() return
values will simply not fit. In the long run, this is probably the
single largest barrier to making the FFI work smoothly. It’s too easy
to run into large integer values.
Right now I consider it a proof of concept; an FFI really can be
done this way. I don’t have any particular uses in mind, and, outside
of the “cool factor,” I can’t actually think of any useful
applications. If a solid FFI already existed, I may have tried to use
it for EmacSQL rather than use this subprocess trick. My FFI is
probably mature enough to drive SQLite, so maybe this is the future of
EmacSQL.
If you can think of a good use for an Emacs FFI, please share it. I
need good test ideas.
WeWork Soho West
8th floor lounge
69 Charlton St.
New York, NY 10014
We’ll be starting at 6:30 with pizza and beer. Drew Werner will be giving a full-length talk, and we’ll also have a lightning talk by Evan Misshula.
Drew Werner (twitter, github) will be talking about using Emacs as a Python IDE:
With a little effort, Emacs can be a powerful, multi-language IDE with code completion, documentation lookup, and more. Taking Python as our language, we will show how to use the the auto-complete, epc, and jedi.el libraries to create a semantically rich editing experience. We’ll discuss how these libraries work together and how to customize them for your own setup.
Evan Misshula will be giving a lightning talk on The best of both worlds: Combining the best REPL (IPython 2.0) and Editor (Emacs):
Evan will provide an introduction to incremental analysis through IPython moderated by the editing features of Emacs. A brief introduction to integrating with Org-mode will also be given. Advantages over the R console and Bash shell will be discussed.
Internet Relay Chat (IRC) has been a popular P2P messaging system
since 1988. Since it uses a text-based interface, it’s a perfect fit
for Emacs! We’ll be looking at ERC, the built-in Emacs IRC client
and the solution to all our social needs. ERC easily fits into
anyone’s workflow and has tons of libraries to make it even more
seamless.
Emacs can be thought of as a big Lisp interpreter, so you can’t
master Emacs without learning some Emacs Lisp. We’ll be introducing
Emacs Lisp by describing its simple syntax, demonstrating a few Lisp
functions for manipulating buffers, regions, and strings, writing a
few utility functions, and binding those functions to custom
keys. By the end of the talk, you should able to do the same.
thoughtbot NYC
1st floor of the WeWork at Bryant Park
54 W. 40th St.
New York, NY
Harry Schwartz will be giving an introduction to Emacs Lisp:
Emacs can be thought of as a big Lisp interpreter, so you can’t master Emacs without learning some Emacs Lisp. We’ll be introducing Emacs Lisp by describing its simple syntax, demonstrating a few Lisp functions for manipulating buffers, regions, and strings, writing a few utility functions, and binding those functions to custom keys. By the end of the talk, you should able to do the same.
Internet Relay Chat (IRC) has been a popular P2P messaging system since 1988. Since it uses a text-based interface, it’s a perfect fit for Emacs! We’ll be looking at ERC, the built-in Emacs IRC client and the solution to all our social needs. ERC easily fits into anyone’s workflow and has tons of libraries to make it even more seamless.
One of the drawbacks of Emacs Lisp is the lack of namespaces. Every
defun, defvar, defcustom, defface, defalias, defstruct,
and defclass establishes one or more names in the global scope. To
work around this, package authors are strongly encouraged to prefix
every global name with the name of its package. That way there should
never be a naming conflict between two different packages.
While this has solved the problem for the time being, attaching the
package name to almost every identifier, including private function
and variable names, is quite cumbersome. Namespaces can almost be
hacked into the language by using multiple obarrays,
but symbols have internal linked lists that prohibit
inclusion in multiple obarrays.
By convention, private names are given a double-dash after the
namespace. If a “bar counter” is an implementation detail that may
disappear in the future, it will be called mypackage--bar-counter to
warn users and other package authors not to rely on it.
There’s been a recent push to follow this namespace-prefix policy more
strictly, particularly with the depreciation of cl and introduction
of cl-lib. I suspect someday when namespaces are finally introduced,
packages with strictly clean namespaces with be at an advantage,
somehow automatically supported. Nic Ferrier has proposed ideas
for how to move forward on this.
How strict are we talking?
Over the last few years I’ve gotten much stricter in my own packages
when it comes to namespace prefixes. You can see the progression going
from javadoc-lookup (2010) where I was completely sloppy about
it, to EmacSQL (2014) where every single global identifier
is meticulously prefixed.
For a time I considered names such as make-* and with-* to be
exceptions to the rule, since these names are idioms inherited from
Common Lisp. The namespace comes after the expected prefix. I’ve
changed my mind about this, which has caused me to change my usage of
defstruct (now cl-defstruct).
Just as in Common Lisp, by default cl-defstruct defines a
constructor starting with make-*. This is fine in Common Lisp, where
it’s a package-private function by default, but in Emacs Lisp this
pollutes the global namespace.
This constructor isn’t namespace clean, so package authors should
avoid defstruct’s default. If the package is named circle then all
of the accessors are perfectly fine, though.
To fix this, I now use another, more recent Emacs Lisp idiom: name the
constructor create. That is, for the package circle, we desire
circle-create. To get this behavior from cl-defstruct, use the
:constructor option.
This affords a new opportunity to craft a better constructor. Have
cl-defstruct define a private constructor, then manually write a
constructor with a nicer interface. It may also do additional work,
like enforce invariants or initialize dependent slots.
(cl-defstruct(circle(:constructorcircle--create))xyradius)(defuncircle-create(xyradius)(let((circle(circle--create:xx:yy:radiusradius)))(if(<radius0)(error"must have non-negative radius")circle)))(circle-create001);; => [cl-struct-circle 0 0 1](circle-create00-1);; error: "must have non-negative radius"
This is now how I always use cl-defstruct in Emacs Lisp. It’s a tidy
convention that will probably become more common in the future.
Yesterday I made the first official release of EmacSQL, an
Emacs package I’ve been working on for the past few weeks. EmacSQL is
a high-level SQL database for Emacs. It primarily targets SQLite as a
back-end, but it also currently supports PostgreSQL and MySQL.
While there’s a non-Elisp component, SQLite, there are no special
requirements for the user to worry about. When the package’s Elisp is
compiled, if a C compiler is available it will use it to compile a
SQLite binary for EmacSQL. If not, it will later offer to download a
pre-built binary that I built. Ideally this makes the non-Elisp part
of EmacSQL completely transparent and users can pretend Emacs has a
built-in relational database.
The official SQLite command line shell is not used even if present,
and I’ll explain why below.
Just as Skewer jump started my web development experience,
EmacSQL has been a crash course in SQL and relational databases.
Before starting this project I knew little about this topic and I’ve
gained a lot of appreciation for it in the process. Building an Emacs
extension is a very rapid way to dive into a new topic.
If you’re a total newb about this stuff like I was and want to learn
SQL for SQLite yourself, I highly recommend Using SQLite. It’s
a really solid introduction.
High-level SQL Compiler
By “high-level” I mean that it goes beyond assembling strings
containing SQL code. In EmacSQL, statements are assembled from
s-expressions which, behind the scenes, are compiled into SQL using
some simple rules. This means if you already know SQL you should be
able to hit the ground running with EmacSQL. Here’s an example,
(require'emacsql);; Connect to the database, SQLite in this case:(defvardb(emacsql-connect"~/office.db"));; Create a table with 3 columns:(emacsqldb[:create-tablepatients([name(idinteger:primary-key)(weightfloat)])]);; Insert a few rows:(emacsqldb[:insert:intopatients:values(["Jeff"1000184.2]["Susan"1001118.9])]);; Query the database:(emacsqldb[:select[nameid]:frompatients:where(<weight150.0)]);; => (("Susan" 1001));; Queries can be templates, using $s1, $i2, etc. as parameters:(emacsqldb[:select[nameid]:frompatients:where(>weight$s1)]100);; => (("Jeff" 1000) ("Susan" 1001))
A query is a vector of keywords, identifiers, parameters, and data.
Thanks to parameters, these s-expression statements should not need to
be constructed dynamically at run-time.
The compilation rules are listed in the EmacSQL documentation so I
won’t repeat them in detail here. In short, lisp keywords become SQL
keywords, row-oriented information is always presented as vectors,
expressions are lists, and symbols are identifiers, except when
quoted.
Also, any readable lisp value can be stored in an
attribute. Integers are mapped to INTEGER, floats are mapped to REAL,
nil is mapped to NULL, and everything else is printed and stored as
TEXT. The specifics vary depending on the back-end.
Parameters
A symbol beginning with a dollar sign is a parameter. It has a type —
identifier (i), scalar (s), vector (v), schema (S) — and an argument
position.
[:select[$i1]:from$i2:where(<$i3$s4)]
Given the arguments name people age 21, three symbols and an
integer, it compiles to:
SELECTnameFROMpeopleWHEREage<21;
A vector parameter refers to rows to be inserted or as a set for an
IN expression.
[:insert-intopeople[nameage]:values$v1]
Given the argument (["Jim" 45] ["Jeff" 34]), a list of two rows,
this becomes,
When writing these expressions keep in mind the command
emacsql-show-last-sql. It will display in the minibuffer the SQL
result of the s-expression statement before the point.
Schemas
A table schema is a list whose first element is a column specification
vector (i.e. row-oriented information is presented as vectors). The
remaining elements are table constraints. Here are the examples from
the documentation,
;; No constraints schema with four columns:([nameidbuildingroom]);; Add some column constraints:([(name:unique)(idinteger:primary-key)buildingroom]);; Add some table constraints:([(name:unique)(idinteger:primary-key)buildingroom](:unique[buildingroom])(:check(>id0)))
In the handful of EmacSQL databases I’ve created for practice and
testing, I’ve put the schema in a global constant. A table schema is a
part of a program’s type specifications, and rows are instances of
that type, so it makes sense to declare schemas up top with things
like defstructs.
These schemas can be substituted into a SQL statement using a $S
parameter (capital “S” for Schema).
Everything I’ve discussed so far is restricted to the SQL statement
compiler. It’s completely independent of the back-end implementations,
themselves mostly handling strings of SQL statements.
SQLite Implementation Difficulties
A little over a year ago I wrote a pastebin webapp in
Elisp. I wanted to use SQLite as a back-end for storing pastes but
struggled to get the SQLite command shell, sqlite3, to cooperate with
Emacs. The problem was that all of the output modes except for “tcl”
are ambiguous. This includes the “csv” formatted output. TEXT values
can dump newlines, allowing rows to span an arbitrary number of lines.
They can dump things that look like the sqlite3 prompt, so it’s
impossible to know when sqlite3 is done printing results. I ultimately
decided the command shell was inadequate as an Emacs subprocess.
Recently there was some discussion from alexbenjm and Andres
Ramirez on an Elfeed post about using SQLite as an Elfeed back-end.
This inspired me to take another look and that’s when I came up with a
workaround for SQLite’s ambiguity: only store printed Elisp values for
TEXT values! With print-escape-newlines set, TEXT values no longer
span multiple lines, and I can use read to pull in data from
sqlite3. All of sqlite3’s output modes were now unambiguous.
However, after making significant progress I discovered an even bigger
issue: GNU Readline. The sqlite3 binary provided by Linux package
repositories is almost always compiled with Readline support. This
makes the tool much more friendly to use, but it’s a huge problem for
Emacs.
First, sqlite3 the command shell is not up to the same standards as
SQLite the database. Not by a long shot. In my short time working with
SQLite I’ve already discovered several bugs in the command shell. For
one, it’s not properly integrated with GNU Readline. There’s an
.echo meta-command that turns command echoing on and off. That is,
it repeats your command back to you. Useful in some circumstances,
though not mine. The bug is that this echo is separate from GNU
Readline’s echo. When Readline is active and .echo is enabled, there
are actually two echos. Turn it off and there’s one echo.
Pseudo-terminals
Under some circumstances, like when communicating over a pipe rather
than a PTY, Readline will mostly become deactivated. This would have
been a workaround, but when Readline is disabled sqlite3 heavily
buffers its output. This breaks any sort of interaction. Even worse,
on Windows stderr is not always unbuffered, so sqlite3’s
error messages may not appear for a long time (another bug).
Besides the problem of getting Readline to shut up, another problem is
getting Readline to stop acting on control characters. The first 32
characters in ASCII are control characters. A pseudo-terminal (PTY)
that is not in raw mode will immediately act upon any control
characters it sees. There’s no escaping them.
Emacs communicates with subprocesses through a PTY by default
(probably an early design mistake), limiting the kind of data that can
be transmitted. You can try this yourself in a comint mode sometime
where a subprocess is used (not a socket like SLIME). Fire up M-x
sql-sqlite (part of Emacs) and try sending a string containing byte
0x1C (28, file separator). You can type one by pressing C-q C-\.
Send that byte and the subprocess dies.
There are two ways to work around this. One is to use a pipe (bind
process-connection-type to nil). Pipes don’t respond to control
characters. This doesn’t work with sqlite3 because of the
previously-mentioned buffering issue.
The other way to work around this is to put the PTY in raw mode.
Unfortunately there’s no function to do this so you need to call
stty. Of course, this program needs to run on the same PTY, so a
start-process-shell-command is required.
(start-process-shell-commandnamebuffer"stty raw && <your command>")
Windows has neither stty nor PTYs (nor any of PTY’s issues) so
you’ll need to check the operating system before starting the process.
Even this still doesn’t work for sqlite3 because Readline itself will
respond to control characters. There’s no option to disable this.
There’s a package called esqlite that is also a SQLite
front-end. It’s built to use sqlite3 and therefore suffers from all of
these problems.
A Custom SQLite Binary
Since sqlite3 proved unreliable I developed my own protocol and
external program. It’s just a tiny bit of C that accepts a SQL string
and returns results as an s-expression. I’m not longer constrained to
storing readable values, but I’m still keeping that paradigm. First,
it keeps the C glue program simple and, more importantly, I can rely
entirely on the Emacs reader to parse the results. This makes
communication between Emacs and the subprocess as fast as it can
possibly be. The reader is faster than any possible Elisp program.
As I mentioned before, this C program is compiled when possible, and
otherwise a pre-built binary is fetched from my server (popular
platforms only, obviously). It’s likely EmacSQL will have at least one
working back-end on whatever you’re using.
Other Back-ends
Both PostgreSQL and MySQL are also supported, though these require the
user have the appropriate client programs installed (psql or mysql).
Both of these are much better behaved than sqlite3 and, with the
stty trick, each can reliably be used without any special help. Both
pass all of the unit tests, so, in theory, they’ll work just as well
as SQLite.
To use them with the example at the beginning of this article, require
emacsql-psql or emacsql-mysql, then swap emacsql-connect for the
constructors emacsql-psql or emacsql-mysql (along with the proper
arguments). All three of these constructors return an
emacsql-connection object that works with the same API.
EmacSQL only goes so far to normalize the interfaces to these
databases, so for any non-trivial program you may not be able to swap
back-ends without some work. All of the EmacSQL functions that operate
on connections are generic functions (EIEIO), so changing back-ends
will only have an effect on the program’s SQL statements. For example,
if you use q SQLite-ism (dynamic typing) it won’t translate to either
of the other databases should they be swapped in.
I’ll cover the connections API, and what it takes to implement a new
back-end, in a future post. Outside of the PTY caveats, it’s actually
very easy. The MySQL implementation is just 80 lines of code.
EmacSQL’s Future
I hope this becomes a reliable and trusted database solution that
other packages can depend upon. Twice so far, the pastebin demo and
Elfeed, I’ve really wanted something like this and, instead, ended up
having to hack together my own database.
I’ve already started a branch on Elfeed re-implementing its database
in EmacSQL. Someday it may become Elfeed’s primary database if I feel
there’s no disadvantage to it. EmacSQL builds SQLite with the
full-text search engine enabled, which opens to the door to a
powerful, fast Elfeed search API. Currently the main obstacle is
actually Elfeed’s database API being somewhat incompatible with ACID
database transactions — shortsightedness on my part!
*Update: Emacs 25.1 (released Sept. 2016) formally introduced
finalizers to Emacs Lisp. This article is left here for historical
purposes.
Problem: You have a special resource, such as a buffer or process,
associated with an Emacs Lisp object which is not managed by the
garbage collector. You want this resource to be cleaned up when the
owning lisp object is garbage collected. Unlike some other languages,
Elisp doesn’t provide finalizers for this job, so what do
you do?
Solution: This is Emacs Lisp. We can just add this feature to the
language ourselves!
I’ve already implemented this feature as a package called finalize,
available on MELPA. I will be using it as part of a larger, upcoming
project.
Process and buffers are special types of objects. Immediately after
instantiation these objects are added to a global list. They will
never become unreachable without explicitly being killed. The garbage
collector will never manage them for you.
This is a problem for APIs like those provided by the url package. The
functions url-retrieve and url-retrieve-synchronously create
buffers and hand them back to their callers. Ownership is transfered
to the caller and the caller must be careful to kill the buffer, or
transfer ownership again, before it returns. Otherwise the buffer is
“leaked.” The url package tries to manage this a little bit with
url-gc-dead-buffers, but this can’t be relied upon.
Another issue is when a process is started and is stored in a struct
or some other kind of object. There is probably a “close” function
that accepts one of these structs and kills the process. But if that
function isn’t called, due to a bug or an error condition, it will
become a “dangling” process. If the struct is completely lost, it will
probably be inconvenient to deal with the process — the “close”
function is no longer useful.
With Macros
A common way to deal with this problem is using a with- macro. This
macro establishes a resource, evaluates a body, and ensures the
resource is properly cleaned up regardless of the body’s termination
state. The latter is accomplished using unwind-protect. For example,
with-temp-buffer,
;; Fetch the first 10 bytes of foo.txt(with-temp-buffer(insert-file-contents"foo.txt"nil010)(buffer-string))
This expands (roughly) to the following expression.
For dealing with open files, Common Lisp has with-open-stream. It
establishes a binding for a new stream over its body and ensures the
stream is closed when the body is complete. There’s no chance for a
stream to be left open, leaking a system resource.
However, with- macros aren’t useful in asynchronous situations. In
Emacs this would be the case for asynchronous sub-processes, such as
an attached language interpreter. The extent of the process goes
beyond a single body.
Finalizers
What would really be useful is to have a callback — a finalizer —
that runs when an object is garbage collected. This ensures that the
resource will not outlive its owner, restoring management back to the
garbage collector. However, Emacs provides no such hook.
Fortunately this feature can be built using weak hash tables and the
post-gc-hook, a list of functions that are run immediately after
garbage collection.
Weak References
I’ve discussed before how to create weak references in Elisp.
The only weak references in Emacs are built into weak hash tables.
Normally the language provides weak references first and hash tables
are built on top of them. With Emacs we do this backwards.
The make-hash-table function accepts a key argument :weakness to
specify how strongly keys and values should be held by the table. To
make a weak reference just create a hash table of size 1 and set
:weakness to t.
The same trick can be used to detect when an object is garbage
collected. If the result of deref is nil, then the object was
garbage collected. (Or the weakly-referenced object is nil, but this
object will never be garbage collected anyway.)
To check if we need to run a finalizer all we have to do is create a
weak reference to the object, then check the reference after garbage
collection. This check can be done in a post-gc-hook function.
Registration
To avoid cluttering up post-gc-hook with one closure per object
we’ll keep a register of all watched objects.
Now to try it out. Create a process, stuff it in a vector (like a
defstruct), register delete-process as a finalizer, and, for the
sake of demonstration, immediately forget the vector.
;;; -*- lexical-binding: t; -*-(let((process(start-process"ping"nil"ping""localhost")))(register(vectorprocess)(lambda()(delete-processprocess))));; Assuming the garbage collector has not already run.(get-process"ping");; => #<process ping>;; Force garbage collection.(garbage-collect)(get-process"ping");; => nil
The garbage collector killed the process for us!
There are some problems with this implementation. Using cl-remove-if
is unwise in a post-gc-hook function. It allocates lots of new cons
cells but garbage collection is inhibited while the function is run.
The docstring warns us:
Garbage collection is inhibited while the hook functions run, so be
careful writing them.
Similarly, all of the finalizers are run within the context of this
memory-sensitive hook. Instead they should be delayed until the next
evaluation turn (i.e. run-at-time of 0). Some of the finalizers
could also fail, which would cause the remaining finalizers to never
run. The real implementation deals with all of these issues.
A major drawback to these Emacs Lisp finalizers compared to other
languages is that the actual object is not available. We don’t know
it’s getting collected until after it’s already gone. This solves the
object resurrection problem, but it’s darn inconvenient. One possible
workaround in the case of defstructs and EIEIO objects is to make a
copy of the original object (copy-sequence or clone) and run the
finalizer on the copy as if it was the original.
The Real Implementation
The real implementation is more carefully namespaced and its API has
just one function: finalize-register. It works just like register
above but it accepts &rest arguments to be passed to the finalizer.
This makes the registration call simpler and avoids some
significant problems with closures.
To make things cleaner for EIEIO classes there’s also a finalizable
mixin class that ensures the finalize generic function is called on
a copy of the object (the original object is gone) when it’s garbage
collected.
Here’s how it would be used for the same “pinger” concept, this time
as an EIEIO class. An advantage here is that anyone can manually call
finalize early if desired.
A couple of weeks ago I wrote a library to measure the retained memory
footprint of arbitrary Elisp objects for the purposes of optimization.
It’s called Caliper.
The reason I wanted this was that I came across a post on reddit where
someone had scraped 217,000 Jeopardy! questions from
J! Archive and dumped them out into a single, large JSON
file. The significance of the effort is that it dealt with some of the
inconsistencies of J! Archive’s data presentation, normalizing them
for the JSON output.
When I want to examine a JSON dataset like this I have three preferred
options:
Load it into a browser page and poke at it from JavaScript remotely
with Skewer. With the JSON text weighing in at 53MB and
with such a large object count, I decided this was too large for a
browser page. It definitely could be done, it’s just that the
browser is not the place to be working on large datasets.
Load it into Clojure. I’m familiar with Clojure’s
data.json. This is not a bad choice, but there’s
something else I always reach for first if I can.
Load it into Emacs using json.el (part of Emacs). This is what I
ended up doing.
Here, jeopardy is bound to a vector of 216,930 association lists
(alists). I’m curious exactly how much heap memory this data structure
is using. To find out, we need to walk the data structure and sum the
sizes of everything we come across. However, care must be taken not to
count the identical objects twice, such as symbols, which, being
interned, appear many times in this data.
Measuring Object Sizes
This is lisp so let’s start with the cons cell. A cons cell is just a
pair of pointers, called car and cdr.
These are used to assemble lists.
So a cons cell itself — the shallow size — is two words: 16 bytes
on a 64-bit operating system. To make sure Elisp doesn’t happen to
have any additional information attached to cons cells, let’s take a
look at the Emacs source code.
structLisp_Cons{/* Car of this cons cell. */Lisp_Objectcar;union{/* Cdr of this cons cell. */Lisp_Objectcdr;/* Used to chain conses on a free list. */structLisp_Cons*chain;}u;};
The return value from garbage-collect backs this up. The first value
after each type is the shallow size of that type. From here on, all
values have been computed for 64-bit Emacs running on x86-64
GNU/Linux.
A Lisp_Object is just a pointer to a lisp object. The retained
size of a cons cell is its shallow size plus, recursively, the
retained size of the objects in its car and cdr.
Integers and Floats
Integers are a special case. Elisp uses what is called tagged
integers. They’re not heap-allocated objects. Instead they’re
embedded inside the object pointers. That is, those Lisp_Object
pointers in Lisp_Cons will hold integers directly. This means to
Caliper integers have retained size of 0. We can use this to verify
Caliper’s return value for cons cells.
Tagged integers are fast and save on memory. They also compare
properly with eq, which is just a pointer (identity) comparison.
However, because a few bits need to be reserved for differentiating
them from actual pointers these integers have a restricted dynamic
range.
Floats are not tagged and exist as immutable objects in the heap.
That’s why eql is still useful in Elisp — it’s like eq but will
handle numbers properly. (By convention you should use eql for
integers, too.)
Symbols and Strings
Not counting the string’s contents, a string’s base size is 32 bytes
according to garbage-collect. The length of the string can’t be
used here because that counts characters, which vary in size. There’s
a string-bytes function for this. A string’s size is 32 plus its
string-bytes value.
As you can see from above, symbols are huge. Without even counting
either the string holding the name of the symbol or the symbol’s
plist, a symbol is 48 bytes.
(caliper-object-size'hello);; => 1038
This 1,038 bytes is a little misleading. The symbol itself is 48
bytes, the string "hello" is 37 bytes, and the plist is nil. The
retained size of nil is significant. On my system, nil’s plist has 4
key-value pairs, which themselves have retained sizes. When examining
symbols, caliper doesn’t care if they’re interned or not, including
symbols like nil and t. However, nil is only counted once, so it
will have little impact on a large data structure.
Miscellaneous
Outside of vectors, measuring object sizes starts to get fuzzy. For
example, it’s not possible to examine the exact internals of a hash
table from Elisp. We can see its contents and the number of elements
it can hold without re-sizing, but there’s intermediate structure
that’s not visible. Caliper makes rough estimates for each of these
types.
Circularity and Double Counting
To avoid double counting objects, a hash table with a test of eq is
dynamically bound by the top level call. It’s used like a set. Before
an object is examined, the hash table is checked. If the object is
listed, the reported size is 0 (it consumes no additional space than
already accounted for).
This automatically solves the circularity problem. There’s no way we
can traverse into the same data structure a second time because we’ll
stop when we see it twice.
Using Caliper
So what’s the total retained size of the jeopardy structure? About
124MB.
(caliper-object-sizejeopardy);; => 130430198
For fun, let’s see if how much we can improve on this.
json.el will return alists for objects by default, but this can be
changed by setting json-object-type to something else. Initially I
thought maybe using plists instead would save space, but I later
realized that plists use exactly the same number of cons cells as
alists. If this doesn’t sound right, try to picture the cons cells
in your head (an exercise for the reader).
They’re (now) plists of 7 pairs. All of the keys are symbols, and, as
such, are interned and consuming very little memory. All of the values
are strings. Surely we can do better here. The strings can be interned
and the numbers can be turned into tagged integers. The :category
values would probably be good candidates for conversion into symbols.
Here’s an interesting fact about Jeopardy! that can be exploited for
our purposes. While Jeopardy! covers a broad range of trivia,
it does so very shallowly. The same answers appear many
times. For example, the very first answer from our dataset,
Copernicus, appears 14 times. That makes even the answers good
candidates for interning.
A string pool is trivial to implement. Just use a weak, equal hash
table to track strings. Making it weak keeps it from leaking memory by
holding onto strings for longer than necessary.
That’s down to 79MB of memory. Not bad! If we print-circle this,
taking advantage of string interning in the printed representation, I
wonder how it compares to the original JSON.
Byte-code compilation is an underdocumented — and in the case of the
recent lexical binding updates, undocumented — part of Emacs. Most
users know that Elisp is usually compiled into a byte-code saved to
.elc files, and that byte-code loads and runs faster than uncompiled
Elisp. That’s all users really need to know, and the GNU Emacs Lisp
Reference Manual specifically discourages poking around too much.
People do not write byte-code; that job is left to the byte
compiler. But we provide a disassembler to satisfy a cat-like
curiosity.
Screw that! What if I want to handcraft some byte-code myself? :-) The
purpose of this article is to introduce the internals of Elisp
byte-code interpreter. I will explain how it works, why lexically
scoped code is faster, and demonstrate writing some byte-code by hand.
The Humble Stack Machine
The byte-code interpreter is a simple stack machine. The stack holds
arbitrary lisp objects. The interpreter is backwards compatible but
not forwards compatible (old versions can’t run new byte-code). Each
instruction is between 1 and 3 bytes. The first byte is the opcode and
the second and third bytes are either a single operand or a single
intermediate value. Some operands are packed into the opcode byte.
As of this writing (Emacs 24.3) there are 142 opcodes, 6 of which have
been declared obsolete. Most opcodes refer to commonly used built-in
functions for fast access. (Looking at the selection, Elisp really is
geared towards text!) Considering packed operands, there are up to 27
potential opcodes unused, reserved for the future.
opcodes 48 - 55
opcode 97
opcode 128
opcodes 169 - 174
opcodes 180 - 181
opcodes 183 - 191
The easiest place to access the opcode listing is in
bytecomp.el. Beware that some of the opcode comments are
currently out of date.
Segmentation Fault Warning
Byte-code does not offer the same safety as normal Elisp. Bad
byte-code can, and will, cause Emacs to crash. You can try out for
yourself right now,
Or evaluate the code manually in a buffer (save everything first!),
(#[0"\300\207"[]0])
This segfault, caused by referencing beyond the end of the constants
vector, is not an Emacs bug. Doing a boundary test would slow down
the byte-code interpreter. Not performing this test at run-time is a
practical engineering decision. The Emacs developers have instead
chosen to rely on valid byte-code output from the compiler, making a
disclaimer to anyone wanting to write their own byte-code,
You should not try to come up with the elements for a byte-code
function yourself, because if they are inconsistent, Emacs may crash
when you call the function. Always leave it to the byte compiler to
create these objects; it makes the elements consistent (we hope).
You’ve been warned. Now it’s time to start playing with firecrackers.
The Byte-code Object
A byte-code object is functionally equivalent to a normal Elisp vector
except that it can be evaluated as a function. Elements are accessed
in constant time, the syntax is similar to vector syntax ([...] vs.
#[...]), and it can be of any length, though valid functions must
have at least 4 elements.
There are two ways to create a byte-code object: using a byte-code
object literal or with make-byte-code.
Like vector literals, byte-code literals don’t need to be
quoted.
The parameter list takes on two different forms depending on if the
function is lexically or dynamically scoped. If the function is
dynamically scoped, the argument list is exactly what appears in lisp
code.
(byte-compile(lambda(ab&optionalc)));; => #[(a b &optional c) "\300\207" [nil] 1]
There’s really no shorter way to represent the parameter list because
preserving the argument names is critical. Remember that, in dynamic
scope, while the function body is being evaluated these variables are
globally bound (eww!) to the function’s arguments.
When the function is lexically scoped, the parameter list is packed
into an Elisp integer, indicating the counts of the different kinds of
parameters: required, &optional, and &rest.
The least significant 7 bits indicate the number of required
arguments. Notice that this limits compiled, lexically-scoped
functions to 127 required arguments. The 8th bit is the number of
&rest arguments (up to 1). The remaining bits indicate the total
number of optional and required arguments (not counting &rest). It’s
really easy to parse these in your head when viewed as hexadecimal
because each portion almost always fits inside its own “digit.”
The names of the arguments don’t matter in lexical scope: they’re
purely positional. This tighter argument specification is one of the
reasons lexical scope is faster: the byte-code interpreter doesn’t
need to parse the entire lambda list and assign all of the variables
on each function invocation.
Unibyte String Byte-code
The second element is a unibyte string — it strictly holds octets and
is not to be interpreted as any sort of Unicode encoding. These
strings should be created with unibyte-string because string may
return a multibyte string. To disambiguate the string type to the lisp
reader when higher values are present (> 127), the strings are printed
in an escaped octal notation, keeping the string literal inside the
ASCII character set.
(unibyte-string100200250);; => "d\310\372"
It’s unusual to see a byte-code string that doesn’t end with 135
(#o207, byte-return). Perhaps this should have been implicit? I’ll
talk more about the byte-code below.
Constants Vector
The byte-code has very limited operands. Most operands are only a few
bits, some fill an entire byte, and occasionally two bytes. The meat
of the function that holds all the constants, function symbols, and
variables symbols is the constants vector. It’s a normal Elisp vector
and can be created with vector or a vector literal. Operands
reference either this vector or they index into the stack itself.
(byte-compile(lambda(ab)(my-funcba)));; => #[(a b) "\302\134\011\042\207" [b a my-func] 3]
Note that the constants vector lists the variable symbols as well as
the external function symbol. If this was a lexically scoped function
the constants vector wouldn’t have the variables listed, being only
[my-func].
Maximum Stack Usage
This is the maximum stack space used by this byte-code. This value can
be derived from the byte-code itself, but it’s pre-computed so that
the byte-code interpreter can quickly check for stack overflow.
Under-reporting this value is probably another way to crash Emacs.
Docstring
The simplest component and completely optional. It’s either the
docstring itself, or if the docstring is especially large it’s a cons
cell indicating a compiled .elc and a position for lazy access. Only
one position, the start, is needed because the lisp reader is used to
load it and it knows how to recognize the end.
Interactive Specification
If this element is present and non-nil then the function is an
interactive function. It holds the exactly contents of interactive
in the uncompiled function definition.
The interactive expression is always interpreted, never byte-compiled.
This is usually fine because, by definition, this code is going to be
waiting on user input. However, it slows down keyboard macro playback.
Opcodes
The bulk of the established opcode bytes is for variable, stack, and
constant access opcodes, most of which use packed operands.
129, 192-255 : (constant) direct constants vector access
Except for the last item, each kind of instruction comes in sets of 8.
The nth such instruction means access the nth thing. For example, the
instruction “2” copies the third stack item to the top of the stack.
An instruction of “9” pushes onto the stack the value of the
variable named by the second element listed in the constants vector.
However, the 7th and 8th such instructions in each set take an operand
byte or two. The 7th instruction takes a 1-byte operand and the 8th
takes a 2-byte operand. A 2-byte operand is written in little-endian
byte-order regardless of the host platform.
For example, let’s manually craft an instruction that returns the
value of the global variable foo. Each opcode has a named constant
of byte-X so we don’t have to worry about their actual byte-code
number.
(require'bytecomp); named opcodes(defvarfoo"hello")(defalias'get-foo(make-byte-code#x000; no arguments(unibyte-string(+0byte-varref); ref variable under first constantbyte-return); pop and return[foo]; constants1)); only using 1 stack space(get-foo);; => "hello"
Ta-da! That’s a handcrafted byte-code function. I left a “+ 0” in
there so that I can change the offset. This function has the exact
same behavior, it’s just less optimal,
(defalias'get-foo(make-byte-code#x000(unibyte-string(+3byte-varref); 4th form of varrefbyte-return)[nilnilnilfoo]1))
If foo was the 10th constant, we would need to use the 1-byte
operand version. Again, the same behavior, just less optimal.
(defalias'get-foo(make-byte-code#x000(unibyte-string(+6byte-varref); 7th form of varref9; operand, (constant index 9)byte-return)[nilnilnilnilnilnilnilnilnilfoo]1))
Dynamically-scoped code makes heavy use of varref but
lexically-scoped code rarely uses it (global variables only), instead
relying heavily on stack-ref, which is faster. This is where the
different calling conventions come into play.
Calling Convention
Each kind of scope gets its own calling convention. Here we finally
get to glimpse some of the really great work by Stefan Monnier
updating the compiler for lexical scope.
Dynamic Scope Calling Convention
Remembering back to the parameter list element of the byte-code
object, dynamically scoped functions keep track of all its argument
names. Before executing a function the interpreter examines the lambda
list and binds (varbind) every variable globally to an argument.
If the caller was byte-compiled, each argument started on the stack,
was popped and bound to a variable, and, to be accessed by the
function, will be pushed back right onto the stack (varref). There’s
a lot of argument indirection for each function call.
Lexical Scope Calling Convention
With lexical scope, the argument names are not actually bound for the
evaluation byte-code. The names are completely gone because the
compiler has converted local variables into stack offsets.
When calling a lexically-scoped function, the byte-code interpreter
examines the integer parameter descriptor. It checks to make sure the
appropriate number of arguments have been provided, and for each
unprovided &optional argument it pushes a nil onto the stack. If the
function has a &rest parameter, any extra arguments are popped off
into a list and that list is pushed onto the stack.
From here the function can access its arguments directly on the stack
without any named variable misdirection. It can even consume them
directly.
;; -*- lexical-binding: t -*-(defunfoo(x)x)(symbol-function#'foo);; => #[#x101 "\207" [] 2]
The byte-code for foo is a single instruction: return. The
function’s argument is already on the stack so it doesn’t have to do
anything. Strangely the maximum stack usage element is wrong here (2),
but it won’t cause a crash.
;; (As of this writing `byte-compile' always uses dynamic scope.)(byte-compile'foo);; => #[(x) "\010\207" [x] 1]
It takes longer to set up (x is implicitly bound), it has to make an
explicit variable dereference (varref), then it has to clean up by
unbinding x (implicit unbind). It’s no wonder lexical scope is
faster!
Note that there’s also a disassemble function for examining
byte-code, but it only reveals part of the story.
The Elisp byte-compiler has an intermediate language called lapcode
(“Lisp Assembly Program”), which is much easier to optimize than
byte-code. It’s basically an assembly language built out of
s-expressions. Opcodes are referenced by name and operands, including
packed operands, are handled whole. Each instruction is a cons cell,
(opcode . operand), and a program is a list of these.
We didn’t have to worry about which form of varref we were using or
even how to encode a 2-byte operand. The lapcode “assembler” took care
of that detail.
Project Ideas?
The Emacs byte-code compiler and interpreter are fascinating. Having
spent time studying them I’m really tempted to build a project on top
of it all. Perhaps implementing a programming language that targets
the byte-code interpreter, improving compiler optimization, or, for a
really big project, JIT compiling Emacs byte-code.
I’ve stated before that one of the unique features of Emacs Lisp is
that its closures are readable. Closures can be serialized by the
printer and read back in with the reader. I am unaware of any other
programming language that has this feature. In fact it’s essential for
Elisp byte-code compilation because byte-compiled Elisp files are
merely s-expressions of byte-code dumped out as source.
Lisp Printing
The Lisp family of languages are homoiconic. Lisp source code is
written in the syntax of its own data structures, s-expressions. Since
a compiler/interpreter is usually provided at run-time, a consequence
of this is that reading and printing are a fundamental feature of
Lisps. A value can be handed to the printer, which will serialize the
value into an s-expression as a sequence of characters. Later on the
reader can parse the s-expression back into an equal value.
To compare, JavaScript originally had half of this in place.
JavaScript has convenient object syntax for defining an associative
array, known today as JSON. The eval function could (dangerously) be
used as a reader for parsing a string containing JSON-encoded data
into a value. But until JSON.stringify() became standard, developers
had to write their own printer. Lisp s-expression syntax is much more
powerful (and complicated) than JSON, maintaining
both identity and cycles (e.g. *print-circle*).
Not all values can be read. They’ll still print (when *print-readably*
is nil) but will do so using special syntax that will signal an error
in the reader: #<. For example, in Emacs Lisp buffers cannot be
serialized so they print using this syntax.
It doesn’t matter what’s between the angle brackets, or even that
there’s a closing angle bracket. The reader will signal an error as
soon as it hits a #<.
Almost Everything Prints Readably
Elisp has a small set of primitive data types. All of these primitive
types print readably:
integer (1024, ?a)
float (1.7)
cons/list ((...))
vector (one-dimensional, [...])
bool-vector (#&n"...")
string ("...")
char-table (#^[...])
hash-table (readable as of Emacs 23.3, #s(hash-table ...))
byte-code function object (#[...])
symbol
Here are all the non-readable types. Each one has a good reason for
not being serializable.
buffer
process (external state)
frame (user interface element)
marker (live, automatically updates)
overlay (belongs to a buffer)
built-in functions (native code)
user-ptr (opaque pointers from Emacs 25 dynamic modules)
And that’s it. Every other value in Elisp is constructed from one or
more of these primitives, including keymaps, functions, macros, syntax
tables, defstruct structs, and EIEIO objects. This means that as
long as these values don’t refer to an unreadable value, they
themselves can be printed.
An interesting note here is that, unlike the Common Lisp Object System
(CLOS), EIEIO objects are readable by default. To Elisp they’re just
vectors, so of course they print. CLOS objects are unreadable without
manually defining a print method per class.
Elisp Closures
Elisp got lexical scoping in Emacs 24, released in June 2012. It’s now
one of the relatively few languages to have both dynamic and lexical
scope. Like Common Lisp, variables declared with defvar (and family)
continue to have dynamic scope. For backwards compatibility with old
Lisp code, lexical scope is disabled by default. It’s enabled for a
specific file or buffer by setting lexical-binding to non-nil.
With lexical scope, anonymous functions become closures, a powerful
functional programming primitive: a function plus a captured lexical
environment. It also provides some performance benefits. In my own
tests, compiled Elisp with lexical scope enabled is about 10% to 15%
faster than with the default dynamic scope.
What do closures look like in Emacs Lisp? It takes on two forms
depending on whether the closure is compiled or not. For example,
consider this function, foo, that takes two arguments and returns a
closure that returns the first argument.
An uncompiled closure is a list beginning with the symbol closure.
The second element is the lexical environment, the third is the
argument list (lambda list), and the rest is the body of the function.
Here we can see that both x and y have been “closed over.” This is
a little bit sloppy because the function never makes use of y.
Capturing it has a few problems.
The closure has a larger footprint than necessary.
Values are held longer than necessary, delaying collection.
It affects the readability of the closure, which I’ll get to later.
Fortunately the compiler is smart enough to see this and will avoid
capturing unused variables. To prove this, I’ve now compiled foo so
that it returns a compiled closure.
(foo:bar:ignored);; => #[0 "\300\207" [:bar] 1]
What’s returned here is a byte-code function object, with the #[...]
syntax. It has these elements:
The function’s lambda list (zero arguments)
Byte-codes stored in a unibyte string
Constants vector
Maximum stack space needed by this function
Notice that the lexical environment has been captured in the constants
vector, specifically noting the lack of :ignored in this vector. The
compiler didn’t capture it.
For those curious about the byte-code here’s an explanation. The
string syntax shown is in octal, representing a string containing two
bytes: 192 and 135. The
Elisp byte-code interpreter is stack-based. The 192
(constant 0) says to push the first constant onto the stack. The 135
(return) says to pop the top element from the stack and return it.
(coerce"\300\207"'list);; => (192 135)
The Readable Closures Catch
Since closures are byte-code function objects, they print readably.
You can capture an environment in a closure, serialize it, read it
back in, and evaluate it. That’s pretty cool! This means closures can
be transmitted to other Emacs instances in a multi-processing setup
(i.e. Elnode, Async)
The catch is that it’s easy to accidentally capture an unreadable
value, especially buffers. Consider this function bar which uses a
temporary buffer as an efficient string builder. It returns a closure
that returns the result. (Weird, but stick with me here!)
The temporary buffer is mistakenly captured in the closure making it
unreadable, but only in its uncompiled form. This creates the
awkward situation where compiled and uncompiled code has different
behavior.
This past week I added Clojure-style multimethods to Emacs
Lisp through a package I call predd (predicate dispatch). I
believe it is Elisp’s very first complete multiple dispatch object
system! That is, methods are dispatched based on the dynamic,
run-time type of more than one of its arguments.
(Unfortunately I was unaware of the
other Clojure-style multimethod library when I wrote mine.
However, my version is much more complete, has better performance,
and is public domain.)
As of version 23.2, Emacs includes a CLOS-like object system cleverly
named EIEIO. While CLOS (Common Lisp Object System) is multiple
dispatch, EIEIO is, like most object systems, only single dispatch.
The predd package is also very different than my other Elisp object
system, @, which was prototype based and, therefore, also single
dispatch (and comically slow).
The Clojure multimethods documentation provides a good
introduction. The predd package works almost exactly the same way,
except that due to Elisp’s lack of namespacing the function names are
prefixed with predd-. Also different is that the optional hierarchy
(h) argument is handled by the dynamic variable predd-hierarchy,
which holds the global hierarchy.
Combination Example
To define a multimethod, pick a name and give it a classifier
function. The classifier function will look at the method’s arguments
and return a dispatch value. This value is used to select a
particular method. What makes predd a multiple dispatch system is the
dispatch value can be derived from any number of methods arguments.
Because the dispatch value is computed at run-time this is called a
late binding.
Here I’m going to define a multimethod called combine that takes two
arguments. It combines its arguments appropriately depending on their
dynamic run-time types.
(predd-defmulticombine(lambda(ab)(vector(type-ofa)(type-ofb)))"Appropriately combine A and B.")
The classifier uses type-of, an Elisp built-in, to examine its
argument types. It returns them as tuple in the form of a vector. The
classifier of a method can be accessed with predd-classifier, which
I’ll use to demonstrate what these dispatch values will look like.
I chose a vector for the dispatch value because I like the bracket
style when defining methods (you’ll see below). The dispatch value can
be literally anything that equal knows how to compare, not just
vectors. Note that it’s actually faster to create a list than a vector
up to a length of about 6, so this multimethod would be faster if the
classifier returned a list — or even better: a single cons.
Now define some methods for different dispatch values.
(combine12); => 3(combine"a""b"); =>"ab"(combine'(12)'(34)); => (1 2 3 4)(combine1'(34)); error: "No method found in combine for [integer cons]"
Notice in the last case it didn’t know how to combine these two types,
so it threw an error. In this simple example where we’re only calling
a single function, so rather than use the predd-defmethod macro
these methods can be added directly with the predd-add-method
function. This has the exact same result except that it has slightly
better performance (no wrapper functions).
Hmmm, the + function is already polymorphic. It seamlessly operates
on both floats and integers. So far it seems there’s no way to exploit
this with multimethods. Fortunately we can solve this by defining our
own ad hoc hierarchy using predd-derive. Both integers and floats
are a kind of number. It’s important to note that type-of never
returns number. We’re introducing that name here ourselves.
(type-of1.0); => float(predd-derive'integer'number)(predd-derive'float'number);; Types can derive from multiple parents, like multiple inheritance(predd-derive'integer'exact)(predd-derive'float'inexact)
This says that integer and float are each a kind of number. Now
we can use number in a dispatch value. When it sees something like
[float integer] it knows that it matches [number number].
We can check the hierarchy explicitly with predd-isa-p (like
Clojure’s isa?). It compares two values just like equal, but it
also accounts for all predd-derive declarations. Because of this
extra concern, unlike equal, predd-isa-p is not commutative.
(Remember that 0 is truthy in Elisp.) The integer returned is a
distance metric used by method dispatch to determine which values are
“closer” so that the most appropriate method is selected.
You might be worried that introducing number will make the
multimethod slower. Examining the hierarchy will definitely have a
cost after all. Fortunately predd has a dispatch cache, so
introducing this indirection will have no additional performance
penalty after the first call with a particular dispatch value.
Struct Example
Something that really sets these multimethods apart from other object
systems is a lack of concern about encapsulation — or really about
object data in general. That’s the classifier’s concern. So here’s an
example of how to combine predd with defstruct from cl/cl-lib.
Imagine we’re making some kind of game where each of the creatures is
represented by an actor struct. Each actor has a name, hit points,
and active status effects.
The defstruct macro has a useful inheritance feature that we can
exploit for our game to create subtypes. The parent accessors will
work on these subtypes, immediately providing some (efficient)
polymorphism even before multimethods are involved.
As a side note: this isn’t necessarily the best way to go about
modeling a game. We probably shouldn’t be relying on inheritance too
much, but bear with me for this example.
Say we want an attack method for handling attacks between different
types of monsters. Elisp structs have a very useful property by
default: they’re simply vectors whose first element is a symbol
denoting its type. We can use this in a multimethod classifier.
(make-player);; => [cl-struct-player "Unknown" 100 nil nil](predd-defmultiattack(lambda(attackervictim)(vector(arefattacker0)(arefvictim0)))"Perform an attack from ATTACKER on VICTIM.")
Let’s define a base case. This will be overridden by more specific
methods (determined by that distance metric).
We could have instead used :default for the dispatch value, which is
a special catch-all value. The actor-hp function will signal an
error for any victim non-actors anyway. However, not using :default
will force both argument types to be checked. It will also demonstrate
specialization for the example.
However, before we can make use of this we need to teach predd about
the relationship between these structs. It doesn’t check defstruct
hierarchies. This step is what makes combining defstruct and predd
a little unwieldy. A wrapper macro is probably due for this.
If the monster applied a status effect in addition to the default
attack behavior then CLOS-style method combination would be far more
appropriate here (if only it was available in Elisp). The method would
instead be defined as an “after” method and it would automatically run
in addition to the default behavior.
If I was actually building a system combing structs and predd, I would
be using this helper function for building classifiers. It returns a
dispatch value for selected arguments.
;;; -*- lexical-binding: t; -*-(defunstruct-classifier(&restpattern)(lambda(&restargs)(loopforselect-pinpatternandarginargswhenselect-pcollect(eltarg0))));; Takes 3 arguments, dispatches on the first 2 argument types.(predd-defmultispeak(struct-classifierttnil));; Messages sent to the player are displayed.(predd-defmethodspeak'(cl-struct-actorcl-struct-player)(fromtomessage)(message"%s says %s."(actor-namefrom)message))
The Future
As of this writing there isn’t yet a prefer-method for
disambiguating equally preferred dispatch values. I will add it in the
future. I think prefer-method gets unwieldy quickly as the type
hierarchy grows, so it should be avoided anyway.
I haven’t put predd in MELPA or otherwise published it yet. That’s
what this post is for. But I think it’s ready for prime time, so feel
free to try it out.
A couple of months ago I wrote an Emacs Lisp wrapper for the
reddit API. I didn’t put it in MELPA,
not yet anyway. If anyone is finding it useful I’ll see about getting
that done. My intention was give it some exercise and testing before
putting it out there for people to use, locking down the API. You can
find it here,
Except for logging in, the library is agnostic about the actual API
endpoints themselves. It just knows how to translate between Elisp and
the reddit API protocol. This makes the library dead simple to use. I
had considered supporting OAuth2 authentication rather than
password authentication, but reddit’s OAuth2 support is pretty rough
around the edges.
Library Usage
The reddit API has two kinds of endpoints, GET and POST, so there are
really only three functions to concern yourself with.
reddit-login
reddit-get
reddit-post
And one variable,
reddit-session
The reddit-login function is really just a special case of
reddit-post. It returns a session value (cookie/modhash tuple) that
is used by the other two functions for authenticating the user. Just
as you get automatically with almost all Elisp data structures —
probably more so than any other popular programming language — it
can be serialized with the printer and reader, allowing a reddit
session to be maintained across Emacs sessions.
The return value of reddit-login generally doesn’t need to be
captured. It automatically sets the dynamic variable reddit-session,
which is what the other functions access for authentication. This can
be bound with let to other session values in order to switch between
different users.
Both reddit-get and reddit-post take an endpoint name and a list
of key-value pairs in the form of a property list (plist). (The
api-type key is automatically supplied.) They each return the JSON
response from the server in association list (alist) form. The actual
shape of this data matches the response from reddit, which,
unfortunately, is inconsistent and unspecified, so writing any sort of
program to operate on the API requires lots of trial and error. If the
API responded with an error, these functions signal a reddit-error.
Typical usage looks like so. Notice that values need not be only
strings; they just need to print to something reasonable.
;; Login first(reddit-login"your-username""your-password");; Subscribe to a subreddit(reddit-post"/api/subscribe"'(:sr"t5_2s49f":actionsub));; Post a comment(reddit-post"/api/comment/"'(:text"Hello world.":thing_id"t1_cd3ar7y"))
For plists keys I considered automatically converting between dashes
and underscores so that the keywords could have Lisp-style names. But
the reddit API is inconsistent, using both, so there’s no correct way
to do this.
To further refine the API it might be worth defining a function for
each of the reddit endpoints, forming a facade for the wrapper
library, hiding way the plist arguments and complicated responses.
That would eliminate the trial and error of using the API.
(defunreddit-api-comment(parentcomment)(if(nullreddit-session)(error"Not logged in.");; TODO: reduce the return value into a thing/struct(reddit-post"/api/comment/"'(:thing_idparent:textcomment))))
Furthermore there could be defstructs for comments, posts, subreddits,
etc. so that the “thing” ID stuff is hidden away. This is basically
what was already done for sessions out of necessity. I might add these
structs and functions someday but I don’t currently have a need for
it.
It would be neat to use this API to create an interface to reddit from
within Emacs. I imagine it might look like one of the Emacs mail
clients, or like Elfeed. Almost everything, including
viewing image posts within Emacs, should be possible.
A few months ago we decided to institute a self-post-only Sunday. All
day Sunday, midnight to midnight Eastern time, only self-posts are
allowed in the subreddit. One of the other moderators was turning this
on and off manually, so I offered to write a bot to do the job. There
weren’t any Lisp wrappers yet (though raw4j could be used
with Clojure), so I decided to write one.
As mentioned before, the reddit API leaves a lot to be desired. It
randomly returns errors, so a correct program needs to be prepared to
retry requests after a short delay, depending on the error. My
particular annoyance is that the /api/site_admin endpoint requires
that most of its keys are supplied, and it’s not documented which ones
are required. Even worse, there’s no single endpoint to get all of the
required values, the key names between endpoints are inconsistent, and
even the values themselves can’t be returned as-is, requiring
massaging/fixing before returning them back to the API.
Today is Thanksgiving in the United States. It also happens to be
Hanukkah. There’s been news going around that Thanksgiving and
Hanukkah will not coincide again for about 80,000 years. This
sounded somewhat unbelievable to me because
the Gregorian repeats every 400 years. I decided to
compute it for myself to double-check this figure.
I’m not Jewish and I know very little about Hanukkah, so I had to look
it up. After learning that Hanukkah is based on the Hebrew calendar,
the rumors were sounding more believable. The Hebrew calendar repeats
every 689,472 Hebrew years. This means the correspondence between
Gregorian and Hebrew calendars is about 14 billion years.
That 80,000 seems lowball.
Since I decided to use Emacs Lisp for the computation, I fortunately
was able to ignore all the unfamiliar, complicated rules for the
Hebrew calendar: Emacs knows how to compute Hebrew dates. It can be
accessed through the function calendar-hebrew-date-string.
Hanukkah begins on the 25th of Kislev, so I can write a
quick-and-dirty function to detect if a date is the first day of
Hanukkah.
(defunhanukkah-p(date)"Return non-nil if DATE is Hanukkah."(string-match-p"^Kislev 25"(calendar-hebrew-date-stringdate)))
Next I need a function to compute Thanksgiving, which is really
simple. Thanksgiving falls on the fourth Thursday of November.
(defunthanksgiving(year)"Return the date of Thanksgiving for YEAR."(loopfordayfrom1upto7when(=4(calendar-day-of-week`(11,day,year)))return`(11,(+day21),year)))
If there was no calendar-day-of-week I could compute it using
Zeller’s algorithm, which I already happen to have
implemented,
(defuncal/day-of-week(yearmonthday)"Return day of week number (0-7)."(let*((Y(if(<month3)(1-year)year))(m(1+(mod(+month9)12)))(y(modY100))(c(/Y100)))(mod(+day(floor(-(*26m)2)10)y(/y4)(/c4)(*-2c))7)))
Now for each year find Thanksgiving and test it for Hanukkah. I
started with 1942 because that’s when the fourth-Thursday-of-November
rule was established. Presumably due to the regexp part, this
expression takes a moment to compute.
This past weekend I had some questions from next-user-here (NUH) on my
original Elfeed post about changing some of Elfeed’s
behavior. NUH is an Elisp novice so accomplishing some of the
requested modifications wasn’t obvious. A novice is mostly limited to
setting variables, not defining advice or using hooks. I’ve also been
using Elfeed daily for about three months now as my sole web feed
reader and along the way I’ve developed some best practices. In
addition to responding to some of NIH’s questions here, I’d like to
share some tips and tricks.
Custom Entry Launchers
Currently you can press “b” to launch one or more entries in your
browser. You can use “y” to copy an single entry to the clipboard.
What if you want to make another action.
In my configuration I have a fancy binding that sends the entry URLs
in the selected region to youtube-dl for downloading the
videos. It’s too large to share as a snippet so here’s a small example
of something similar using a program called xcowsay.
Now when I hit “x” over an entry in Elfeed I’m greeted by a cow
announcing the title.
Entry Listing Customization
The search buffer you see when starting Elfeed, where entries are
listed, can be customized a few different ways. First, this buffer
does grow dynamically. After re-sizing the window/frame horizontally
you just have to refresh the view by pressing g (an Emacs
convention). How it fills out depends on the settings of these
variables,
elfeed-search-title-max-width
elfeed-search-title-min-width
elfeed-search-trailing-width
They control how wide the different columns should be as the window
size changes. An important caveat to this is that the cache stored in
elfeed-search-cachemust be cleared before the changes will be
reflected in the display. This cache exists because building the
display, assembling all the special faces, is actually quite
CPU-intensive. It was an optimization I established early on.
(clrhashelfeed-search-cache)
If you set these variables in your start-up configuration you don’t
need to worry about clearing the cache because it will already be
empty. It’s only a concern when playing with the settings.
Date Display
Another question was about adding time to the entry listing. Elfeed
only displays the entry’s date. Dates are formatted by the function
elfeed-search-format-date. This can be redefined to display dates
differently.
Feeds and entries in the database can be manipulated to become
whatever you want them to be. Because Elfeed is regularly modifying
the database, the trick is to perform the manipulation at just the
right time.
Feed Title Changes
Say you want to change a feed title because you don’t like the title
supplied by the feed. For example, the title to my blog’s feed is
“null program” but instead you think it should be “Seriously Handsome
Programmer” (head injury, remember?). The function
elfeed-db-get-feed can be used to fetch a feed’s data structure from
the database, given it’s exact URL as listed in your elfeed-feeds.
Hold it, that didn’t work. First, that display cache is getting in the
way again. Feed titles change very infrequently so they’re cached
aggressively. More importantly, next time you update your feeds Elfeed
will re-synchronize the feed title with the official title. It’s going
to fight against your intervention.
The solution is to do it with a little bit of advice just before the
title is displayed. Advise the function elfeed-search-update with
some “before” advice.
Automatic entry modification should happen immediately upon discovery
so that it looks like the entry arrived that way. This is done through
the elfeed-new-entry-hook. Generally this would be used for applying
custom tags. These examples are from the documentation:
;; Mark all YouTube entries(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:feed-url"youtube\\.com":add'(videoyoutube)));; Entries older than 2 weeks are marked as read(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:before"2 weeks ago":remove'unread));; Building subset feeds(add-hook'elfeed-new-entry-hook(elfeed-make-tagger:feed-url"example\\.com":entry-title'(not"something interesting"):add'junk:remove'unread))
Due to a feature I recently ported from my personal configuration,
this tagger helper function is less necessary. You can put lists in
your elfeed-feeds list to supply automatic tags.
(setqelfeed-feeds'(("https://nullprogram.com/feed/"blogemacs)"http://www.50ply.com/atom.xml"; no autotagging("http://nedroid.com/feed/"webcomic)))
The same trick could be used to remove advertising, change the date,
change the title, etc. The elfeed-deref and elfeed-ref parts are
needed to fetch and store content in the content database. Only a
reference is stored on the structure. You can actually use these
functions at any time outside of Elfeed, but they’ll eventually get
garbage collected if Elfeed doesn’t know about them.
A question that’s been asked few times is if entries can be deleted.
To start off, the answer to that question is “no.” There is no
function provided to remove entries from the database. If you want to
remove entries you’re probably taking the wrong approach.
The main problem with removal is that Elfeed needs to keep track of
what it’s seen before. If an entry is removed and then rediscovered,
it will reappear as unread. There are better ways to “remove” entries,
such as tagging them specially.
On a moderately-powerful computer Elfeed can easily handle at least
several tens of thousands of database entries. If “too many entries”
ever becomes a performance problem I’d rather solve it by making the
database faster than by removing information from the database. It’s
already very date-oriented so that older entries are infrequently
touched.
If storage is a concern, you shouldn’t get too worked up about that.
As of this post I have about 6,000 entries in my database and the
index file is only 3.5 MB. The content database after garbage
collection, which is the data/ directory under ~/.elfeed/, with
these 6k entries is 17MB. When I run M-x elfeed-db-compact,
currently an experimental feature, it drops down to 1.8MB. That’s less
than 1 kB per entry. It’s also less than my personal Liferea database
of roughly the same amount of content (~15MB) before I wrote Elfeed.
If even this storage is still too much you can always blow away your
data/ content database directory. This is safe to do even while
Emacs is running. You’ll still see all of the entries listed in the
search buffer but won’t be able to read them within Emacs until after
the next database update (when it re-fetches the most recent entry
content).
You can also clear out the content database from within Elisp by
visiting every entry and clearing its content field.
The same sort of expression can be used to run over all known entries
to perform other changes. If there was a delete function you might use
it here to remove entries older than a certain date, then hope they’re
not rediscovered.
If you never want to store entry content (you never read entries
within Emacs), you can use a hook to always drop it on the floor as it
arrives,
If you have any questions or suggestions about how to make Elfeed do
what you want it to do, feel free to ask. Some things may actually
require that I make changes to Elfeed to support it, though I hope
I’ve anticipated your particular need well enough to avoid that.
The design of Elfeed’s database took some experimentation
before any part of it was settled. A major design constraint was
Emacs’ very limited file input/output. There’s no random access and,
without the aid of an external program, files must always be read and
written wholesale. That’s not database-friendly at all! In the end I
settled on a design that minimized the size of the frequently
rewritten parts, an index with two different data models, by storing
immutable data in a loose-file, content-addressable database.
At the moment there really aren’t any pure-Elisp database solutions
for Emacs. This is almost certainly due to the aforementioned I/O
limitations. I ran into this same problem last year when I created
an Emacs pastebin server. I attempted, and failed, to
interface with a SQLite database through it’s command line program.
Nic Ferrier has published a generic database interface,
but it lacks concrete implementations.
As a bit of good news, as far as I know Emacs does properly handle
atomic file updates across all platforms, so a pure-Elisp database
developer would never have to worry about only writing half the
database. It’s always a safe operation. Worst case scenario you’re
left with an old version of data rather than no data at all.
A real possibility for a database would be connecting to an
established database server via TCP with an Emacs network process. If
the server has a specified wire protocol Elisp could talk to it
efficiently. In fact, there’s exists pg.el that does exactly
this for PostgreSQL. Unfortunately I was not able to get this working
with my pastebin, nor is this solution appropriate for Elfeed. It
would be unreasonable to require users to first set up a PostgreSQL
server just to read web feeds!
Ultimately it would seem that any efficient Emacs database requires
the help of an external program. The notmuch mail client,
which inspired Elfeed, does this. To access the notmuch database a
command line program is run once for each request. A query is passed
as a program argument and the output of the program is parsed into the
result.
The Early Database
For the first few days of its existence Elfeed only had an in-memory
database. Closing Emacs would lose everything. For my personal usage
patterns, where I read, or at least address, all entries that arrive
— and especially because I use Elfeed on a couple of different
computers — I don’t really need to track things long term. I could
easily mark everything after a certain date as read and forget about
them. However, it would be nice to have and, more importantly, many
people wouldn’t use Elfeed without persistence between Emacs sessions.
So, for the first database I did what I always do: dumped the data
structure to a file using the printer and parsed it back in later
using the reader. This is dead simple in Lisp, it’s very fast, and it
even works for circular data structures. It’s something I missed so
much with the much-less-capable JSON format earlier this year that I
wrote a JavaScript library to do it.
(defunsave-data(filedata)(with-temp-filefile(let((standard-output(current-buffer))(print-circlet)); Allow circular data(prin1data))))(defunload-data(file)(with-temp-buffer(insert-file-contentsfile)(read(current-buffer))))(save-data"demo.dat"'(abc["1"23]))(load-data"demo.dat");; => (a b c ["1" 2 3])
Anything with a printed representation can be serialized and stored
this way, including symbols, string, numbers, lists, vectors (structs,
objects), hash tables, and even compiled functions (.elc files).
Basically every Emacs library that stores data on disk uses this
technique.
Unfortunately, this is where I hit another serious database
constraint: print-circle is broken in Emacs 24.3,
the current stable release. This means Elfeed cannot take advantage of
this useful feature, at least not for a long time, as I had been
counting on. The final database is slightly slower and larger than
strictly required as a result.
The Content Database
After breaking the circular references of the in-memory database I
finally had persistence for the first time. With the naive
printer/reader approach it was slow, almost 1 second to write just a
few thousand entries on my 6-year-old laptop (my minimum requirements
target machine). I wanted Elfeed to support hundreds of thousands of
entries, if not millions, so this was much too slow.
The big slowdown was writing out all the entry content each time the
database is saved. These large strings containing HTML that rarely
change. There’s no reason to write these out every time, nor is there
a reason to even keep them in memory all the time, as it’s rarely
accessed. The solution is a loose-file, content-addressable database,
very similar to an unpacked Git object database.
The content database stores immutable sequences of characters — not
just raw bytes, but rather multibyte strings — using an unspecified
coding system (right now it’s UTF-8 for all platforms). The filename
for the content is the content hashed with SHA-1
(“content-addressable”). To limit the number of files per directory,
these files are stored in subdirectories named by the first
hex-encoded byte of the hash (just like Git). A database of 4 items
might look like this:
Something really neat about the content database is that it’s
completely agnostic about Elfeed. If it weren’t for Elfeed’s garbage
collector, anyone could use it to store arbitrary content. The
function elfeed-ref accepts a string and returns a reference into
the database. Because of the hash, providing the same string in the
future will return the same reference without actually performing a
write. References are dereferenced with elfeed-deref.
With content stored elsewhere, entries are a struct containing only
some small metadata: title, link, date, and a content database
reference. Writing out many of them at once is much, much faster.
I don’t expect it happens often, but this also means content is
de-duplicated. If two entries happen to have the same content they’ll
share content database storage. A small savings.
At this point it’s really tempting to get fancier and really put this
content database to use. The core index itself could be stored as raw
content, and the root to accessing the database would be a single
SHA-1 hash referencing it — again, very similar to Git. If an index
stores a reference to the previously written index, then the the
Elfeed database would be an immutable structure tracking its entire
history. Such a change would cost virtually nothing in performance,
just disk space.
Multiple Representations
With all the content out of the way, the database is now just a lean
index. At this point it’s a hash table mapping feed IDs to feeds.
Feeds contain a list of its entries. To build the entry listing for
the elfeed-search buffer, Elfeed needs to visit each feed in the hash
table, gather its entries into one giant list, then finally sort that
list by date. At around O(n log n), that sort operation is a real
performance killer. Completely unacceptable. To fix this we need to
think about how the data is updated and used.
First, entries are always viewed in date order, no exceptions.
From my experience of using web feeds for the last six years I never
had a reason to list feed entries by any other order. The vast
majority of the time, newer entries are most relevant, and if I need
to look for something specific I can search for it.
We definitely want to store entries in date-order so we can create
entry listings without performing a sort: something around O(n) or so.
Inserting new entries into this structure should also be efficient.
Second, entries are never removed from the database. This isn’t
e-mail. Even if a user doesn’t want to see an entry again, we have to
keep track of it. Otherwise it will show up as new if it’s discovered
in a feed again, which is likely. Things are added to the database and
never removed. In Elfeed, I use a junk tag to completely hide
entries I don’t want to see, and I always have a -junk element in my
filter.
There’s an important caveat to this one that I had missed until after
the public release: entry dates can change! When a previously
discovered entry is read from a feed, Elfeed updates (read: mutates)
the entry struct to reflect the new state. This includes the date.
It’s very likely that a date-sorted representation won’t tolerate date
changes underneath it since it’s keying off of them. Either we refuse
to update the entry date, or we remove the entry, update the date, and
then re-insert it (how it currently works).
Third, entries are generally added with a recent date. After the
database is initially populated, it’s only picking up new items. We
should prefer adding recently-dated entries be faster than adding
older entries. I didn’t get a chance to take advantage of this, but
it’s something to keep in mind.
Fourth, entries need to be keyed by an ID string. Each entry has a
unique, unchanging identifier string, either provided by the feed
itself (RSS’s guid or Atom’s id) or generated intelligently by
Elfeed. Especially because of the print-circle bug, we need to be
able to talk about feeds in terms of their ID — an indirect pointer.
(Actually, even when RSS guid tags are present, they’re permalinks
by default. So, unfortunately, RSS IDs are not at all resistant to
collisions across feeds. To work around this, entry identifiers are a
pair of strings: feed ID and entry ID. Atom doesn’t have this
problem, but we’re stuck with the lowest common denominator.)
A date-oriented representation would be unable to efficiently look up
an entry by its ID, so it needs to be supplemented by an ID-oriented
representation. This means we need two representations in our
database: date-oriented and ID-oriented.
So what do we use? Well, for keeping entries sorted by date we want
some sort of balanced tree. A B-tree is probably a good choice. Rather
than write one I went with an AVL tree since Emacs comes with a
library for it (avl-tree). It’s already debugged and optimized! The
bad news is that the internal structure is unspecified, so there are
no guarantees that it can be serialized. A future update to the
library may break the Elfeed database. I also had to hack into it to
work around a security issue. The comparison function is embedded in
the tree. After deserializing the database, Elfeed needs to ensure
that no one stuck a malicious function in there.
The choice for an ID database was super-easy: a hash table. Due to the
print-circle bug, this is actually the main representation. The AVL
tree only stores IDs and it has to reach into the hash table to do any
date comparisons. If print-circle was working I could store the same
exact entry objects in the AVL tree as the hash table, so mutating
them would update them in all representations. However, with
print-circle off, on deserialization these would become unique
objects and updates would break.
The Future
That’s where the database is today. I put in a few extra fields that
aren’t actually used yet, so that there’s room to make a few changes
without breaking the database. Perhaps someday I’ll work out a whole
new database structure, or maybe a proper database library will come
into existence, and this post will simply document the old database.
Unsatisfied with my the results of
recent search for a new web feed reader, I created my own
from scratch, called Elfeed. It’s built on top of Emacs and
is available for download through MELPA. I intend it to be
highly extensible, a power user’s web feed reader. It supports both
Atom and RSS.
The design of Elfeed was inspired by notmuch, which is
my e-mail client of choice. I’ve enjoyed the notmuch search
interface and the extensibility of the whole system — a side-effect
of being written in Emacs Lisp — so much that I wanted a similar
interface for my web feed reader.
The search buffer
Unlike many other feed readers, Elfeed is oriented around entries —
the Atom term for articles — rather than feeds. It cares less about
where entries came from and more about listing relevant entries for
reading. This listing is the *elfeed-search* buffer. It looks like
this,
This buffer is not necessarily about listing unread or recent entries,
it’s a filtered view of all entries in the local Elfeed database.
Hence the “search” buffer. Entries are marked with various tags,
which play a role in view filtering — the notmuch model. By default,
all new entries are tagged unread (customize with
elfeed-initial-tags). I’ll cover the filtering syntax shortly.
From the search buffer there are a number of ways to interact with
entries. You can select an single entry with the point, or multiple
entries at once with a region, and interact with them.
b: visit the selected entries in a browser
y: copy the selected entry URL to the clipboard
r: mark selected entries as read
u: mark selected entries as unread
+: add a specific tag to selected entries
-: remove a specific tag from selected entries
RET: view selected entry in a buffer
(This list can be viewed within Emacs with the standard C-h m.)
The last action uses the Simple HTTP Renderer (shr), now part of
Emacs, to render entry content into a buffer for viewing. It will even
fetch and display images in the buffer, assuming your Emacs has been
built for it. (Note: the GNU-provided Windows build of Emacs doesn’t
ship with the necessary libraries.) It looks a lot like reading an
e-mail within Emacs,
The standard read-only keys are in action. Space and backspace are for
page up/down. The n and p keys switch between the next and
previous entries from the search buffer. The idea is that you should
be able to hop into the first entry and work your way along reading
them within Emacs when possible.
Configuration
Elfeed maintains a database in ~/.elfeed/ (configurable). It will
start out empty because you need to tell it what feeds you’d like to
follow. List your feeds elfeed-feeds variable. You would do this in
your .emacs or other initialization files.
Once set, hitting G (capitalized) in the search buffer or running
elfeed-update will tell Elfeed to fetch each of these feeds and load
in their entries. Entries will populate the search buffer as they are
discovered (assuming they pass the current filter), where they can be
immediately acted upon. Pressing g (lower case) refreshes the search
buffer view without fetching any feeds.
Everything fetched will be added to the database for next time you run
Emacs. It’s not required at all in order to use Elfeed, but I’ll
discuss some of
the details of the database format in another post.
The search filter
Pressing s in the search buffer will allow you to edit the search
filter in action.
There are three kinds of ways to filter on entries, in order of
efficiency: by age, by tag, and by regular expression. For an entry to
be shown, it must pass each of the space-delimited components of the
filter.
Ages are described by plain language relative time, starting with @.
This component is ultimately parsed by Emacs’ time-duration
function. Here are some examples.
@1-year-old
@5-days-ago
@2-weeks
Tag filters start with + and -. When +, entries must be tagged
with that tag. When -, entries must not be tagged with that tag.
Some examples,
+unread: show only unread posts.
-junk +unread: don’t show unread “junk” entries.
Anything else is treated like a regular expression. However, the
regular expression is applied only to titles and URLs for both
entries and feeds. It’s not currently possible to filter on entry
content, and I’ve found that I never want to do this anyway.
Putting it all together, here are some examples.
linu[xs] @1-year-old: only show entries about Linux or Linus from
the last year.
-unread +youtube: only show previously-read entries tagged
with youtube.
Note: the database is date-oriented, so age filtering is by far the
fastest. Including an age limit will greatly increase the performance
of the search buffer, so I recommend adding it to the default filter
(elfeed-search-search-filter).
Tagging
Generally you don’t want to spend time tagging entries. Fortunately
this step can easily be automated using elfeed-make-tagger. To tag
all YouTube entries with youtube and video,
Any functions added to elfeed-new-entry-hook are called with the new
entry as its argument. The elfeed-make-tagger function returns a
function that applies tags to entries matching specific criteria.
This tagger tags old entries as read. It’s handy for initializing an
Elfeed database on a new computer, since I’ve likely already read most
of the entries being discovered.
Tagging is also really handy for fixing some kinds of broken feeds or
otherwise filtering out unwanted content. I like to use a junk tag
to indicate uninteresting entries.
There are a few feeds I’d like to follow but do not because the
entries lack dates. This makes them difficult to follow without a
shared, persistent database. I’ve contacted the authors of these feeds
to try to get them fixed but have not gotten any responses. I haven’t
quite figured out how to do it yet, but I will eventually create a
function for elfeed-new-entry-hook that adds reasonable dates to
these feeds.
Custom actions
In my own .emacs.d configuration I’ve added a new entry action
to Elfeed: video downloads with youtube-dl. When I hit d on a
YouTube entry either in the entry “show” buffer or the search buffer,
Elfeed will download that video into my local drive. I consume quite a
few YouTube videos on a regular basis (I’m a “cord-never”), so this
has already saved me a lot of time.
Adding custom actions like this to Elfeed is exactly the extensibility
I’m interested in supporting. I want this to be easy. After just a
week of usage I’ve already customized Elfeed a lot for myself — very
specific customizations which are not included with Elfeed.
Web interface
Elfeed also includes a web interface! If you’ve loaded/installed
elfeed-web, start it with elfeed-web-start and visit this URL in
your browser (check your httpd-port).
http://localhost:8080/elfeed/
Elfeed exposes a RESTful JSON API, consumable by any application. The
web interface builds on this using AngularJS, behaving as a
single-page application. It includes a filter search box that filters
out entries as you type. I think it’s pretty slick, though still a bit
rough.
It still needs some work to truly be useful. I’m intending for this to
become the “mobile” interface to Elfeed, for remote access on a phone
or tablet. Patches welcome.
Try it out
After Google Reader closed I tried The Old Reader for awhile. When
that collapsed under its own popularity I decided to go with a local
client reader. Canto was crushed under the weight of all my feeds, so
I ended up using Liferea for awhile. Frustrated at Liferea’s lack of
extensibility and text-file configuration, I ended up writing Elfeed.
Elfeed now serving 100% of my personal web feed reader needs. I think
it’s already far better than any reader I’ve used before. Another case
of “I should have done this years ago,” though I think I lacked the
expertise to pull it off well until fairly recently.
At the moment I believe Elfeed is already the most extensible and
powerful web feed reader in the world.
For the last 8 years I have been using Gmail as my e-mail provider,
during which 3 years I was a student. It was very convenient,
inexpensive (cost-free!), and especially suitable for a student using
various lab computers and stuck behind a fairly restrictive firewall
(an anti-file-sharing measure). I didn’t have to worry about filtering
spam or maintaining or paying for a server. Easy, easy, easy. This has
finally come to an end.
That convenience kept me as a user after college up until now. As of
two weeks ago I changed my e-mail address. You can find it listed
below my portrait on this page (if you’re reading this on my website).
Not only am I using my own domain name, but I’m running my own e-mail
server. After attempting, and failing, at creating a decent setup with
mu4e, I ended up following this guide:
It’s built around the superb notmuch mail indexer, which
includes a powerful, fast Emacs e-mail client. Just from its technical
superiority I’m wishing I switched to notmuch years ago. I’m now
understanding for the first time why all those old fogey hackers like
to use e-mail for everything: mailing lists, software patches, bug
reporting, etc. E-mail a user interface agnostic system, giving
everyone their own choice. The tricky part is setting up a decent
interface to it.
Prompting the change
As you know, Google Reader shut down this past July. This
left me using only two Google services: Talk and Mail. Not only is
Talk easily replaceable — it has no significant data on Google’s
servers — it will be shut down soon as well. I would need to
move to a different XMPP server anyway. If I could move off of Gmail,
I would finally be able to discontinue my Google account for good!
Why would I want to do this? It’s become increasingly apparent,
especially this year, that there is very little privacy to be had when
logged into a Google account. Various intelligence and law enforcement
organizations have easy — likely automated — access to user data,
especially e-mail. I’d really like to take that privacy back.
There are also the technical reasons. It’s a bit embarrassing to
admit, but the final straw that pushed me to finally leaving Gmail was
the new compose interface — a technical issue rather than
a privacy issue — which was pushed out just a few days before I left.
I found it to be very unpleasant and, worse, completely incompatible
with Pentadactyl, as there is no longer a plain-text option (the one
listed is a fake). A huge technical step backwards, towards the layman
and away from the power user. I could do so much better than this.
Also embarrassing was being unable to have any meaningful use of PGP
with e-mail all these years. That’s something I’ve always wanted to
fix.
Brian was a guinea pig for me, because, between the two of
us, he was actually the first one to move to his own e-mail platform.
Seeing his success was a big encouragement for me. Not only could it
be done fairly easily, but the results would be a huge improvement.
This was all within my grasp!
A daunting task
Switching everything about my e-mail — provider, client, spam
filter, server, domain — and running it myself would be a daunting
task. There’s 8 years of archived e-mail to manage, though I kept it
trimmed to a relatively light 1 GB of storage (which I cut down to 200
MB before exporting). I’ve made backups of all this e-mail on
occasion, but I never had to worry about searching or actually using
it. The backups were done “just in case.”
Gmail has excellent spam filtering, an advantage of having so many
samples available, and I’m a complete newbie at dealing with it
myself. Despite having my e-mail address published on this blog for
the last 6 years, I surprisingly only receive about one spam message
per day, so this wasn’t actually a huge risk for me.
Then there’s the issue of not looking like spam myself. My e-mail
server needs to sit in a friendly IP neighborhood. I need to have a
proper PTR record (reverse DNS). I need to generally look legitimate.
No showing up to deliver mail to other mail servers in just a t-shirt.
This is actually something I’m still struggling with right now. In
fact, if I’ve sent you personal e-mail in the past and you’re using
Gmail, you should check your spam folder right now, because something
I’ve sent recently may likely have been caught in it. I don’t know why
yet.
I would also need to learn the ropes of a new e-mail client. I used
Eudora from 1995 to 2005, then Gmail up until now. Now, I’m the last
person to be reluctant about learning how to operate a new piece of
software. I’m constantly on the lookout for better software. The
problem is that I use e-mail for a lot of very important things. I
can’t afford mistakes. I need to hit the ground running on this one.
notmuch vs mu4e
Since I decided early on to go with an Emacs-based e-mail client,
learning a new e-mail client got a lot easier. I know Emacs Lisp
pretty darn well. In the worst case of getting stuck, I could very
easily study the client’s source code and work out for myself whatever
is going on. I could even monkey patch it in my configuration if it
was causing problems for me (and I’ve already done exactly that).
I wanted to use the Maildir format, something I could hack
on if needed. The two obvious choices for this were mu4e and notmuch,
both started in 2009. I initially reached for mu4e. Compared to
notmuch, it follows Emacs idioms more closely. For example, the e-mail
listing is oriented around a mark and execute paradigm, like a dired
buffer. After an initial glance, it felt more integrated.
Unfortunately, mu4e is still not mature enough for real productive
use, making it far too risky for e-mail. I found out the hard way that
the database format has varied regularly between versions. Worse, mu4e
is not suitable for remote access. Not only does it assume the Maildir
directory is on the local host, it uses absolute paths to access it,
so it won’t work over sshfs as I had hoped. Bummer.
In contrast, the notmuch client
is specifically designed to be operated remotely. Emacs
doesn’t realize that it runs the notmuch client over SSH. Emacs
doesn’t need to touch the Maildir directly. It’s a beautiful setup,
one very friendly to versioning dotfiles. I’ve already
done some pretty heavy configuration to get it exactly the way I want
it. On top of this, notmuch is incredibly fast and stable. It’s been a
very enjoyable client. So much so that it inspired me to build a web
feed reader with a similar interface (to be described in my next
post).
The mail server
My early plan was to run an e-mail server on a Raspberry Pi. It’s
low-powered, making it very inexpensive and quiet to operate. It’s
also very portable, so I wouldn’t need to lug a server around if I
needed to move it — no more difficult than moving a cell phone
charger. I could run it from my nightstand next to my bed if I wanted
to. On the other hand, I would be a little nervous running my mail
server on a residential connection. The downtime would be a product of
my ISP, my power company, and my router. It’s not bad, but it’s risky
when I’m worried about receiving important e-mail. If I was in the
middle of job hunting I probably wouldn’t attempt it at all.
Fortunately e-mail servers will retry over the course of several days,
so I think this would generally be manageable.
This plan was struck down by Comcast’s network policies. The good news
is that my IP address has not changed in years. The bad news is that
they block port 25 both incoming and outgoing as an anti-spam measure.
This makes it impossible to run an e-mail server, because e-mail
must be received on port 25 (MX records were misdesigned feature).
For outgoing, I would need to send e-mail through Comcast’s smarthost
server, which brings up the privacy issue again. I assume the same
organizations have this tapped as well. Even if port 25 wasn’t
blocked, I wouldn’t be able to set a PTR record and my IP neighborhood
would be suspicious.
I ended up going with Digital Ocean, as the linked guide
suggested. The smallest, cheapest offering is more than suitable for
my needs both as an e-mail server and an XMPP server. It will probably
be handy for other short-lived, experimental servers
too.
I used getmail to get all of my old mail onto the mail
server in the Maildir format. It was completely straightforward and
probably the easiest part of it all.
PGP
I can finally use GnuPG with my e-mail. An important factor in this
setup is that encryption and decryption is done locally, not on my
e-mail server. I don’t need to trust the server with my private keys.
However, verification of signatures is done on the server, which is
slightly less than ideal, but manageable.
I thought I might need to generate a fresh PGP key for this new e-mail
address, but instead I learned something new about PGP. A key can have
multiple identities attached to it, so all I needed to do was edit the
key and add the new e-mail address. The PGP designers had already
thought of this problem two decades ago! The updated key is linked
next to my portrait, as well as distributed on the public keyservers.
I look forward to making more use of cryptography with my e-mail.
The old address
It will take me awhile, a year or more, to move everything off of my
old e-mail address. I have a lot of accounts associated with it, many
of which won’t allow me to change my e-mail address. So, in the
meantime, anything sent there will continue to be forwarded to me.
That’s now the only purpose of my Google account.
I’m a little worried about using my new e-mail address as my Digital
Ocean account because it presents a circularity problem. I could
easily get myself locked out of everything. I’ll have to figure out
how I’m going to handle that situation. (Use my work e-mail address?
So long as I don’t get locked out of my e-mail and get fired all at
the same time.)
If you’re a hacker, I encourage you to run your own e-mail server if
you’re not doing so already. It’s been extremely liberating for me.
I don’t know what to title or name this thing, so bear with me.
Two years ago I started versioning my Emacs configuration.
One year ago I started versioning the rest of my dotfiles
the same way. This has composed beautifully with Debian, which truly
is the universal operating system. To create my
comfortable development environment from scratch on a new (or used)
computer, all I need to do is
install a bare-bones Debian system, then direct it to
automatically install a short list of my preferred software (apt-get),
and finally clone these two repositories into place. Given a decent
Internet connection, the whole process takes under an hour to go from
blank hard drive to highly-productive computer system.
In fact, this whole process is so straightforward that it can be
automated using an amazing tool called live-build!
Taking the next step in versioning and automation, I wrote a
live-build build that creates a live system with my personal
configuration baked in.
A link to the latest ISO build can be found in the above link. To
try it out, burn it to a CD, write it onto a flash drive (it’s a
hybrid ISO), or just fire it up in your favorite virtual machine. You
will be booted directly into very nearly my exact configuration,
down to the same random wallpaper selection. It’s extremely minimal and
will look like this.
I don’t know for sure yet if this will be useful to anyone except me.
On occasions where I need to make quick use of some arbitrary
computer, or maybe just for system rescue, having this will be
incredibly handy. Knoppix is nice, but working without my
own configuration can be discouraging; it’s so slow in comparison.
It’s also a chance for others to glimpse at my workflow without any
commitment (i.e. potential dotfile clobbering). I like to study other
developers’ workflows, stealing their ideas for my own, so I want to
make mine easy to study. I think I’m doing some innovative things with
a hacked-together pseudo-tiling window manager, my Firefox
configuration, and my Emacs configuration. At least two of my
co-workers’ Emacs configurations are forked from mine (you know who
you are!). From a selfish perspective, the more people using workflows
like mine, the better these workflows will be supported by the
community at large!
I had said that it’s “very nearly” my exact configuration. At the time
of this writing, what’s missing is Quicklisp and
Leiningen, since these aren’t available as fully-functioning
Debian packages at the moment. I’ll work them in eventually. The build
is non-incremental and takes about an hour right now, so adding these
little extras by trial-and-error will take some time.
Pentadactyl really shines here, because it allows me to
completely configure Firefox/Iceweasel from a version-friendly text
file. Except for some user scripts (still figuring out how to install
those at build time), the browser in that image is identical to what
I’m using right now. Even though V8 is the king of performance,
Firefox still wins hands-down for power users due to its superior
configurability.
However, this Firefox configuration still has an annoyance. Several of
the browser add-ons that I pre-install always pop up their first-run
welcome messages. These messages flip a setting in Firefox’s registry
so they don’t run again, a setting that is lost when the system shuts
down. I can toggle these settings from the Pentadactyl configuration
file, but not early enough. By the time Pentadactyl gets to applying
these settings it’s too late. The messages, including Pentadactyl’s,
are already queued to be shown. I don’t think it’s possible to
completely fix this, even if Pentadactyl is fixed.
Anything not already installed is readily installable through apt-get.
Right now I’m considering the feasibility of some sort of lazy-install
system based on command-not-found. Debian has a package called
command-not-found which intercepts the shell’s error handler when an
issued command is not found. Instead of just giving the normal
warning, it prints out what package needs to be installed in order to
provide the command. What I think would be really neat is if the
needed package is automatically installed, then the requested command
is then re-run, all without returning control to the shell in the
interim. It would be a lot like Emacs autoloads. As long as I have an
Internet connection, most of Debian’s packages would be virtually
installed on my live system as far as I’m concerned. The initial run
of any program just takes a little longer.
I’ll continue to tweak this image over time, not only as I figure out
how to make things work in Debian live-build, but also as my
preferences and workflows evolve. Adjusting my configuration to work
on a live-system has been enlightening, revealing all sorts of little
manual things that I hadn’t yet automated. Perhaps someday this build
will replace traditional operating system installations for me, at
least for productive work. I could do all of my work from a portable
read-only live system with a bit of short-lived (i.e. local cache)
user-data persistence stored on a separate writable medium.
One of my regular commenters, and as of recently co-worker, Ahmed
Fasih, sent me a video, Live coding in Lua. The author of the
video added support to his IDE for scaling numbers in source code by
dragging over them with the mouse. This feature was directly inspired
by Bret Victor, a user interface visionary, probably best
introduced through his presentation Inventing on Principle.
I think Bret’s interface ideas are interesting and his demos very
impressive. However, I feel they’re too specialized to generally be
very useful. Skewer suffers from the same problem: in order
to truly be useful, programs need to be written in a form that expose
themselves well enough for Skewer to manipulate at run-time. Some
styles of programming are simply better suited to live development
than others. This problem is amplified in Bret’s case by the extreme
specialty of the tools. They’re fun to play with, and probably great
for education, but I can’t imagine any time I would find them useful
while being productive.
Anyway, Ahmed wanted to know if it would be possible to implement this
feature in Emacs. I said yes, knowing that Emacs ships with
artist-mode, where the mouse can be used to draw with
characters in an editing buffer. That’s proof that Emacs has the
necessary mouse events to do the job. After spending a couple of hours
on the problem I was able to create a working prototype:
mouse-slider-mode.
It’s a bit rough around the edges, but it works. When this minor mode
is enabled, right-clicking and dragging left or right on any number
will decrease or increase that number’s value. More so, if the current
major mode has an entry in the mouse-slider-mode-eval-funcs alist,
as the value is scaled the expression around it is automatically
evaluated in the live environment. The documentation shows how to
enable this in js2-mode buffers using skewer-mode. This is actually a
step up from the other, non-Emacs implementations of this mouse slider
feature. If I understood correctly, the other implementations
re-evaluate the entire buffer on each update. My version only needs to
evaluate the surrounding expression, so the manipulated code doesn’t
need to be so isolated.
There is one limitation that cannot be fixed using Elisp. If the mouse
exits the Emacs window, Elisp stops receiving valid mouse events.
Number scaling is limited by the width of the Emacs window. Fixing
this would require patching Emacs itself.
This is purely a proof-of-concept. It’s not installed in my Emacs
configuration and I probably won’t ever use it myself, except to show
it off as a flashy demo with an HTML5 canvas. If anyone out there
finds it useful, or thinks it could be better, go ahead and adopt it.
Update April 2015: I now use use-package instead of
the with-package macro explained below. It’s cleaner, nicer, and
better maintained.
I was inspired by a post recently written by Milkypostman
(the M in MELPA). He describes some of his init.el configuration,
specifically focusing on an after macro that wraps the misdesigned
eval-after-load function. I wanted to take this macro further in
three ways:
The delayed expression should be properly byte-compiled,
which doesn’t happen by default with eval-after-load.
In a few cases my expression depends on multiple, independent
packages but eval-after-load only accepts one.
If I’m specifying packages when using my macro, why bother listing
them at the top of my initialize file? I could DRY things up by
learning what packages to install when the macro is used. Here’s
the kicker: I can pretend that every available package is already
installed like built-in packages!
The result is a pair of macros with-package and with-package*
which can be found in package-helper.el. The latter form
doesn’t wait but immediately loads the specified packages with
require. It’s shaped just like Milkypostman’s after macro, except
that it can accept a list of packages in place of a single symbol.
Also, the package names aren’t quoted; they don’t need to be since
this is a macro instead of a function.
Here’s a typical use case for each macro. That expose higher-order
function is from my personal utility library. The
expressions to be evaluated depend on both packages and neither needs
to be loaded immediately, so I’m using the first form of the macro.
For the second one, I’m going to be using smex right away (takes over
M-x), so I use the second form, which immediately loads smex. The
macro isn’t really necessary at all here since I could just use
require and follow it with these expressions, but I really like
how this organizes my init.el. It creates a domain-specific language
(DSL) just for Emacs configuration. Each package configuration is
grouped up in a clean let-like form. Since I’ve added syntax
highlighting to with-package it looks very elegant. Normal syntax
highlighters aren’t going to do this, so here’s a screenshot of my
buffer.
JavaScript developers with a keen eye may notice a familiar pattern
here. This macro is shaped a bit like the
Asynchronous Module Definition (AMD), with asynchronousy in
mind. Since this is Lisp with a powerful macro system, I get to hide
away the function wrapper part.
Using this macro has caused me to use eval-after-load with just
about everything. This has cut my initialization time down to about
10% of what it was before! On those occasions that I do restart
Emacs, it’s really nice that it’s back to under 1 second (0.6 seconds
vs 6 seconds).
The problem of eval-after-load
I’m calling eval-after-load poorly designed because it’s a perfect
example of an inappropriate use of eval. In function form it
should have accepted a function as its second argument instead of an
s-expression, so it would work like a hook. This is even more
inappropriate now that Emacs has proper lexical closures, which is the
perfect mechanism for delayed evaluation. The whole point of
eval-after-load is to speed up Emacs initialization time, but using
eval is slow. To the compiler, this isn’t code, just data. This
means no byte-compilation and no compiler warnings.
A possible alternative design for eval-after-load would be a hook
named something like <package>-load-hook. Then when load or
require loads a file, it runs the hook with the matching name. This
removes eval-after-load as its own standalone language concept.
(add-hook'skewer-mode-load-hook(lambda()...))
The problem here is when the package is already loaded the hook is
never run. In contrast, when eval-after-load is used on an
already-loaded package, the expression is immediately evaluated.
Given this, if there was something I could change about this it would
simply be for eval-after-load, whatever it would be called, to take
a function for the second argument. I would also provide a simple
macro just like after that wraps this function. Why not just a
macro? The function form would be really useful for a situation like
this,
(eval-after-load'skewer-mode#'skewer-setup)
Here there’s no need to instantiate a new anonymous function or
s-expression. If all it’s doing is calling a zero-arity function, that
function can be passed in directly.
A month ago Zane Ashby made a pull request that added another minor
mode to Skewer: skewer-html-mode. It’s analogous to the
skewer-css minor mode in that it evaluates HTML “expressions” in the
context of the current page. The original pull request was mostly a
proof of concept, with evaluated HTML snippets being appended to the
end of the page (body) unless a target selector is manually
specified.
This mode is still a bit rough around this edges, but since I think
it’s useful enough for productive work I’ve merged it in.
Replacing HTML
Unsatisfied with just appending content, I ran with the idea and
updated it to automatically replace structurally-matching content on
the page when possible. Zane’s fundamental idea remained intact: a CSS
selector is sent to the browser along with the HTML. Skewer running in
the browser uses querySelector() to find the relevant part of the
document and replaces it with the provided HTML. This is done with the
command skewer-html-eval-tag (default: C-M-x), which selects the
innermost tag enclosing the point.
To accomplish this, an important piece of skewer-html exists to
compute this CSS selector. It’s a purely structural selector, ignoring
classes, IDs, and so on, instead relying on the pseudo-selector
:nth-of-type. For example, say this is the content of
the buffer and the point is somewhere inside the second heading (Bar).
<html><head></head><body><divid="main"><h1>Foo</h1><p>I am foo.</p><h1>Bar</h1><p>I am bar.</p></div></body></html>
The function skewer-html-compute-selector will generate this
selector. Note that :nth-of-type is 1-indexed.
The > syntax requires that these all be direct descendants and
:nth-of-type allows it to ignore all those paragraph elements. This
means other types of elements can be added around these headers, like
additional paragraphs, without changing the selector. The :nth-of-type
on body is obviously unnecessary, but this is just to keep
skewer-html dead simple. It doesn’t need to know the semantics of
HTML, just the surface syntax. There will only ever be one body tag,
but to skewer-html it’s just another HTML tag.
Side note: this is why I strongly prefer to use /> self-closing
syntax in HTML5 even though it’s unnecessary. Unlike XML, that closing
slash is treated as whitespace and it’s impossible to self-close tags.
The schema specifies which tags are “void” (always self-closing:
img, br) and which tags are “normal” (explicitly closed: script,
canvas). This means if you don’t use /> syntax, your editor
would need to know the HTML5 schema in order to properly understand
the syntax. I prefer not to require this of a text editor — or
anything else doing dumb manipulations of HTML text — especially with
the HTML5 specification constantly changing.
When I was writing this I originally included html in the selector.
Selector computation would just walk up to the root of the document
regardless of what the tags were. Curiously, including this causes the
selector to fail to match even though this is literally the page
structure. So, out of necessity, skewer-html knows enough to leave it
off.
For replacement, rather than a simple innerHTML assignment on the
selected element, Skewer is parsing the HTML into an node object,
removing the selected node object, and putting the new one in its
place. The reason for this is that I want to include all of the
replacement element’s attributes.
Another HTML oddity is that the body and head elements cannot be
replaced. It’s a limitation of the DOM. This means these tags cannot
be “evaluated” directly, only their descendants. Brian and I also ran
into this issue in impatient-mode while trying to work around a
strange HTML encoding corner case: scripts loaded with a script tag
created by document.write() are parsed with a different encoding
than when loaded directly by adding a script element to the page.
This last part is actually a small saving grace for skewer-css, which
works by appending new stylesheets to the end of body. Why body
and not head? Because some documents out there have stylesheets
linked from body, and properly overriding these requires appending
stylesheets after them. If body is replaced by skewer-html, all of
the dynamic stylesheets appended by skewer-css would be lost,
reverting the style of the page. Since we can’t do that, this isn’t an
issue!
Appending HTML
So what happens when the selector doesn’t match anything in the
current document? Skewer fills in the missing part of the structure
and sticks the content in the right place. Next time the tag is
evaluated, the structure exists and it becomes a replacement
operation. This means the document in the browser can start completely
empty (like the run-skewer page) and you can fill in content as you
write it.
But what if the page already has content? There’s an interactive
command skewer-html-fetch-selector-into-buffer. You select a part of
the page and it gets inserted into the current buffer (probably a
scratch buffer). The idea is that you can then modify and then
evaluate it to update the page. This is the roughest part of
skewer-html right now since I’m still figuring out a good workflow
around it.
If you have Skewer installed and updated, you already have
skewer-html. It was merged into master about a month ago. If you
have any ideas or opinions for how you think this minor mode should
work, please share it. The intended workflow is still not a
fully-formed idea.
Update 2013-06-01: I ultimately decided that Skewer should not
modify any mode hooks automatically. Instead the major mode hooks can
be configured by putting (skewer-setup) in your initialization file.
This function is designed to play well with autoloading, so using it
won’t increase your startup time.
There’s a discussion happening on a Skewer issue on GitHub:
Problems with skewer-css autoload. The issue was opened by
Steve Purcell. Right now Skewer’s CSS minor mode is enabled by default
in less-css-mode, which is, of course, incompatible with the minor
mode. It’s enabled because less-css-mode is derived from css-mode, so
it runs all of css-mode’s hooks.
There are actually two separate problems here.
The hook to activate the minor mode needs to check what major mode
is activating it, because css-mode-hook is run by other modes. This
is easy to fix, though it’s not very elegant. I would need to do
the same for skewer-html-mode.
Skewer is eagerly configuring itself once it’s installed. This is
intentional: I want Skewer to be really easy to use right
out-of-the-box. There’s no “install the package, then add these
lines to your startup configuration.” If I remove this behavior,
the previous problem becomes the user’s problem, since it’s up to
them to activate the minor mode.
Steve is telling me that this auto-configuration is a bad idea; it so
often causes these sorts of messes. The gist of his argument is that
installing a package is separate from enabling a package. It’s up
to the user to decide how and when they want to use a package. Steve
maintains a number of popular Emacs packages and, even more
importantly, he’s one of the MELPA maintainers. He knows this stuff
much better than I do. He even used to share my current opinion
up until two months ago when someone changed his mind.
On the other hand, I really dislike when software has such awful
defaults that it’s unusable without first configuring it. Skewer’s
case wouldn’t be too bad since it can be enabled manually without
editing any configuration, but practical use would require that users
configure Skewer in their startup files. It would also make it harder
for users to discover Skewer’s features. They might not otherwise even
be aware there are CSS and HTML minor modes! If the concern is
separating installation and activation, package.el does have a
variable, package-load-list, for this purpose, though using it isn’t
very convenient.
Right now I’m stuck in this dilemma like a deer caught in the
headlights. Formal packages are a very new thing for Emacs, so there
doesn’t seem to be a community consensus on this issue yet. I really
like when the packages I install behave like Skewer does now (i.e.
nrepl.el) so I don’t need to configure them. But I would also
be easily frustrated if that configuration magic was getting in my
way. This particular annoyance happens to me outside of Emacs often
enough (i.e. Chromium), though it’s worse because it generally can’t
easily be fixed like in Emacs.
I’m still trying to make up my mind about this. If you have an opinion
on the matter I’d like to hear it. You can leave a comment here, or
much better, leave your comment on the issue on GitHub. It’s not going
to come down to a vote or anything like that. I just want to get a
feel for how people expect Emacs packages to work.
With this, I reduce the js2-mode modeline lighter from "JavaScript IDE" to just "JS2".
I stole it from Bodil's
.emacs.d and macroified it a little.
The first argument is the package name, the second is the mode in
question, and the third is the new lighter for the mode.
I recently added support to Skewer for loading libraries on
the fly using Bower’s package infrastructure. Just make sure
you’re up to date, then while skewering a page run M-x
skewer-bower-load. It will prompt for a package and version, download
the library, then inject it into the currently skewered page.
Because the Bower infrastructure is so simple, Bower is not actually
needed in order to use this. Only Git is required, configured by
skewer-bower-git-executable, which it tries to configure itself from
Magit if it’s been loaded.
Motivation
Skewer comes with a userscript that adds a small toggle button
to the top-right corner every page I visit. Here’s a screenshot of the
toggle on this page.
When that little red triangle is clicked, the page is connected to
Emacs and the triangle turns green. Click it again and it disconnects,
turning red. It remembers its state between page refreshes so that I’m
not constantly having to toggle.
It’s mainly for development purposes, but it’s occasionally useful to
Skewer an arbitrary page on the Internet so that I can poke at it from
Emacs. One habit that I noticed comes up a lot is that I want to use
jQuery as I fiddle with the page, but jQuery isn’t actually loaded for
this page. What I’ll do is visit a jQuery script in Emacs and load
this buffer (C-c C-k). As expected, this is tedious and easily
automated.
Rather than add specific support for jQuery, I thought it would be
more useful to hook into one of the existing JavaScript package
managers. Not only would I get jQuery but I’d be able to load anything
else provided by the package manager. This means if I learn about a
cool new library, chances are I could just switch to my *javascript*
scratch buffer, load the library with this new Skewer feature, and
play with it. Very convenient.
How it Works
There are a number of package managers out there. I chose
Bower because of its emphasis on client-side JavaScript and, more so,
because its infrastructure is so simple that I wouldn’t actually
need to use Bower itself to access it. In adding this feature to
Skewer, I wrote half a Bower client from scratch very easily.
The only part of the Bower infrastructure hosted by Bower itself is a
tiny registry that maps package names to Git repositories. This host
also accepts new mappings, unauthenticated, for registering new
packages. The entire database is served up as plain old JSON.
To find out what versions are available, clone this repository with
Git and inspect the repository tags. Tags that follow the
Semantic Versioning scheme are versions of the package
available for use with Bower. Once a version is specified, look at
bower.json in the tree-ish referenced by that tag to get the rest of
the package metadata, such as dependencies, endpoint listing, and
description.
This is all very clever. The Bower registry doesn’t have to host any
code, so it remains simple and small. It could probably be rewritten
from scratch in 15-30 minutes. Almost all the repositories are on
GitHub, which most package developers are already comfortable with.
Package maintainers don’t need to use any tools or interact with any
new host systems. Except for adding some metadata they just keep doing
what they’re doing. I think this last point is a big part of
MELPA’s success.
Bower’s Fatal Weaknesses
Unfortunately Bower has two issues, one of which is widespread, that
seriously impacts its usefulness.
Dependency Specification
Even though Bower specifies Semantic Versioning for package
versions, which very precisely describes version syntax and
semantics, the dependencies field in bower.json is
underspecified. There’s no agreed upon method for specifying
relative dependency versions.
Say your package depends on jQuery and it relies on the newer jQuery
1.6 behavior of attr(). You would mark down that you depend on
jQuery 1.6.0. Say a user of your package is also using another package
that depends on jQuery, it’s using the on() method, which requires
jQuery 1.7 or newer. It specifies jQuery 1.7.0. This is a dependency
conflict.
Of course your package works perfectly fine with 1.7.0. It works fine
at 1.6.0 and later. In other package management systems, you would
probably have marked that you depend on “>=1.6.0” rather than just
1.6.0. Unfortunately, Bower doesn’t specify this as a valid dependency
version. Some package maintainers have gone ahead and specified
relative versions anyway, but inconsistently. Some use the “>=” prefix
like I did above, some prefix with “~” (“about this version”), which
is pretty useless.
And this leads into the other flaw.
Most Bower Packages are Broken
While some parts of Bower are underspecified, most packages don’t
follow the simple specifications that already exist! That is to say,
most Bower packages are broken. This is incredibly unfortunate
because it means at least half of the packages can’t be loaded by this
new Skewer feature.
How are they broken? As of this writing, there are 2,195 packages list
in Bower’s registry.
113 (5%) of them have unreachable or unresponsive repositories.
About half of these are due to invalid repository URLs.
1,830 (83%) have no bower.json metadata file. This means the client
has to guess at the metadata.
1,034 (47%) have unguessable endpoints. My client looks for other
package management metadata outside of Bower’s, as well as tries to
guess base on the package name. Failing to guess causes the package
to fail to load. These packages aren’t a subset of the last set
with missing bower.json files. Sometimes the bower.json files
contain incorrect information, which causes my client to drop into
guessing mode.
1400 (64%) don’t use Semantic Versioning: either no versioning at
all or some other arbitrary versioning system.
In total, 2041 (93%) of all Bower packages have invalid or
missing metadata — bad registry entry, missing bower.json file,
or lack of semantic version tags.
The good news is that most of the important libraries, like jQuery and
Underscore, work properly. I’ve also registered two of my JavaScript
libraries, ResurrectJS and rng-js, so these can be
loaded on the fly in Skewer.
Nine years ago I bought my first laptop computer. For the first time I
could carry my computer around and do productive things at places
beyond my desk. In the meantime a new paradigm of mobile computing has
arrived. Following a similar pattern, this month I bought a Samsung
Galaxy Note 10.1, an Android tablet computer. Having never owned a
smartphone, this is my first taste of modern mobile computing.
Once the technology caught up, laptops were capable enough to fully
replace desktops. However, this tablet is no replacement for my
laptop. Mobile devices are purely for consumption, so I will
continue to use desktops and laptops for the majority of my computing.
I’m writing this post on my laptop, not my tablet, for example.
Owning a tablet has opened up a whole new platform for me to explore
as a programmer. I’m not particularly interested in writing Android
apps, though. I’m obviously not alone in this, as I’ve found that
nearly all Android software available right now is somewhere between
poor and mediocre in quality. The hardware was worth the cost of the
device, but the software still has a long way to go. I’m optimistic
about this so I have no regrets.
A New Web Platform
Instead, I’m interested in mobile devices as a web platform. One of
the few high-quality pieces of software on Android are the web
browsers (Chrome and Firefox), and I’m already familiar with
developing for these. Even more, I can develop software live on the
tablet remotely from my laptop using Skewer —
i.e. the exact same development tools and workflow I’m already using.
What’s new and challenging is the user interface. Instead of
traditional clicking and typing, mobile users tap, hold, swipe, and
even tilt the screen. Most challenging of all is probably
accommodating both kinds of interfaces at once.
One of the first things I wanted to play with after buying the tablet
was the gyro. The tablet knows its acceleration and orientation at all
times. This information can be accessed in JavaScript using
a fairly new API. The two events of interest are
ondevicemotion and ondeviceorientation. Using
simple-httpd I can transmit all this information
to Emacs as it arrives.
Instead of writing a new servlet for this, to try it out I used
skewer.log(). Connect a web page viewed on the tablet to Skewer
hosted on the laptop, then evaluate this in a js2-mode buffer on the
laptop.
It would be neat to pump these into a 3D plot display as they come in,
such that my laptop displays the current tablet orientation on the
screen as I move it around, but I didn’t see any quick way to do this.
Here are some acceleration values at rest. Since I took these samples
on Earth the units are obviously in meters per second per second.
Now that I have the hardware for it, I really want to use this API to
do something interesting in a web application. I just don’t have any
specific ideas yet.
Undo in region is one of those mind-blowing things about emacs.
However, the region keeps jumping about when I use it. So I added this:
;; Keep region when undoing in region
(defadviceundo-tree-undo (around keep-region activate)
(if (use-region-p)
(let ((m (set-marker (make-marker) (mark)))
(p (set-marker (make-marker) (point))))
ad-do-it
(goto-char p)
(set-mark m)
(set-marker p nil)
(set-marker m nil))
ad-do-it))
Now the region stays in place while I'm undoing.
Since I use undo-tree, that's what it advises, but I would guess it
works the same for regular old undo too.
I created this macro to help me set up local vars. So in the
example, any files in projects/zombietdd will see
these slime-js-target-url and the find-file-in-projects patterns.
I keep these in a projects-folder to keep track of all the different
settings for my projects.
Reflection from the future: This library is super slow and
inefficient. It should probably not be used for anything serious.
Last weekend I had the itch to play around with a multiple-inheritance
prototype-based object system in lisp. It would
look a lot like JavaScript’s object system but wanted to try
experimenting some different ideas. My favorite lisp to hack in is
Emacs Lisp, so that’s what I built it on. What I ended up with is
actually pretty neat. Despite the lack of reader macros in Elisp, I
still managed to introduce new syntax by manipulating symbols at
compile time.
See the README for a quick demonstration. What follows is the long
explanation.
It’s called @, due to the syntax that it adds to Elisp as a
domain-specific language. It’s a mini-language, really. The name is also
a challenge to the code that supports Elisp, because so much of it —
including emacs-lisp-mode and Paredit — doesn’t properly handle @ in
identifiers. Even Maruku, the Markdown to HTML translator I
use for this blog, has bugs that won’t allow it to handle the @
characters in my code, so I had to forgo most syntax highlighting for
this post. (Update: I now use Kramdown so this is no longer an issue.)
Fortunately requiredoes manage just fine.
(require'@)
Objects in @ are vectors with the symbol @ as the first element. The
rest of the elements are implementation specific, but, at the moment,
the second element is a plist (property list) of all of that object’s
properties.
The root object of @ is @, and all other objects are instances of this
object, either directly or indirectly. Because it’s prototype based,
creating a new object is a matter of extending one or more
(multiple-inheritance) existing objects. This is done with the
function @extend.
;; Create a brand new object(defvarfoo(@extend@))
If no objects are given to @extend, @ will be used as the parent
object, so it’s not necessary as an argument above. This is actually
very important, as objects that don’t inherit from @ will not work at
all! I’ll get into that detail in a bit. Additionally, @extend
accepts keyword arguments, which become properties on the created
object.
The function @ is used to access properties on an object. Remember,
Elisp is a lisp-2 meaning that variables and functions exist in
their own namespaces. This means there can be both a variable @ (the
root object) and function @ (property accessor).
Like JavaScript, methods are just functions stored in properties on an
object. In @, the first argument for a method is the object itself,
which is called @@ by convention.
Here’s the first really neat part. I find all that (@ @@ ...)
business to be visually unpleasing. Fortunately this can be fixed by
adding syntax. The macro def@ transforms variables that look like @:
into these @ accessors. The following declaration is equivalent to the
lambda assignment above. It’s meant to be very convenient.
(def@rectangle:area()(*@:width@:height))
This macro walks the body of the function at compile-time (macro
expansion time) and transforms these symbols into the full @ calls
above. Like most lisp macros, this has no run-time performance cost.
Because using funcall all the time and remembering to pass the
object as the first argument is tedious, the @! function is provided
for calling methods.
(@!rectangle:area); => 52
The @: variables become function calls when in function position.
(def@rectangle:double-area()(*2(@:area))
In a lisp-1 this would happen for free, but in Elisp this situation
expands to the @! form.
Inheritance
This rectangle is starting to look like a nice re-usable object.
There’s a @ convention for this: prefix “class” object names with @.
Notice that the foo object doesn’t actually have an :area property
on itself. It was found on its parent, @rectangle by inheritance.
:width and :height were not looked up on the parent because
they’re already bound on foo.
Here’s another re-usable prototype. Notice that @: variables are
also setf-able — using push in this case.
The object system has multiple-inheritance, so colored rectangles can
be created from these two objects. The parent objects of an object are
listed in the :proto property as a list (similar to JavaScript’s
__proto__), which can be modified at any time to change an object’s
prototype chain.
Even though the initial property was read from the parent, the
assignment (push), like all assignments, actually occurred on foo.
Setters and Getters
Remember how I said that objects that don’t eventually inherit from @
will be broken? This is because properties are actually set and
accessed through :set and :get methods. That is, @ calls these
methods as needed. The @ object provides the default actions for
these. An interesting part of the @ code: initially setting :set on
@ is a circularity problem, so there’s a special bootstrap step to
accomplish it.
By providing your own you can fundamentally change how your object
works. For example, here’s an @immutable mix-in which prevents all
property assignments. It’s provided as part of @.
(defvar@immutable(@extend))(def@@immutable:set(property_value)(error"Object is immutable, cannot set %s"property))
This :set method will be found before the @ :set method, so it
gets overridden.
Remember how I said all object have a :proto that can be used to
modify the objects inheritance? This can be used to freeze an
object’s properties in place. Here’s a :freeze method for all
objects.
(def@@:freeze()"Make this object immutable."(push@immutable@:proto))
Pretty cool, eh?
The :get method can be used to provide virtual properties.
(defvar@squares(@extend))(def@@squares:get(property)(if(numberpproperty)(exptproperty2)(@^:getproperty))); explained in a moment(mapcar(lambda(n)(@@squaresn))'(01234)); => (0 1 4 9 16)
I use this technique in the @vector class under lib/ to expose the
elements of the internal vector as if they were properties.
Brian used this trick to make a @buffer prototype that wraps
Emacs’ buffers, with methods provided virtually by :get. For
example, the :string property would return a lambda that calls
buffer-string.
With multiple-inheritance and these setters and getters, there are a
lot of interesting mix-in possibilities. I’m only just discovering
some of them now.
Supermethods
Sometimes it’s really useful to call supermethods. There’s syntax for
this: @^:. This calls the next method of that name in the prototype
chain. For example, here’s a @watchable mix-in (also provided by @)
that allows other code to be notified of changes to an object. It
needs to override :set but still call the original :set.
This behavior is also used for constructors. By convention, the
:init method is the constructor. It should generally call the next
constructor with (@^:init). @ has a no-op, no-argument :init
method to bottom-out this process.
As shown, the :new method provided by the @ object combines both
@extend and :init to provide simple single-object inheritance.
The Cost of @
In the lib/ directory there are a bunch of example objects
implemented: including @vector, @queue, @stack, and @heap. I found
these to be very enjoyable to write, and they’ve been the testing
grounds for @. @heap uses an internal @vector instance and exercises
@’s features the most.
The performance cost of @ very apparent with @heap. Even byte-compiled
it’s slower than the naive implementation (compose push and sort)
for even as high as 1,000 elements. While I think @ leads to elegant
code, there’s still plenty to do for performance. It’s comically slow.
This really caught Brian’s interest, because it was an opportunity to
put on his programming language designer’s hat — which I believe to
be his favorite hat. He’s been trying different caching strategies to
reduce all the walking of the prototype chain. This effort can be
found in the other repository branches and in his fork. The system is
so dynamic that cache invalidation is a really complex problem.
Every time a property is set, @ has to find the :set property for
that object, which generally means walking all the way up to @.
Because :proto can be modified at any time, every property look-up
requires computing the precedence order (lazily). This all makes
property assignment quite expensive! I can understand why real object
systems aren’t this flexible. It comes at a high price.
How many times should a random number from [0, 1] be drawn to have
it sum over 1?
If you want to figure it out for yourself, stop reading now and come
back when you’re done.
The answer is e. When I came across this question I took
the lazy programmer route and, rather than work out the math, I
estimated the answer using the Monte Carlo method. I used the language
I always use for these scratchpad computations: Emacs Lisp. All I need
to do is switch to the *scratch* buffer and start hacking. No
external program needed.
The downside is that Elisp is incredibly slow. Fortunately, Elisp is
so similar to Common Lisp that porting to it is almost trivial. My
preferred Common Lisp implementation, SBCL, is very, very fast so it’s
a huge speed upgrade with little cost, should I need it. As far as I
know, SBCL is the fastest Common Lisp implementation.
Even though Elisp was fast enough to determine that the answer is
probably e, I wanted to play around with it. This little test
program doubles as a way to estimate the value of e,
similar to estimating pi. The more trial runs I give it the
more accurate my answer will get — to a point.
Here’s the Common Lisp version. (I love the loop macro, obviously.)
Using SBCL 1.0.57.0.debian on an Intel Core i7-2600 CPU, once
everything’s warmed up this takes about 9.4 seconds with 100 million
trials.
(time (monte-carlo 100000000))
Evaluation took:
9.423 seconds of real time
9.388587 seconds of total run time (9.380586 user, 0.008001 system)
99.64% CPU
31,965,834,356 processor cycles
99,008 bytes consed
2.7185063
Since this makes for an interesting benchmark I gave it a whirl in
JavaScript,
I ran this on Chromium 24.0.1312.68 Debian 7.0 (180326) which uses V8,
currently the fastest JavaScript engine. With 100 million trials,
this only took about 2.7 seconds!
monteCarlo(100000000);// ~2.7 seconds, according to Skewer// => 2.71850356
Whoa! It beat SBCL! I was shocked. Let’s try using C as a
baseline. Surely C will be the fastest.
I used the highest optimization setting on the compiler.
$ gcc -ansi -W -Wall -Wextra -O3 temp.c
$ time ./a.out
2.718359
real 0m3.782s
user 0m3.760s
sys 0m0.000s
Incredible! JavaScript was faster than C! That was completely
unexpected.
The Circumstances
Both the Common Lisp and C code could probably be carefully tweaked to
improve performance. In Common Lisp’s case I could attach type
information and turn down safety. For C I could use more compiler
flags to squeeze out a bit more performance. Then maybe they could
beat JavaScript.
In contrast, as far as I can tell the JavaScript code is already as
optimized as it can get. There just aren’t many knobs to tweak. Note
that minifying the code will make no difference, especially since I’m
not measuring the parsing time. Except for the functions themselves,
the variables are all local, so they are never “looked up” at
run-time. Their name length doesn’t matter. Remember, in JavaScript
global variables are expensive, because they’re (generally) hash
table lookups on the global object at run-time. For any decent
compiler, local variables are basically precomputed memory offsets —
very fast.
The function names themselves are global variables, but the V8
compiler appears to eliminate this cost (inlining?). Wrapping the
entire thing in another function, turning the two original functions
into local variables, makes no difference in performance.
While Common Lisp and C may be able to beat JavaScript if time is
invested in optimizing them — something to be done rarely — in a
casual implementation of this algorithm, JavaScript beats them both. I
find this really exciting.
An Emacs buffer always has one major mode and zero or more minor
modes. Major modes tend to be significant efforts, especially when it
comes to automatic indentation. In contrast, minor modes are often
simple, perhaps only overlaying a small keymap for additional
functionality. Creating a new minor mode is really easy, it’s just a
matter of understanding Emacs’ conventions.
Mode names should end in -mode and the command for toggling the mode
should be the same name. They keymap for the mode should be called
mode-map and the mode’s toggle hook should be called
mode-hook. Keep all of this in mind when picking a name for your
minor mode.
There are a number of other tedious issues that need to be taken into
account when manually building a minor mode. The good news is that no
one needs to worry about most of it! Lisp has macros for cutting down
on boilerplate code and so there’s a macro for this very purpose:
define-minor-mode. Here’s all it takes to make a new minor mode,
foo-mode.
(define-minor-modefoo-mode"Get your foos in the right places.")
This creates a command foo-mode for toggling the minor mode and a
hook called foo-mode-hook. There’s a strange caveat about the hook:
it’s not immediately declared as a variable. My guess is that this is
some archaic optimization which now exists as bad design. The hook
function add-hook will create this variable lazily when needed and
the function run-hooks will ignore hook variables that don’t yet
exist, so it doesn’t get tripped up by this situation. So despite its
strange initial absence, the new minor mode will use this hook as
soon as functions are added to it.
Minor Mode Options
This mode doesn’t do anything yet. It doesn’t have its own keymap
and it doesn’t even show up in the modeline. It’s just a toggle and a
hook that’s run when the toggle is used. To add more to the mode,
define-minor-mode accepts a number of keywords. Here are the
important ones.
:lighter: the name, a string, to show in the modeline
:keymap: the mode’s keymap
:global: specifies if the minor mode is global
The :lighter option has one caveat: it’s concatenated to the rest of
the modeline without any delimiter. This means it needs to be prefixed
with a space. I think this is mistake, but we’re stuck with it
probably forever. Otherwise this string should be kept short: there’s
generally not much room on the modeline.
(define-minor-modefoo-mode"Get your foos in the right places.":lighter" foo")
New, empty keymaps are created with (make-keymap) or
(make-sparse-keymap). The latter is more efficient when the map will
contain a small number of keybindings, as is the case with most minor
modes. The fact that these separate functions exist is probably
another outdated, premature optimization. To avoid confusing others, I
recommend you use the one that matches your intended usage.
The keymap can be provided directly to :keymap and it will be bound
to foo-mode-map automatically. I could just put an empty keymap here
and define keys separately outside the define-minor-mode
declaration, but I like the idea of creating the whole map in one
expression.
(defuninsert-foo()(interactive)(insert"foo"))(define-minor-modefoo-mode"Get your foos in the right places.":lighter" foo":keymap(let((map(make-sparse-keymap)))(define-keymap(kbd"C-c f")'insert-foo)map))
The :global option means the minor mode is not local to a buffer,
it’s present everywhere. As far as I know, the only global minor mode
I’ve ever used is YASnippet.
Minor Mode Body
The rest of define-minor-mode is a body for arbitrary Lisp, like a
defun. It’s run every time the mode is toggled off or on, so it’s
like a built-in hook function. Use it to do any sort of special setup
or teardown, such hooking or unhooking Emacs’ hooks. A likely thing to
be done in here is specifying buffer-local variables.
Any time the Emacs interpreter is evaluating an expression there’s
always a current buffer acting as context. Many functions that
operate on buffers don’t actually accept a buffer as an
argument. Instead they operate on the current buffer. Furthermore,
some variables are buffer-local: the binding is dynamic over the
current buffer. This is useful for maintaining state relevant only to
a particular buffer.
Side note: the with-current-buffer macro is used to specify a
different current buffer for a body of code. It can be used to access
other buffer’s local variables. Similarly, with-temp-buffer creates
a brand new buffer, uses it as the current buffer for its body, and
then destroys the buffer.
For example, let’s say I want to keep track of how many times
foo-mode inserted “foo” into the current buffer.
(defvarfoo-count0"Number of foos inserted into the current buffer.")(defuninsert-foo()(interactive)(setqfoo-count(1+foo-count))(insert"foo"))(define-minor-modefoo-mode"Get your foos in the right places.":lighter" foo":keymap(let((map(make-sparse-keymap)))(define-keymap(kbd"C-c f")'insert-foo)map)(make-local-variable'foo-count))
The built-in function make-local-variable creates a new buffer-local
version of a global variable in the current buffer. Here, the
buffer-local foo-count will be initialized with the value 0 from the
global variable but all reassignments will only be visible in the
current buffer.
However, in this case it may be better to use
make-variable-buffer-local on the global variable and skip the
make-local-variable. The main reason is that I don’t want
insert-foo to clobber the global variable if it happens to be used
in a buffer that doesn’t have the minor mode enabled.
(make-variable-buffer-local(defvarfoo-count0"Number of foos inserted into the current buffer."))
A big advantage is that this buffer-local intention for the variable
is documented globally. This message will appear in the variable’s
documentation.
Automatically becomes buffer-local when set in any fashion.
Which method you use is up to your personal preference. The Emacs
documentation encourages the former but I think the latter is nicer
in many situations.
Automatically Enabling the Minor Mode
Some minor modes don’t have any particular major mode association and
the user will toggle it at will. Some minor modes only make sense when
used with particular major mode and it might make sense to
automatically enable along with that mode. This is done by hooking
that major mode’s hook. So long as the mode follows Emacs’ conventions
as mentioned at the top, this hook should be easy to find.
(add-hook'text-mode-hook'foo-mode)
Here, foo-mode will automatically be activated in all text-mode
buffers.
Full Code
Here’s the final code for our minor mode, saved to foo-mode.el. It
has one keybinding and it’s easily open for users to define more keys
in foo-mode-map. It also automatically activates when the user is
editing a plain text file.
(make-variable-buffer-local(defvarfoo-count0"Number of foos inserted into the current buffer."))(defuninsert-foo()(interactive)(setqfoo-count(1+foo-count))(insert"foo"));;;###autoload(define-minor-modefoo-mode"Get your foos in the right places.":lighter" foo":keymap(let((map(make-sparse-keymap)))(define-keymap(kbd"C-c f")'insert-foo)map));;;###autoload(add-hook'text-mode-hook'foo-mode)(provide'foo-mode)
I added some autoload declarations and a provide in case this mode
is ever distributed or used as a package. If an autoloads script is
generated for this minor mode, a temporary function called foo-mode
will be defined whose sole purpose is to load the real foo-mode.el
and then call foo-mode again with its new definition, which was
loaded overtop the temporary definition.
The autoloads script also adds this temporary foo-mode function to
the text-mode-hook. If a text-mode buffer is created, the hook
will call foo-mode which will load foo-mode.el, redefining
foo-mode to its real definition, then activate foo-mode.
The point of autoloads is to defer loading code until it’s needed. You
may notice this as a short delay the first time you activate a mode
after starting Emacs. This is what keeps Emacs’ start time reasonable
despite having millions of lines of Elisp virtually loaded at startup.
Ever since
I started using the Emacs package archive,
specifically MELPA, I’d been wanting to tidy up
my Emacs Java extensions, java-mode-plus, into a
nice, official package. Observing my own attitude after the switch, I
noticed that if a package isn’t available on ELPA or MELPA, it
practically doesn’t exist for me. Manually installing anything now
seems like so much trouble in comparison, and getting a package on
MELPA is so easy that there’s little excuse for package authors not to
have their package in at least one of the three major Elisp
archives. This is exactly the attitude my own un-archived package
would be facing from other people, and rightfully so.
Before I dive in, this is what the user configuration now looks like,
That’s right: it knows how to find, fetch, and index documentation on
its own. Keep reading if this sounds useful to you.
The Problem
The problem was that java-mode-plus was doing two unrelated things:
Supporting Ant-oriented Java projects. Not
being a fan of Maven, I’ve used Ant for all of my own
personal projects. (However, I really do like the Maven
infrastructure, so I use Apache Ivy.) It seems Maven is a lot more
popular, so this part isn’t useful for many people.
Quick Javadoc referencing, which I was calling java-docs. I think
this is generally useful for anyone writing Java in Emacs, even if
they’re using another suite like JDEE or writing in another JVM
language. It would be nice for people to be able to use this without
pulling in all of java-mode-plus — which was somewhat intrusive.
I also didn’t like the names I had picked. java-mode-plus wasn’t even
a mode until recently and its name isn’t conventional. And “java-docs”
is just stupid. I recently solved all this by splitting the
java-mode-plus into two new packages,
ant-project-mode — A minor mode that
performs the duties of the first task above. Since I’ve
phased Java out from my own personal projects
and no longer intend to write Java anymore, this part isn’t very
useful to me personally at the moment. If I do need to write Java for
work again I’ll probably dust this off. It’s by no means
un-maintained, it’s just in maintenance mode for now. Because of
this, this is not in any Emacs package archive
javadoc-lookup — This is java-docs renamed and
with some new goodies! I also put this on MELPA, where it’s
easy for anyone to use. This is continues to be useful for me as I
use Clojure.
javadoc-lookup
This is used like java-docs before it, just under a different
name. The function javadoc-lookup asks for a Java class for
documentation. I like to bind this to C-h j.
The function javadoc-add-roots provides filesystem paths to be
indexed for lookup.
Also, as before, if you don’t provide a root for the core Java API, it
will automatically load an index of the official Javadoc hosted
online. This means it can be installed from MELPA and used immediately
without any configuration. Good defaults and minimal required
configuration is something I highly value.
Back in the java-docs days, when I started using a new library I’d
track down the Javadoc jar, unzip it somewhere on my machine, and add
it to be indexed. I regularly do development on four different
computers, so this gets tedious fast. Since the Javadoc jars are
easily available from the Maven repository, I maintained a small Ant
project within my .emacs.d for awhile just to do this fetching, but it
was a dirty hack.
Finally, the Goodies
Here’s the cool new part: I built this functionality into
javadoc-lookup. It can fetch all your documentation for you!
Instead of providing a path on your filesystem, you name an artifact
that Maven can find. javadoc-lookup will call Maven to fetch the
Javadoc jar, unzip it into a cache directory, and index it for
lookups. You will need Maven installed either on your $PATH or at
maven-program-name (Elisp variable).
Here’s a sample configuration. It’s group, artifact, version provided
as a sequence. I say “sequence” because it can be either a list or a
vector and those names can be either strings or symbols. I prefer the
vector/symbol method because it requires
the least quoting, plus it looks Clojure-ish.
Put that in your initialization and all this documentation will appear
in the lookup index. It only needs to fetch from Maven once per
artifact per system — a very very slow process. After that it
operates entirely from its own cache which is very fast, so it won’t
slow down your startup.
This has been extremely convenient for me so I hope other people find
it useful, too.
As a final note, javadoc-lookup also exploits structural sharing in
its tables, using a lot less memory than java-docs. Not that it was a
problem before; it’s a feel-good feature.
;; Function to create new functions that look for a specific pattern
(defunffip-create-pattern-file-finder (&rest patterns)
(lexical-let ((patterns patterns))
(lambda ()
(interactive)
(let ((ffip-patterns patterns))
(find-file-in-project)))))
;; Find file in project, with specific patterns
(global-unset-key (kbd "C-x C-o"))
(global-set-key (kbd "C-x C-o ja")
(ffip-create-pattern-file-finder "*.java"))
(global-set-key (kbd "C-x C-o js")
(ffip-create-pattern-file-finder "*.js"))
(global-set-key (kbd "C-x C-o jp")
(ffip-create-pattern-file-finder "*.jsp"))
This function limits the search to files of a specific file type.
I've got loads more of these keybindings, all of them with the
two-letter mnemonic shortcut.
It really speeds up finding files. Both
because ido-completing-read has less matches to worry
about, because there are fewer similarly named files, and especially
when the .java, the .js and the .jsp share a name.
But sometimes I just want to browse a little. Or move a few lines
down. These keybindings let me do that more quickly
than C-nC-nC-nC-nC-nC-n
...
In fact, with these I can navigate to any line within a distance of
11 in 3 keystrokes or less. Or close enough to count. Two of them
require 4 keystrokes. Can you figure out which ones?
This evening Skewer gained support for live CSS.
When editing CSS code, you can send your rules and declarations from
the editing buffer to be applied in the open page in the browser. It
makes experimenting with CSS really, really easy. The functionality is
exposed through the familiar interaction keybindings, so if you’re
already familiar with other Emacs interaction modes
(SLIME, nREPL, Skewer,
Geiser, Emacs Lisp), this should feel right at home.
To provide the keybindings in css-mode there’s a new minor mode,
skewer-css-mode. CSS “expressions” are sent to the browser through the
communication channel already provided by Skewer. It’s essentially an
extension to Skewer: it could have been created without making any
changes to Skewer itself.
Unfortunately Emacs’ css-mode is nowhere near as sophisticated as
js2-mode — which reads in and exposes a full JavaScript AST. I had to
write my own very primitive CSS parsing routines to tease things
apart. It should generally be able to parse declarations and rules
reasonably no matter how it’s indented, but it’s not very good at
navigating around comments, especially when they contain CSS
syntax. If I find a way to parse CSS more easily sometime I’ll see
about fixing it, but it’s plenty good enough for now.
To “evaluate” the CSS, the code is simply dropped into the page as a
new <style> tag. I had considered other approaches, but this seemed
to be by far the simplest way to support arbitrary selectors and
shorthand properties. The more programmatic approaches would require
re-writing something that browser already does.
The consequence of this is that every “evaluation” adds a new
<style> tag to the page, which adds more and more load to style
computation, most of which completely mask each other. Since there’s
no way to tell when a particular <style> tag has been completely
masked I can’t remove any of them from the page. That might revert a
declaration that’s still in usde. I haven’t seen it happen yet but I
wonder if it’s possible to run into browser problems during extended
CSS interaction, when thousands of stylesheets have built up on a
single page. Time will tell.
Just before doing all this, I added full support for Cross-resource
Resource Sharing (CORS), which means any page from any server can be
skewered, not just pages hosted by Emacs itself … as long as you can
get skewer.js in the page as a script. To help with that, I wrote a
Greasemonkey userscript that can automatically skewer any
visited page. I can now manipulate from Emacs the JavaScript and CSS
of any page I visit in my browser. It feels really powerful. I
already have a good use for this at work right now.
Today at work I was using impatient-mode to share some code
with Brian. It makes for a really handy live pastebin. To
limit the buffer to the relevant code, I narrowed it down with
narrow-to-region. However, the browser wouldn’t update to show only
the narrowed region until I made an edit. This makes sense because
impatient-mode hooks after-change-functions. Narrowing the buffer
doesn’t change anything in the buffer, so, as expected, this hook is
not called.
The solution would be to also join whatever hook is called when the
buffer restriction changes. Unfortunately,
no such hook exists. I thought I could create this hook with
some advice, but this turns out to be currently impossible.
Emacs Advice
What’s advice? It’s a handy feature of Emacs lisp that allows users to
modify the behavior of almost any function without having to redefine
it. It works a little bit like methods in the Common Lisp Object
System (CLOS): advice is code than can be evaluated before, after, or
around a function.
Advice is defined with defadvice. Duh. For example, say we wanted to
be silly and have Emacs say “Ouch!” when a line is killed with
kill-line. We can advise this function to display a message.
This says we want to advise the function kill-line, we want this
advise to execute afterkill-line has run, our advice is named
“say-ouch”, and we want to immediately activate this advice so it
gets used right away. The rest is the body of the advice, like the
body of a function. After evaluating this defadvice, every time I
hit C-k Emacs says “Ouch!” in the minibuffer. Cool!
narrow-to-region and widen
A hook is a variable that holds a list of functions. (Or maybe hooks
are the functions in this list? Emacs’ documentation calls both of
these things hooks.) These functions are called, usually without
arguments, when some specific event occurs. For example, every mode
has its own mode hook which is called when the mode is activated in a
buffer. This allows users to extend or modify the mode — like by
enabling additional minor modes — without editing the mode’s source
code directly.
To make our hook work we need to advise narrow-to-region and widen
to run the hook after they’ve done their work. These are the primitive
narrowing functions which all the other narrowing functions eventually
call, like narrow-to-defun, narrow-to-page, and any other
mode-specific narrowing. Advising these two functions will cover all
buffer narrowing. It should be this simple.
At first this seems to work. I can add a test hook see them activate
when I use M-x narrow-to-region and M-x widen. However, when I use
other narrowing functions, like narrow-to-defun, my hook functions
aren’t called.
Is there a narrowing primitive I missed? I check the source
code. Nope, these are lisp functions which ultimately call
narrow-to-region. Is the advice not getting used when called
indirectly? I test that out.
(defunfoo()(interactive)(narrow-to-region12))
This works fine. Hmmm, these other functions are byte-compiled, maybe
that’s the problem.
(byte-compile'foo)
Bingo. The advice has stopped working. It has something to do with
byte-compilation.
I don’t know too much about Emacs’ byte code, but here’s the gist of
it. A compiled function is a special type of vector (hence the #[]
form). This is a legal s-expression which you can use directly in
regular Elisp code just like it was a function. The only reason you’d
do so is for obfuscation, so it would look very suspicious.
The first element of this function vector is the parameter list —
empty in this case. The second is a string containing the actual
bytecodes. The rest holds the various constants from the function
body. This includes the symbols of other functions called by this
function. It’s important to note that narrow-to-region does not
appear in this list!
Curious. Let’s take a closer look at the bytecode.
Looking at bytecomp.el from the Emacs distribution I can see that
codes 192 and 193 are used for accessing constants. This pushes my
constants 1 and 2 onto a stack for use as function arguments. Next up
is 125, which corresponds to byte-narrow-to-region. Gotcha!
It turns out narrow-to-region is so special — probably because it’s
used very frequently — that it gets its own bytecode. The primitive
function call is being compiled away into a single instruction. This
means my advice will not be considered in byte-compiled code. Darnit.
The same is true for widen (code 126).
Where to go now?
Since it’s not possible to hook or advise the buffer-narrowing
primitives, impatient-mode would need to hook some other event that
tends to happen at the same time. Perhaps any time a command is
executed in the current buffer it could check for changes to the
buffer restriction and, if so, update any attached web clients. I’ll
figure something out.
First of all, Meta M- needs to be really easy to hit. On a Mac
keyboard, that means Command - and not the default Option - since
we want the key that is right next to Space.
The good news is that now Option is available for
Super s-. And even more amazing, you can also bind the
Function-key to Hyper H- - without losing the ability to
change the volume or pause/play.
So now I can use crazy keybindings
like H-SPChyperspace. I haven't entirely
decided what I should be using this newfound superpower for, but one
thing I've done is reserve all the C-s- prefixed letters for
refactorings with js2-refactor,
as you can see here.
I love the symbiosis between expand-region and delete-selection-mode.
;; making paredit work with delete-selection-mode
(put 'paredit-forward-delete 'delete-selection 'supersede)
(put 'paredit-backward-delete 'delete-selection 'supersede)
(put 'paredit-open-round 'delete-selection t)
(put 'paredit-open-square 'delete-selection t)
(put 'paredit-doublequote 'delete-selection t)
(put 'paredit-newline 'delete-selection t)
This makes paredit-mode work with delete-selection-mode, replacing
its wrapping behavior. If I want to wrap, I'll do it with the
paredit-wrap-* commands explicitly.
Yesterday Kototama commented about another neat paredit addition: duplicating sexps. This is my take on that:
(defunparedit--is-at-start-of-sexp ()
(and (looking-at "(\\|\\[")
(not (nth 3 (syntax-ppss))) ;; inside string
(not (nth 4 (syntax-ppss))))) ;; inside comment
(defunparedit-duplicate-closest-sexp ()
(interactive)
;; skips to start of current sexp
(while (not (paredit--is-at-start-of-sexp))
(paredit-backward))
(set-mark-command nil)
;; while we find sexps we move forward on the line
(while (and (bounds-of-thing-at-point 'sexp)
(<= (point) (car (bounds-of-thing-at-point 'sexp)))
(not (= (point) (line-end-position))))
(forward-sexp)
(while (looking-at " ")
(forward-char)))
(kill-ring-save (mark) (point))
;; go to the next line and copy the sexprs we encountered
(paredit-newline)
(yank)
(exchange-point-and-mark))
Like Kototama
says in
his blogpost, duplicating a line is very useful, but sometimes
it leads to invalid sexps. In the blogpost he shows a snippet that
will duplicate the sexp after point. I immediately realized I had
really been wanting this.
The version listed here is a little modified: It will duplicate the
sexp you are currently inside, or looking at, or looking behind at.
So basically, point can be in any of these positions:
|(my sexp) ;; in front
(my| sexp) ;; inside
(my sexp)| ;; at the end
With point in front of a sexp, paredit-wrap-round
(bound to M-(), will open a paren in front the the sexp,
and place the closing paren at the end of it. That's pretty handy.
This snippet does the same, but from the other end. It saves me
a C-M-b ever so often. I like it.
As a new user of nREPL I was poking around
nrepl.el, seeing what sorts of Elisp tricks I could learn. Even though
it was written 6 months before Skewer, and I was completely unaware of
nREPL’s existence until two weeks ago, there’s a lot of similarity
between nrepl.el and Skewer. Due to serving the
same purpose for different platforms, this isn’t very surprising.
In particular, Skewer has skewer-eval for sending a string to the
browser for evaluation. Like JavaScript, Emacs Lisp is
single-threaded: there’s only one execution context at a time and it
has to return to the top-level before a new context can execute. There
are no continuations or coroutines. skewer-eval requires
coordination with an external process (the browser) making it
inherently asynchronous. So as a second, optional argument, a callback
can be provided for receiving the result.
;; Echo the result in the minibuffer.(skewer-eval"Math.pow(2.1, 3.1)"(lambda(r)(message(cdr(assoc'valuer)))))
However, the equivalent function in nrepl.el, nrepl-eval, is
synchronous! It returns the evaluation result. “That’s not true!
That’s impossible!”
Well, it turns out what I said above about execution contexts wasn’t
completely true. There’s exactly one sneaky function that breaks the
rule: accept-process-output. It blocks the current execution context
allowing some other execution contexts to run, including timers and
I/O. However, it will lock up Emacs’ interface. nrepl-eval uses this
function to poll for a response from the nREPL process.
When I saw this, a lightbulb went off in my head. This lone loophole
in Emacs execution model can be abused to provide interesting
benefits. Specifically, it can be used to create a latch
synchronization primitive.
The full source code is here if you want to dive right in. I’ll be
going over a simplified version piece-by-piece below.
The idea of a latch is that a thread can wait on the latch, blocking
its execution. It will remain in that state until another thread
notifies the latch, releasing any threads blocked on the
latch. Here’s how it might look in Lisp.
(defvarresultnil)(defvarmy-latch(make-latch))(defunget-result()(ifresultresult(waitmy-latch); Block, waiting for the resultresult))(defunset-result(value)(setfresultvalue)(notifymy-latch)); Release anyone waiting on my-latch
The pattern above is similar to a promise, which we will later
implement on top of latches. In our latch implementation I’d also like
to optionally pass a value from notify to anyone waiting, which
would make the above simpler.
Emacs doesn’t have threads but instead non-preemptive execution
contexts. Ignoring the Emacs UI lockup, we can mostly ignore that
distinction for now.
To exploit accept-process-output each latch needs to have its own
process object. When blocking on a latch it will simply wait for that
process to receive input. To notify a latch, we need to send data to
that process.
For the process, we’ll ask Emacs to make a pseudo-terminal “process.”
It’s basically just a pipe for Emacs to talk to itself. It’s possible
to literally make a pipe, which is better for this purpose, but
that’s currently broken. To make such a process, we call
start-process with nil as the program name (third argument).
This class has two slots, process and value. The process slot
holds the aforementioned process we’ll be blocking on. The value
slot will be used to pass a value from notify to wait. The
process slot is initialized with a brand new process object upon
instantiation.
To wait, call accept-process-output on the latch’s private
process. This function won’t return until data is sent to the
process. By that time, the value slot will be filled in with the
value from notify.
To notify, send a newline with process-send-string. The data to send
is arbitrary, but I wanted to send as little as possible (one byte)
and I figure a newline might be safer when it comes to flushing any
sort of buffer. Buffers tend to flush on newlines. Before sending
data, we set the value slot to the value that wait will return.
That’s basically it! However, processes are not garbage collected by
Emacs, so we need a destroy destructor method. The name destroy
here is not special to Emacs. It’s something for the user of the
library to call.
I also made a convenience constructor function make-latch, with the
conventional name make-, since users shouldn’t have to call
make-instance for our classes.
That’s enough to turn skewer-eval into a synchronous function.
In combination with lexical-let, apply-partially returns a closure
that will notify the latch with the return value passed to it from
skewer. We need to get the return value from wait, destroy the
latch, then return the value, so I use a prog1 for this.
One-use Latches
In my experimenting, I noticed the prog1 pattern coming up a
lot. Having to destroy my latch after a single use was really
inconvenient. Fortunately this pattern can be captured by a subclass:
one-time-latch.
If eieio was a more capable mini-CLOS I could also demonstrate a
countdown-latch, but this would require an :around method. Most
uses of notify would need to skip over the superclass method.
Promises
We can build promises on top of our latch implementation. Basically, a
promise is a one-time-latch where we can query the notify value more
than once. In a one-time-latch we can only wait once.
Our promise will have two similar methods, deliver (like notify),
and retrieve (like wait). If a value has been delivered already,
retrieve will return that value. Otherwise, it will block and wait
until a value is delivered,
It has three slots, the one-time-latch used for blocking, a Boolean
determining the delivery status, and the value of the promise.
(defmethoddeliver((promisepromise)value)(if(slot-valuepromise'delivered)(error"Promise has already been delivered.")(setf(slot-valuepromise'value)value)(setf(slot-valuepromise'delivered)t)(notify(slot-valuepromise'latch)value)))(defmethodretrieve((promisepromise))(if(slot-valuepromise'delivered)(slot-valuepromise'value)(wait(slot-valuepromise'latch))))
A promise can only be delivered once, so it throws an error if it is
attempted more than once. Otherwise it updates the promise state and
releases anything waiting on it.
What to do with this?
Locking up Emacs’ UI really limits the usefulness of this
library. Since Emacs’ primary purpose is being a text editor, it needs
to remain very lively or else the user will become annoyed. If I used
a synchronous version of skewer-eval, Emacs would completely lock up
(easily interrupted with C-g) until the browser responds — which
would be never if no browser is connected. That’s unacceptable.
Also, not very many Emacs functions have the callback pattern. The
only core function I’m aware of that does is url-retrieve, but it
already has a url-retrieve-synchronously counterpart.
Please tell me if you have a neat use of any of this!
According to my e-mail archives I’ve been interested in
Clojure for about three and a half years now. During that
period I would occasionally spend an evening trying to pick it up,
only to give up after getting stuck on some installation or
configuration issue. With a little bit of pushing from Brian,
and the fact that this installation and configuration is now
trivial, I finally broke that losing streak last week.
I’m Damn Picky
Personally, there’s a high barrier in place to learn new programming
languages. It’s entirely my own fault. I’m really picky about my
development environment. If I’m going to write code in a language I
need Emacs to support a comfortable workflow around it. Otherwise
progress feels agonizingly sluggish. If at all possible this means
live interaction with the runtime (Lisp, JavaScript). If not, then I
need to be able to invoke builds and run tests from within Emacs (C,
Java). Basically, I want to leave the Emacs window as infrequently
possible.
I also need a major mode with decent indentation support. This tends
to be the hardest part to create. Automatic indentation in Emacs is
considered a black magic. Fortunately, it’s unusual to come across a
language that doesn’t already have a major mode written for it. It’s
only happened once for me and that’s because it was a custom language
for a computer languages course. To remedy this, I ended up
writing my own major mode, including in-Emacs evaluation.
Unsatisfied with JDEE, I did the same for Java,
growing my own extensions to support my development for the
couple of years when Java was my primary programming language. The
dread of having to switch back and forth between Emacs and my browser
kept me away from web development for years. That changed this past
October when I wrote skewer-mode to support interactive
JavaScript development. JavaScript is now one of my favorite
programming languages.
I’ve wasted enough time in my life configuring and installing
software. I hate sinking time into doing so without capturing that
work in source control, so that I never need to spend time on that
particular thing again. I don’t mean the installation itself but the
configuration — the difference from the defaults. (And the better the
defaults, the smaller my configuration needs to be.) With
my dotfiles repository and Debian, I can go from a
computer with no operating system to a fully productive development
environment inside of about one hour. Almost all of that time is just
waiting on Debian to install all its packages. Any new language
development workflow needs to be compatible with this.
Clojure Installation
Until last year sometime the standard way to interact with Clojure
from Emacs was through swank-clojure with
SLIME. Well, installing SLIME itself can be a pain.
Quicklisp now makes this part trivial but it’s specific
to Common Lisp. This is also a conflict with Common Lisp, so I’d
basically need to choose one language or the other.
SLIME doesn’t have any official stable releases. On top of this, the
SWANK protocol is undocumented and subject to change at any time. As a
result, SWANK backends are generally tied to a very specific version
of SLIME and it’s not unusual for something to break when upgrading
one or the other. I know because I wrote
my own SWANK backend for BrianScheme. Thanks to
Quicklisp, today this isn’t an issue for Common Lisp users, but it’s
not as much help for Clojure.
The good news is that swank-clojure is now depreciated. The
replacement is a similar, but entirely independent, library called
nREPL. (I’d link to it but there doesn’t seem to be a website.)
Additionally, there’s an excellent Emacs interface to it:
nrepl.el. It’s available on MELPA, so installation is
trivial.
There’s also a clojure-mode package on MELPA, so install that, too.
That covers the Emacs side of things, so what about Clojure itself?
The Clojure community is a fast-moving target and the Debian packages
can’t quite keep up. At the time of this writing they’re too old to
use nREPL. The good news is that there’s an alternative that’s just as
good, if not better: Leiningen.
Leiningen is the standard Clojure build tool and dependency
manager. Here, “dependencies” includes Clojure itself. If you have
Leiningen you have Clojure. Installing Leiningen is as simple as
placing a single shell script in your $PATH. Since I always have
~/bin in my $PATH, all I need to do is wget/curl the script there
and chmod +x it. The first time it runs it pulls down all of its own
dependencies automatically. Right now the biggest downside seems to be
that it’s really slow to start. I think the JVM warmup time is to
blame.
Let’s review. To install a working Emacs live-interaction Clojure
development environment,
Install the nrepl.el package in Emacs. For me this happens
automatically by the configuration in my
.emacs.d repository. I only had to do this step once.
Install the clojure-mode package. Same deal.
Install a JDK. OpenJDK is probably in your system’s package
manager, so this is trivial.
Put the lein shell script in the $PATH. This takes about
five seconds. If even this was too much for my precious
sensibilities I could put this script in my dotfiles repository.
With this all in place, do M-x nrepl-jack-in in Emacs and any
clojure-mode buffer will be ready to evaluate code as expected. It’s
wonderful.
Further Extending Emacs
I made some tweaks to further increase my comfort. Perhaps nREPL’s
biggest annoyance is not focusing the error buffer, like all the other
interactive modes. Once I’m done glancing at it I’ll dismiss it with
q. This advice fixes that.
(defadvicenrepl-default-err-handler(afternrepl-focus-errorsactivate)"Focus the error buffer after errors, like Emacs normally does."(select-window(get-buffer-window"*nrepl-error*")))
I also like having expressions flash when I evaluate them. Both SLIME
and Skewer do this. This uses slime-flash-region to do so when
available.
For Lisp modes I use parenface to de-emphasize parenthesis. Reading
Lisp is more about indentation than parenthesis. Clojure uses square
brackets ([]) and curly braces ({}) heavily, so these now also get
special highlighting. See my .emacs.d for that. Here’s
what it looks like,
Learning Clojure
The next step is actually learning Clojure. I already know Common Lisp
very well. It has a lot in common with Clojure so I didn’t want to
start from a pure introductory text. More importantly, I needed to
know upfront which of my pre-conceptions were wrong. This was
an issue I had, and still have, with JavaScript. Nearly all the
introductory texts for JavaScript are aimed at beginner
programmers. It’s a lot of text for very little new information.
More good news! There’s a very thorough Clojure introductory guide
that starts at a reasonable level of knowledge.
A few hours going through that while experimenting in a *clojure*
scratch buffer and I was already feeling pretty comfortable. With a
few months of studying the API, learning the idioms, and
practicing, I expect to be a fluent speaker.
I think it’s ultimately a good thing I didn’t get into Clojure a
couple of years ago. That gave me time to build up — as a sort of
rite of passage — needed knowledge and experience with Java, which
deliberately, through the interop, plays a significant role in
Clojure.
However, there is some discrepancies in
the completing-read API between newer and older
versions regarding the case where you just press enter to choose the
first item.
To fix these, some of the newer usages of completing read need a
slightly different implementation. These tweaks fix that problem.
(defuncleanup-buffer-safe ()
"Perform a bunch of safe operations on the whitespace content of a buffer.
Does not indent buffer, because it is used for a before-save-hook, and that
might be bad."
(interactive)
(untabify (point-min) (point-max))
(delete-trailing-whitespace)
(set-buffer-file-coding-system 'utf-8))
;; Various superfluous white-space. Just say no.
(add-hook 'before-save-hook 'cleanup-buffer-safe)
(defuncleanup-buffer ()
"Perform a bunch of operations on the whitespace content of a buffer.
Including indent-buffer, which should not be called automatically on save."
(interactive)
(cleanup-buffer-safe)
(indent-region (point-min) (point-max)))
(global-set-key (kbd "C-c n") 'cleanup-buffer)
I use these two literally all the time. The first one removes
trailing whitespace and replaces all tabs with spaces before save.
The last one I've got on a key - it also indents the entire buffer.
These might not be for everybody. Sometimes you do want tabs (I'm looking at
you Makefilegrrrrr). Then this isn't optimal.
The same can be said for when Emacs doesn't indent correctly. But
that is a horrid, unacceptable situation in any case. I always fix
those as soon as I can.
For some reason, renaming the current buffer file is a multi-step process in Emacs.
(defunrename-current-buffer-file ()
"Renames current buffer and file it is visiting."
(interactive)
(let ((name (buffer-name))
(filename (buffer-file-name)))
(if (not (and filename (file-exists-p filename)))
(error"Buffer '%s' is not visiting a file!" name)
(let ((new-name (read-file-name "New name: " filename)))
(if (get-buffer new-name)
(error"A buffer named '%s' already exists!" new-name)
(rename-file filename new-name 1)
(rename-buffer new-name)
(set-visited-file-name new-name)
(set-buffer-modified-p nil)
(message "File '%s' successfully renamed to '%s'"
name (file-name-nondirectory new-name)))))))
(global-set-key (kbd "C-x C-r") 'rename-current-buffer-file)
This defun fixes that. And unlike some other alternatives to perform
this common task, you don't have to type the name out from scratch -
but get the current name to modify. Like it should be.
Luke is doing an interesting threefive-part tutorial on writing
a pastebin in PHP: PHP Like a Pro (2, 3,
4, 5). The tutorial is largely an introduction to
the set of tools a professional would use to accomplish a more
involved project, the most interesting of which, for me, is
Vagrant.
Because I have no intention of ever using PHP, I decided to
follow along in parallel with my own version. I used Emacs Lisp with
my simple-httpd package for the server. I really
like my servlet API so was a lot more fun than I expected it to be!
Here’s the source code,
It has syntax highlighting, paste expiration, and light version
control. The server side is as simple as possible, consisting of only
three servlets,
/pastebin/: static files
/pastebin/get: serves (immutable) pastes in JSON
/pastebin/post: accepts new pastes in JSON, returns the ID
A paste’s JSON is the raw paste content plus some metadata, including
post date, expiration date, language (highlighting), parent paste ID,
and title. That’s it! The server is just a database and static file
host. It performs no dynamic page generation. Instead, the client-side
JavaScript does all the work.
For you non-Emacs users, the repository has a pastebin-standalone.el
which can be used to launch a standalone instance of the pastebin
server, so long as you have Emacs on your computer. It will fetch any
needed dependencies automatically. See the header comment of this file
for instructions.
IDs
A paste ID is four or more randomly-generated numbers, letters, dashes
or underscores, with some minor restrictions (pastebin-id-valid-p).
It’s appended to the end of the servlet URL.
/pastebin/<id>
/pastebin/get/<id>
In the first case, the servlet entirely ignores the ID. Its job is
only to serve static files. In the second case the server looks up the
ID in the database and returns the paste JSON.
The client-side inspects the page’s URL to determine the ID currently
being viewed, if any. It performs an asynchronous request to
/pastebin/get/<id> to fetch the paste and insert the result, if
found, into the current page.
Form submission isn’t done the normal way. Instead, the submission is
intercepted by an event handler, which wraps the form data up in JSON
(much cleaner to parse!) and sends it asynchronously to
/pastebin/post via POST. This servlet inserts the paste in the
database and responds in text/plain with the paste ID it
generated. The client-side then redirects the browser to the paste URL
for that paste.
Features
As I said, the server performs no page generation, so syntax
highlighting is done in the client with
highlight.js. I could have used htmlize
and supported any language that Emacs supports. However, I wanted to
keep the server as simple as possible, and, more importantly, I
really don’t trust Emacs’ various modes to be secure in operating on
arbitrary data. That’s a huge attack surface and these modes were
written without security in mind (fairly reasonable). It’s actually a
deliberate feature for Emacs to automatically eval Elisp in comments
under certain circumstances.
Version control is accomplished by keeping track of which paste was
the parent of the paste being posted. When viewing a paste, the
content is also placed in a textarea for editing. Submitting this form
will create a new paste with the current paste as the parent. When
viewing a paste that has a parent, a “diff” option is provided to view
a diff patch of the current paste with its parent (see the screenshot
above). Again, the server is dead simple, so this patch is computed by
JavaScript after fetching the parent paste from the server.
Databases
As part of my fun I made a generic database API for the servlets, then
implemented three different database backends. I used eieio, Emacs
Lisp’s CLOS-like object system, to implement this API. Creating a new
database backend is just a matter of making a new class that
implements two specific methods.
The first, and default, implementation uses an Elisp hash table for
storage, which is lost when Emacs exits.
The second is a flat-file database. I estimate it should be able to
support at least 16 million different pastes gracefully. The on-disk
format for pastes is an s-expression. Basically, this is read by
Emacs, expiration date checked, converted to JSON, then served to the
client.
To my great surprise there is practically no support for programmatic
access to a SQL database from GNU Emacs Lisp (other Emacsen do). The
closest I found was pg.el, which is asynchronous by
necessity. However, the specific target I had in mind was SQLite.
I did manage to implement a third backend that uses SQLite, but it’s
a big hack. It invokes the sqlite3 command line program once for
every request, asking for a response in CSV — the only output format
that seems to escape unambiguously. This response then has to be
parsed, so long as it’s not too long to blow the regex stack.
This has been an educational project for me. As a tutorial and for
practice I’ll probably write the server again from scratch using other
languages and platforms (Node.js and Hunchentoot maybe?), keeping the
same front-end.
Maybe not when I program elisp, since that's sexp-based, but for
other programming languages these two come in very handy. They
simply move the current line one step up or down.
This adds W to toggle ignoring whitespace in magit.
It has some weird interactions with the changed files list, in that
files with nothing but whitespace changes go missing. Toggle back to
find them again.
;; Auto refresh buffers
(global-auto-revert-mode 1)
;; Also auto refresh dired, but be quiet about it
(setq global-auto-revert-non-file-buffers t)
(setq auto-revert-verbose nil)
Auto revert mode looks for changes to files, and updates them for you.
With these settings, dired buffers are also updated. The last
setting makes sure that you're not alerted every time this happens.
Which is every time you save something.
(global-set-key [remap goto-line] 'goto-line-with-feedback)
(defungoto-line-with-feedback ()
"Show line numbers temporarily, while prompting for the line number input"
(interactive)
(unwind-protect
(progn
(linum-mode 1)
(goto-line (read-number "Goto line: ")))
(linum-mode -1)))
I don't have line numbers visible in the fringe of my Emacs. If I
want to go to a line number, that is usually because it is
referenced in an error message somewhere. Showing them all the time
is just noise.
Still, many people want line numbers visible. I guess that is
because they use them for navigation. This snippet shows line
numbers temporarily just when you're going to a line number
with goto-line.
Notice the nice remap-trick in the key binding. It will
remap all key bindings from goto-line to
goto-line-with-feedback. Neat!
Webjump let's you quickly search Google, Wikipedia, Emacs Wiki and
other pages. I've got it bound to C-x g.
This snippet adds Urban Dictionary to the list of pages, so the next
time you wonder what those dastardly kids mean when they write
faceroll or sassafrassa or Technotard or kthxbye or whatever else is
hip these days, well, then you can find out. With webjump.
;; Settings for currently logged in user
(setq user-settings-dir
(concat user-emacs-directory "users/" user-login-name))
;; Conclude init by setting up specifics for the current user
(when (file-exists-p user-settings-dir)
(mapc 'load (directory-files user-settings-dir nil "^[^#].*el$")))
These are the last lines of my init.el. They will load
any *.el files in the
~/.emacs.d/users/user-login-name/
folder.
;; Elisp go-to-definition with M-. and back again with M-,
(autoload 'elisp-slime-nav-mode "elisp-slime-nav")
(add-hook 'emacs-lisp-mode-hook (lambda () (elisp-slime-nav-mode t)))
(eval-after-load 'elisp-slime-nav '(diminish 'elisp-slime-nav-mode))
Then you need to M-xpackage-install
elisp-slime-nav-mode.
It lets you jump to the definition of a function
with M-., and back again afterwards with M-,.
After a quick M-xpackage-install diminish, you too can
have the pleasure of using a lot of minor modes, without those minor
modes making a mess of the modeline. Mmm.
As for diminish.el itself, it contains the most beautifully poetic
code commentary of all time. Here's an excerpt:
"When we diminish a mode, we are saying we want it to continue doing its
work for us, but we no longer want to be reminded of it. It becomes a
night worker, like a janitor; it becomes an invisible man; it remains a
component, perhaps an important one, sometimes an indispensable one, of
the mechanism that maintains the day-people's world, but its place in
their thoughts is diminished, usually to nothing. As we grow old we
diminish more and more such thoughts, such people, usually to nothing."
Tired of navigating back to where you were last in a file?
;; Save point position between sessions
(require 'saveplace)
(setq-default save-place t)
(setq save-place-file (expand-file-name ".places" user-emacs-directory))
The saveplace package is part of Emacs, and remembers the
position of point - even between emacs sessions.
The last line sets the path to where saveplace stores your position
data. Change it at your peril! *
* Ahem, there really is no peril. That was just melodrama.
;; Write backup files to own directory
(setq backup-directory-alist
`(("." . ,(expand-file-name
(concat user-emacs-directory "backups")))))
;; Make backups of files, even when they're in version control
(setq vc-make-backup-files t)
Backup files are so very annoying, until the day they save your
hide. That's when you don't want to look back and say "Man, I
really shouldn't have disabled those stupid backups."
These settings move all backup files to a central location.
Bam! No longer annoying.
As an added bonus, that last line makes sure your files are backed
up even when the files are in version control. Do it.
Behold the very first lines in my .emacs.d/init.el:
;; Turn off mouse interface early in startup to avoid momentary display
(if (fboundp 'menu-bar-mode) (menu-bar-mode -1))
(if (fboundp 'tool-bar-mode) (tool-bar-mode -1))
(if (fboundp 'scroll-bar-mode) (scroll-bar-mode -1))
;; No splash screen please ... jeez
(setq inhibit-startup-message t)
They hide the menu bar, tool bar, scroll bar and splash screen.
Doing so early avoids ever having to see them - not even for a brief
flash when starting Emacs.
These four lines move us into the tranquil zone of nothing but
the text. A raster interface can never hold the seeming
infinitude of Emacs functionality, so we just let it go.
"What I don't understand is: why should you ever care how your editor
looks, unless you're trying to win a screenshot competition? The
primary factor in looking good should be the choice of a good font
at a comfortable size, and a syntax coloring theme that you like.
And that is not something specific to an editor. Editors like Emacs
and vi have almost no UI! If Emacs is configured right, the only UI
it has is the modeline and the minibuffer."
Inspired by Emacs Rocks! Episode 11 on
swank-js, I spent the last week writing a new extension to
Emacs to improve support for web development. It’s called
Skewer and it allows you to interact with a browser
like you would an inferior Lisp process. It’s written in pure Emacs
Lisp, operates as a servlet for my Elisp webserver,
and requires no special support from your browser or any other
external programs, making it portable and very easy to set up.
It works a little bit like impatient-mode. First,
the browser makes a long poll to Emacs. When you’re ready to send code
to the browser to evaluate, Emacs wraps the expression in a bit of
JSON and sends it to the browser. The browser responds with the result
and starts another long poll.
As such, the browser doesn’t need to do anything special to support
Skewer. If it can run jQuery, it can be skewered. I’ve tested it and
found it working successfully on the latest versions of all the major
browsers, including you-know-who.
To properly grab expressions around/before the point I’m using the
amazingjs2-mode, originally written by the famous Steve
Yegge. If you’re developing JavaScript you should be using this mode
anyway! I thought I was clever with my psl-mode, writing
my own full language parser. Steve Yegge did the same thing on a much
larger scale three years ago with js2-mode. It includes an entire
JavaScript 1.8 parser so the mode has full semantic understanding of
the language. For Skewer, I use js2-mode’s functions to access the AST
and extract complete, valid expressions.
What’s wrong with swank-js?
Skewer provides nearly the same functionality as swank-js, a
JavaScript back-end to SLIME. At a glance my extension seems redundant.
The problem with swank-js is the complicated setup. It requires a
cooperating Node.js server, a particular version of SLIME, and a lot
of patience. I could never get it working, and if I did I wouldn’t
want to have to do all that setup again on another computer. In
contrast, Skewer is just another Emacs package, no special setup
needed. Thanks to package.el installing and using it should be no
more difficult than installing any other package.
Most importantly, with Skewer I can capture the setup in my
.emacs.d repository where it will automatically
work across any operating system, so long as it has Emacs installed.
Getting into JavaScript
I already used Skewer to develop a little boids toy, which
I’m using to demonstrate the mode (the video). Unlike my previous
experiences in web development, this was extremely enjoyable —
probably because it felt a lot like I was writing Lisp. And unlike any
Lisp I’ve used so far, I had a canvas to draw on with my live
code. That’s a satisfying tool to have.
Due to those prior poor experiences, I had avoided web development for
a long time. But now that I have some decent tools configured I’m
going to get into it more. In fact, I’ve decided I’m completely done
with writing Java applets. Bounze will have been my last
one.
This has become a pattern for me. When I want to start using a new
language or platform I need to figure out a work-flow with Emacs. This
involves trying out new modes, reading about how other people do it,
and, ultimately, when I found out the existing stuff is inadequate I
build my own extensions to create the work-flow I desire. I did this
with Java, recently with psl-mode (which was to
be expected), and now web development.
In my recent proper introduction JavaScript in order to create and
demo Skewer mode idiomatically, perhaps the most exciting discovery
this past week was the JavaScript community itself. I’ve been mostly
unaware of this community and taking my first steps into it has been
enlightening.
JavaScript had a rough start. It was designed in a rush by developers
who, at the time, didn’t quite understand the consequences of their
design decisions, and later extended by similar people. The name of
the language itself is evidence of this. Fortunately some really smart
people jumped on board along the way (including Guy Steele of Lisp
fame) and have tried to undo, or at least mitigate, the mistakes.
Due to the coarseness of the language, the JavaScript community is
actually a lot like the Elisp community, but on a larger scale:
there’s still a whole lot of frontier to explore and it’s pretty easy
to make a noticeable splash.
I watched this presentation last night about introducing Clojure in
the workplace, Bootstrapping Clojure. Most of the video is
Tyler presenting code he wrote, so there’s a lot of Clojure code
displayed in the presentation. The way he presented the code itself
was interesting: indentation was highlighted in alternating shades of
gray.
Such emphasis on indentation could be useful specifically for reading
Lisp, because, for humans, what makes s-expressions readable isn’t the
parenthesis but the indentation. That’s why us Lispers shake our heads
when non-Lispers complain that there are too many parenthesis. We
don’t notice them!
He’s a Vim user so I assume this is some Vim extension, but maybe it’s
just the output from a particular pretty printer. I thought it would
be interesting to have this feature in Emacs, so I created a minor
mode for it.
Here’s what it looks like with the default Emacs theme.
And some Java with visual-indentation-width set to 4,
Dark themes (like the Wombat theme I personally use) will work as
well, with the indentation highlighting appearing darker.
It can be enabled by default in all programming modes easily,
(add-hook'prog-mode-hook'visual-indentation-mode)
It completely falls apart when tabs are used for indentation, which
Emacs will use by default. My configuration
forbids tabs (tabs are stupid, people!) but I still need to edit other
people’s code containing tabs. I don’t think there’s a way to apply
highlighting to part of a tab, so I’m not sure if there’s a way to
fix that. Because I don’t intend to actually use the mode regularly,
it’s just a proof of concept, and fixing it would be non-trivial, I
don’t intend to fix it.
Update: This bug was fixed in Emacs 24.4 (released October 2014).
A few months ago I filed a bug report for Emacs
(upstream) when I stumbled across Emacs aborting under very
specific circumstances. I was editing in markdown-mode and a
regular expression replacement on lists would reliably, and
frustratingly, cause Emacs to crash.
Through a sort-of binary search I only loaded only half of
markdown-mode to see in which half it would trigger, then I cut that
half in half again and repeated recursively until I had it down to a
small expression that causes a --no-init-file (-q) Emacs to
abort. It almost looks like I found it through fuzz testing. Change or
remove anything even slightly and it no longer triggers the abort.
To trigger it, there’s an after-change-functions hook that performs
a regular expression search immediately after a replace-regexp. A
peek at the backtrace with gdb shows that this somehow causes the
point to leave the bounds of the buffer. Emacs detects this as an
assertion before dereferencing anything, and it aborts, thus
preventing a buffer overflow vulnerability. This is important for
my Emacs web server because if there’s a way to
trigger this bug in the web server I’d much rather have it abort than
run arbitrary shellcode injected in by a malicious HTTP request.
My bug report has seen no activity since I posted it. I can understand
why. The circumstances to trigger it are unlikely and it’s a very old
bug, so it’s low priority. It’s also a huge pain to debug. Hacking on
Emacs from Lisp is pleasant but hacking on Emacs from C is not. The
bug likely sits in the bowels of the complicated regular expression
engine, making it even more unpleasant. I personally have no interest
in trying to fix it myself.
So, since it looks like it’s here for the long haul it’s kind of fun
to implement an abort function on top of it, allowing Elisp programs
to terminate Emacs abnormally — you know, in case kill-emacs isn’t
fun enough.
(defunabort()"Ask Emacs to abnormally terminate itself (bug#12077)."(interactive)(with-temp-buffer(insert"#\n*\n")(goto-char(point-min))(add-hook'after-change-functions(lambda(abc)(re-search-forward"")))(replace-regexp"^\\*"" *")))
It’s interactive so you could even bind a key to it.
This evening I came across an interesting idea:
using system programs as functions. The original idea goes to
sh, a Python module that
exposes system programs as functions. There’s also a Clojure library
called shake to do the same thing in Clojure.
Thanks to symbols, I think the idea maps especially well onto Lisp
because arguments don’t need to be provided as strings. Here are some
examples,
Evaluating the above will install macros for all programs in your
PATH, except where you already have functions or macros defined. I
messed up on the latter point while writing this and broke Emacs
enough to require a restart. The system program is called
synchronously and the output is returned as a string.
However, because arguments aren’t evaluated (macros) this has
limited usefulness. These function calls are static and can’t be
passed variable arguments. In order to do that arguments would need to
be evaluated and symbols would need to be quoted. For example,
So I think I’d prefer this interface to the one provided by Clojure’s
shake (and my Elisp code at the top). I have little need to call
programs with static arguments.
The first part of the class is oriented around a small programming
language created just for this class called ParselTongue. It
looks like this:
deffun evenp(x)
if ==(x, 0) then
true
else if ==(x, 1) then
false
else evenp(-(x, 2))
in defvar x = 14 in {
while (evenp(x)) { x--; }; # Make sure x odd
print("This is an odd number: ");
print(x);
""; # No output
}
I’ve gotten so used to having a solid Emacs major mode when coding
that I can’t stand writing code without the support of a major
mode. Since this language was invented recently just for this class
there was no mode for it, nor would there be unless someone stepped up
to make one. I ended up taking that role. It was an opportunity to
learn how to create a major mode, something I had never done before.
At first it was just some syntax highlighting (very easy) and some
poor automatic indentation. The indentation function would get
confused by anything non-trivial. It’s actually really hard to get
it right. I’ve grown a much better appreciation for automatic
indentation in other modes.
In an attempt to improve this I decided I would try to fully parse the
language and use the resulting parse tree to determine indentation —
something like the depth of the pointer in the
tree. My experience with Perl’s Parse::RecDescent
some years ago was very positive and I wanted to reproduce that
effect. However, rather than write the grammar in a separate language
that mixes in the programming language, which I find extremely messy,
instead I wanted to use pure s-expressions. A grammar looks very nice
as an alist of symbols.
Arithmetic Parser
For example, here’s a grammar for simple arithmetic expressions,
including operator precedence and grouping (i.e. “4 + 5 * 2.5”,
“(4 + 5) * 2.5”, etc.).
Strings are regular expressions , the only thing to actually match
input text (terminals). Lists are sequences, where each element in
the list must match in order. Vectors (in brackets) are choices
where one of the elements must match. Symbols name an expression so
that it can be referred to by other expression recursively.
Give this alist to the parser and it will return an s-expression of
the parse tree of the current buffer. Due to the way the grammar must
be written this parse tree isn’t really pleasant to handle
directly. For example, a series of multiplications (“1 * 2 * 3 * 4”)
wouldn’t parse to a nice flat list but with further depth for each
additional operand.
To help squash these, the parser will accept an alist of symbols and
functions which process the parse tree at parse time. For example,
these corresponding functions will make sure "4 * 5 * 6" gets parsed
into (* 4 (* 5 (* 6 1))).
Notice how normal Emacs functions could be supplied directly in most
cases! That makes this approach so elegant in my opinion.
Also, in arith-op note the use of destructuring-bind. I’ve found
that macro to be invaluable when writing these syntax tree functions.
In this case, we can be even more clever. Rather than build a nice
parse tree, the expression can be evaluated directly. All it takes is
one small change,
I discovered this useful side effect while making my ParselTongue
parser. The original intention was that I’d parse the buffer for use
in indentation, then maybe I’d create an interpreter to evaluate the
parser output. However, the resulting parse tree was looking a lot
like Elisp. In an epiphany I realized I could simply emit valid Elisp
directly and forgo writing the interpreter altogether. And so I
accidentally created a ParselTongue compiler! This was incredibly
exciting for me to realize.
Because it compiles to such a high level language, and because
ParselTongue is very Lisp-like semantically, it’s a bit
unconventional: the compiler emits code during parsing. In fact,
when the parser backtracks, some emitted code is thrown away.
By the end of the first evening I had implemented the majority of the
compiler, which quickly took precedence over indentation. The compiler
is now integrated as part of psl-mode. The current buffer can be
evaluated at any time with psl-eval-buffer. This function compiles
the buffer and has Emacs eval the result, printing the output in the
minibuffer. Compiler output can be viewed with
psl-show-elisp-compilation (mostly for my own debugging).
After a few days I integrated indentation with parsing, which required
modifying the parser (changes included in rdp itself). The parser
needed to keep track of where the point is in the parse tree. For
indentation it basically counts the depth into the parse tree, plus a
few more checks for special cases.
The parser was intentionally isolated from the rest of psl-mode so
that it could be separated for general use, which I have now
done. It’s been a really handy general purpose tool since then. That
arithmetic parser is only 35 lines of code and took about half-an-hour
to create.
Future Directions
I also wrote a bencode parser — only the
bencode-tokens and bencode-funcs alists are needed to parse
bencode, about 30 LOC. Careful observation will reveal that I cheated
and the result is a little hackish. Due to the way strings work,
bencode is not context-free so it can’t be parsed purely by the
grammar. I can work around it by having the parse tree function for
strings consume input, since it’s called during parsing.
I’ll be using rdp to parse many more things in the future, I’m
sure. It’s much more powerful than I expected.
Taking advantage of Emacs’ image-mode and the handy
Netpbm format it’s
possible to generate and render images inside Emacs using
Elisp. This function will generate a
Sierpinski carpet
and display the result in a buffer.
(defuncolormap(v)"Given a value between 0 and 1.0, insert a P6 color."(dotimes(i3)(insert-char(floor(*256(min0.99(sqrt(*(-3i)v)))))1)))
One of the project ideas on my mental back-burner of things I’ll never
get to is to create a little graphics library for Elisp. It would use
a technique like this to pull it off. Assuming support was compiled
in, Emacs can even render SVGs to a buffer, so creating a rich
graphics library wouldn’t be difficult at all. Plus, unlike bare
Elisp, it would be fast.
You may have been confused by
yesterday’s nonsense post. That’s because it was
generated by a few
Elisp Markov chain functions. It
was fed my entire blog and used to generate a ~1500 word post. I
tidied up a bit to make sure the markup was valid and parenthesis were
balanced, but that’s about it.
The algorithm is really simple and I was quite surprised by the
quality of the output. After feeding it Great Expectations and A
Princess of Mars (easily obtainable from
Project Gutenberg) I had a good laugh at
some of the output. Some choice quotes,
He wiped himself again, as if he didn’t marry her by hand.
I admit having done so, and the summer afternoon toned down into the
house.
My favorite of yesterday’s post was this one,
Suppose you want to read a great story, I recommend it.
The output also looks like some types of spam, so this may be how some
spammers generate content in order to get around spam filters.
To build a Markov chain from input, the program looks at
markov-text-state-size words (default 3) and makes note of what word
follows. Then it slides the window forward one word and repeats. To
generate text, the last markov-text-state-size words outputted is
the state and the next word is selected from these notes at random,
weighted by the frequency of its appearance in the input text. Smaller
state sizes generates more random output and larger state sizes
generates better structured output. Too large and the output is the
input verbatim.
For example, given this sentence and a state size of two words,
Because there are two options for (“he” “ran”), the generator might
loop around that state for awhile like so,
Quickly, he ran and he ran and he ran and he ran until he couldn’t.
Or it might skip the section altogether,
Quickly, he ran until he couldn’t.
Also notice that the punctuation is part of the word. This makes the
output more natural, automatically forming sentences. More so, my
program also holds onto all newlines. This breaks the output into nice
paragraphs without any extra effort. Since I wrote it in Elisp, I use
fill-paragraph to properly wrap the paragraphs as I generate them,
so superfluous single newlines don’t hurt anything.
One problem I did run into with my input text was quotes. I was using
novels so there is a lot of quoted text (character dialog). The
generated text tends to balance quotes poorly. My solution for the
moment is to strip these out along with spaces when forming
words. That’s still not ideal.
I’m going to play with this a bit more, using it as a tool for other
project ideas (ERC bot, etc.). I already did this by including a
lorem ipsum generator
alongside the markov-text package. The input text is Cicero’s De
finibus bonorum et malorum, the original source of lorem
ipsum. This was actually the original inspiration for this project,
after I saw lorem-ipsum.el on EmacsWiki and decided I could do
better.
Update: This post shouldn’t make sense to anyone
(hopefully). Read the follow-up for an
explanation.
When a branch of my posts remains simple.
This is necessary when one will assume Alan is more important than
number 12. By using numbers to repeat them, but this won’t work with
any sort of thing you want to load what’s needed. This includes
reimplementing the reader as it seems you still need to specify any
video-specific parameters, ppmtoy4m is the whole thing is just that,
decorated with some tips on how the current space as visited, then
recurse from the client to read a great story, I recommend you use to
launch a daemon process and prints the variable information to
stdout. As an added bonus, when a second variable for accumulation and
a second argument is relevant.
Suppose you want to read a great story, I recommend it.
This servlet uses the Term::ProgressBar, if it’s any good, but it’s
funny. As anyone with cats knows, it’s not too stupid to call
fsync() to force the write to the snapshot and uninterns any new
symbols. These symbols will be added to the the second experiment.
At this line, you can perform a number from a couple of these and give
them back any other language that can turn out even from a large
header comment in the logs, so getting someone into my honeypot
wouldn’t take long at all. The only proof I could then
cherry-pick/pull the issues from that repository and see the
polynomial interpolation at that time, presented in order. This makes
so much of web development (I think that’s his name). I am an Emacs
person myself, which I use branches all the time, now that they can be
written.
We will run your build system in a web front-end to it, and made a
couple of seconds.
You should also be a good head start, though. The SPARC is big-endian
and the results to seed their program accordingly. You could do this
is by mounting the compromised filesystem in a list. In the
decentralized model, everyone has their own solutions in parallel when
it comes across 10 it emits 0.
Here’s an example of some of the fire gem activated and exploded,
causing no blindness to me. They take a look at the same level as the
printed string. You can grab my source code in response to abuse by
spammers who hide fraudulent URLs behind shortened ones. If these
services ever went down all at once, these shortened URLs would rot,
destroying many of the image, with the FFI.
Because I wrote a shell script that will also remove the execs and
live with nested shells because the zeros cancel out everything else?
Here is the protocol.
Generate a 10-byte random IV. This need not implement this.
Note that the shell script, and the arcfour key scheduler at least n
days.
However, generating a series of commits to all other encounters
nothing changes.
Your program should simulate this by having the user to reseed
somewhere. There’s no direct way to install it to dominate for
awhile. It is strange that Matlab itself doesn’t have any sort of
syntax highlighting. Boring! I finally ran into this image. After each
paste, make a saving throw to prevent an explosion.
Because Gnohkk would also suffer from the bottom are arranged around
the cats in the logs, so getting someone into my honeypot wouldn’t
take long at the link in the block. Another was going to used a
stationary magnet.
Our team went with this array (and replaced the current layer 5). Now,
duplicate the work was done just once by freeing the entire number, it
can perform both compression and decompression on both sides don’t pay
attention to the development loop is just an ordered list of 50 H’s
and T’s. If you implement this in the same time. This is along the
way, clone my repository right into the official website so I had to
do this for any long-blocking function that I use ppmtoy4m to pipe
the new frames to keep, such as n^p mod M, which this will handle
efficiently. For example, to add a new compression algorithm in terms
of brute-force attacks it requires using numbers long enough to fit
three Emacs’ windows side-by-side at 78 columns each. The leftmost one
contains my active work buffer where I do most useful things, a fresh
array every time it sees a free musical. Unfortunately, my writing
skills are even worse. I have gotten good mileage out of a file based
on their website demonstrating how to increment the iterator. I have
to type a negative comment about zip archives and moved on. I am
using a constant amount of memory.
It turns out that everyone is free to share his source code samples,
particularly more recent entries, was that producing the relief
surface was an e-mail address, I get home from work I don’t recommend
doing this with secret Java applets.
There are a few weeks since I last used KOffice, so I could easily
plug it into Emacs and run the test above, I would rather not do
damage, but rather a patient human being. Getting tired of manually
synchronizing them. It was finally time to document the effort as a
single mine is destroyed, the neighboring mines will replicate a
replacement. The minefield itself could therefore hold no secrets
whatsoever. This leaves out any possibility of a rumor among a group
of people. At any given time, each person in the background. My shell
habits looked like the ones you’re seeing after end-package.
It’s really simple way to detect edges all over the weekend I came up
with some rough edges. So I got it right while IE, Opera, Safari, and
Chrome all do it again.
Numbers can be found inside the fake closure provided by
lexical-let. In a previous post about Lua, another about a third of my
name generation code.
S-expressions are handy anywhere.
Two months ago I was so happy when I run the program with the proper
Perl regular expression contains quotes and these will not be worth
it.
I can’t help but think that a knight moving according to the current
symbol table to the existing mountain of elisp code out there,
requiring a massive increase in speed when using OpenCL. In fact,
there is virtually no computation involved. So what I want to look
like SBCL. Fortunately, that’s not all!!! There is a fake service or
computer on a chess board such that it’s somewhat easier to tell when
the handler can present any contents it wants. In this case, rather
than just one, even though I don’t know what it looks good, except you
want to italicize a few bits smaller than a minute. All the other day
I will probably be ordered by their own directory. Modern applications
have moved into a directory under ~/.config/. Your script needs to
be broken into small computation units, because Emacs lacked network
functionality until recently was the package manager, package, and
the Emacs Lisp Package Archive.
One of the info field in the list, which sounds like a .emacs file in
your program. If the slot is already taken, the symbol was in an
external system.
After all this, I thought I’d give it a YouTube URL and a single
password if the required artifacts, digitally signs them, and bundles
them up.
The demo at the same length as the variable declarations are exactly
the right magical string of, say, 31 fractions.
The story is really happening. Optimizing away variables that point to
it.
Oh, and I was just a tiny subset of the memory at once became a lot of
memory. For example, here’s my laptop’s /bin/ls, very roughly
labeled.
The different segments of the game area was a mistake on my rolls and
had some wires, connected to some sort of bad things this may happen
subconsciously, which is given in ImageMagick’s montage tool, which
made the final montage out of the image functions described below.
You can write a lexer or tokenizer without one. Because of this tool,
Samuel Stoddard, gives some in-game context to the light of day. I
just use your own program, the script in your load-path somewhere.
I’ve frequently thought that a Lisp-based shell would be produced by
first individually gzipping each file in first.
For a long ways away from a simple double-click shortcut. If you just
want to duplicate the remaining canines. Her reward for victory was a
very similar process, but without any sort of thing is
transparent. I’ve already used it with a degree in, say, a few
months. I’ve used POSIX threads, Pthreads, before, so it suits my
needs for the first two arguments from filter2, as well as some more
to see my changes, but I don’t know much about it, user AJR spoiled it
with ssh-add and it queries for your passphrase, storing it in two
obarrays at once, these shortened URLs would rot, destroying many of
its input. For example, this is what registration looks like,
Unfortunately, the HTML output is a Harsh Mistress. If you know that
the opposite way that the adventures and characters are riddled with
mistakes and very unbalanced. For an easier way to set up properly in
your configuration.
I strongly recommend that you generally want to have a master pad, K,
that you often generate very improbable series of commits.
To all other encounters nothing changes.
And that’s it! I put this line in your program. If you are subscribed
to the rescue!
After settling in with MELPA I wanted to see
about into turning my Emacs web server into an
installable package. Someone had already uploaded my code to
Marmalade after taking credit for all
the work and slapping the GPL on it (my version is public domain). So,
due to that and because the name httpd.el is already overloaded as
it is, I renamed it to simple-httpd. That’s the name of the package
in MELPA.
I did more than rename the package; it got an overhaul. I rewrote a
few functions, tossed a whole bunch of functions, created
a test suite, and finally added directory
listings — a feature that had long been on the TODO list. To keep
with the name “simple”, I ripped out the
clunky servlet system (sorry Chunye). This new
version was leaner, cleaner, and more useful.
I’ve definitely improved my software development skill over the last
three years since I originally wrote it. In my refactor I made it
buffer oriented. When a request comes in, the server fills a buffer
with the response and sends it back. This means I could send a
Content-Length header and use keep-alive to serve multiple requests
over one connection. It also suggested a new servlet paradigm — the
servlet prepares a buffer and the server sends it to the client.
Servlets
So I ended up adding servlet support again, from scratch. This time
it’s really easy to use. Here’s a “Hello, World” servlet,
The “function name” part is the path to the servlet. This one would be
found at /hello-world. The second is the MIME type as a
symbol. We’re just sending plain text in this example. The third is
the argument list. A servlet takes up to three arguments: the path,
the query alist, and the full request object (which includes the first
two). Unless a more specific servlet is defined, this servlet handles
everything under its root. In this case /hello-world, including
/hello-world/foo and /hello-world/foo/bar.txt. This is why the
path argument is relevant.
In the background I continue to use Chunye’s symbol dispatch
technique, so all servlets are actually functions that begin with
httpd/ (http/hello-world and httpd/hello). For a more advanced
servlet, the function can be written directly. There’s another macro,
with-httpd-buffer to help keep this simple. The server will always
pass four arguments (the three servlet arguments plus one more), so
when creating the function directly it needs to accept at least four
arguments.
The proc object here is the network connection process, providing
more exclusive access to the client. This allows the servlet to do
more interesting things like respond in the future (long polls). The
with-httpd-buffer macro creates a temporary buffer and, when the
body completes, sends an HTTP header and the buffer as the content,
similar to defservlet.
With access to the process, the servlet can do more specialized things
like send custom headers with httpd-send-header, send files with
httpd-send-file, send an error page with httpd-error, or do
redirects with httpd-redirect. The file server part of the server is
actually just another a servlet as well: httpd/. This could be
redefined to redirect the browser to our example servlet (HTTP 301).
I showed this to Brian, like I do everything, and
he found my servlet concept to be compelling, especially the
buffer-serving servlet. I believe his exact words were, “That’s so
simple.” He found it interesting enough that
he wrote a mode based on it called impatient-mode!
It serves a buffer’s content live to the web browser, including syntax
highlighting (via htmlize). Updates to the buffer are communicated by
a long-poll. The browser initiates a request in the background for an
update. Emacs adds the request to a list. A hook in
after-change-functions updates all the browsers waiting for an
update.
Enabling impatient-mode, a minor mode, publishes the buffer. If the
server’s running, the list of published buffers can be found under
/imp —
i.e. http://localhost:8080/imp. The
buffer can be accessed directly at /imp/live/<buffer-name>, which is
where /imp will link.
Perhaps the coolest thing is serving an HTML buffer without
htmlize. That is, send the raw buffer as text/html. Brian has a demo
of this in the linked post. You can tweak CSS and HTML and watch it
update live in the browser as you edit. It’s a really neat way to edit
CSS, since it’s often unintuitive (at least for me).
impatient-mode can also be installed through MELPA.
Emacs 24 comes with a unit testing library, ERT (Emacs Lisp
Regression Testing). I learned about it after watching
Extending Emacs Rocks! and I’ve been using
it ever since. It’s been a pleasant experience; enough so that
I made a key binding for it
so that I can effortlessly run tests at any time. When I recently made
a major overhaul to my Emacs web server I added
a small test suite
using ERT.
Emacs also comes with the ERT manual so it’s easy to start learning,
but here’s the gist of it. There are essentially two macros to worry
about: ert-deftest and should. The first is used to create tests
and the second behaves like assert but with nicer behavior. Here’s
an example,
(ert-deftestexample-test()(should(=(+92)11)))
ert-deftest is what you’d expect from every other def*. The empty
parameter list does nothing at the moment other than to make it feel
like writing a defun. The body is evaluated as normal. This is all
turned into an anonymous function which is stuffed in the plist of
the symbol example-test. When it comes time to running tests, they
are found by searching the plists of every interned symbol.
The other macro, should, takes one argument: a form that should
evaluate to true. There is also a should-not and a should-error,
which do what you would expect.
Tests are run with M-x ert. It will ask for a test selector, where
t selects all defined tests. There are many ways to select a subset
of all tests (:new, :passed, :failed, etc.) but I usually just
run all of them (as my key binding makes obvious). The results are
displayed in a separate pop-up buffer which, as usual, can be
dismissed with q.
Running ERT
What makes should special is error reporting. When tests fail you
will be provided with the forms that failed and their return
values. For example, if we modify the test above to fail.
(ert-deftestexample-test()(should(=(+92)100)))
Then run the test and it will note the failure. There is also some red
coloring not captured here.
Displayed are the forms we were comparing — (+ 9 2) and 100 — and
what they evaluated to: (= 11 100). If I put the point at the test
result and type . it will take me to the test definition so that I
can start looking further. Or I can press b to see a backtrace, m
to see all output messages from that test, or, if I’m in disbelief,
r to rerun that test.
Mocking
Elisp’s dynamic bindings really come in handy when functions need to
be mocked. For example, say I have a function that, at some point,
needs to check whether or not a particular file exists. This would be
done using file-exists-p. Creating or removing the file in the
filesystem before the test isn’t a well-contained unit test. Tests
running in parallel could interfere and there are a number of ways
something could go wrong.
Instead I’ll temporarily override the definition of file-exists-p
with a mock function using let’s cousin, flet. Note that
file-exists-p is a C source function but I can still override it as
if it was any regular lisp function.
This is a very simple mock. For a real unit test I might want the mock
to return t for some filename patterns and nil for others. There’s
an extension to ERT, el-mock.el, which assists in creating more
complex mocks, but I haven’t used or needed it yet.
Since it’s so convenient I’m going to be using ERT more and more until
it becomes second-nature.
Update June 2017: I no longer use Emacs’ package.el and instead
manage packages and their dependencies (manually) through my own
decentralized package system called gpkg (“git package”).
For those who are unaware, Emacs 24 was finally released this past
June. I had been following the official repository for about a year
before the release using what was becoming version 24, very quickly
becoming dependent on several of the new features. Now that
it’s been officially released I’m back to using a stable version of
Emacs, about which I’m quite relieved.
One of the new features that I hadn’t been using until recently was
the package manager, package, and the
Emacs Lisp Package Archive (ELPA). You can now
ask Emacs to download and install new modes and extensions from the
Internet. By default, it only uses the official archive. It only hosts
packages with copyright assigned to the FSF — quite
restrictive. There are alternatives, the most popular of which is
Marmalade. Fortunately it’s easy to ask
package to use additional repositories, so this is a non-issue.
Because it was still unstable and buggy at the time, I avoided using
it when setting up my configuration repository.
Instead I opted to gather packages by way of Git submodules. I’d give
package a shot once Emacs 24 was released. Once it was released in
June it was just a matter of time until I invested into this new
system.
The trigger was an e-mail from one of my readers, Rolando. He asked me
if I could move my recently updated memoization
function into its own repository and touch it up so that it could be
turned into a package with MELPA, another
alternative package repository. This forced me to finally investigate.
It turns out MELPA is really interesting. Each package is described
by a “recipe” file, which is essentially just a tiny s-expression
listing the repository URL. In the case of my memoization package,
From a package maintainer’s point-of-view, this is fantastic. I don’t
have to take any extra steps to publish updates to my package. I just
keep doing what I do and it happens automatically. However, I need to
be more careful about not pushing broken commits — which is why I
started unit testing (to be
covered in a future post). And I need to be extra
careful with my SSH keys, since they’re now used to publish code that
other people automatically trust and execute.
Excited about MELPA and wanting to actually use my own package, I
started throwing out my submodules, replacing them with their package
equivalents. If you follow my configuration repository you probably
noticed all the recent disruption, because updating requires manual
intervention. Git leaves submodules around (for good reason!) so they
need to be manually removed.
I also heavily updated and renamed my web server
(now called simple-httpd) to provide it as a package (also to be
covered in a future post). Thanks to MELPA, I follow the package
rather than my own repository since it follows so closely (< 1 hour).
Another barrier was that I was using an old version of Magit
due to a bad interaction of modern versions with Wombat, my preferred
color theme. After some face tweaking, I not only fixed it
but I made it better than it was before. Sinking a an hour or two into
these sorts of annoyances usually works out really well. I need to
remind myself of this in the future when I run into annoyance issues.
Surprisingly, package doesn’t seem to be written with managed
configuration in mind. The provided functionally is designed to be
used interactively rather than programmatically. package-install is
only meant to be invoked once, so care needs to be taken in listing
packages in a configuration and doing everything in the right
order. Here’s how I have it set up at the moment, after after listing
the packages to use in my-packages,
Upgrading/updating is currently a manual process. Run
package-refresh-contents, list the packages with list-packages,
type U to mark updates, then x to execute the upgrade. Sometime
I may work that into my configuration to be done automatically
once-per-week or something.
I really look forward to making more use of the package manager,
especially as packages can more easily become interdependent, reducing
duplication of effort.
I just updated my Elisp memoization function so
that it’s no longer a dirty hack. To work around the lack of closures,
due to the lack of lexical scope in Elisp, the original version used
uninterned symbols to store the look-up table. The new version in the
post uses lexical-let, which does the same thing internally to fake
a closure. The new version in my dotfiles
repository uses the brand new
Emacs 24 lexical scoping.
It was “dirty” because it built a lambda function out of a list at run
time, taking advantage of the way Elisp currently handles
functions. The reason for this was that I wanted to inject the
original documentation string into the new function which can’t
normally be done when lambda is used the correct way. When I updated
the function I fixed this as well. It uses a trick provided by Elisp,
which is different than the Common Lisp way that I assumed.
Both Elisp and Common Lisp have a documentation function for
programmatically accessing symbol documentation. The Elisp version
only provides function documentation, so it only accepts one
argument.
(defunfoo()"Foo."nil)(documentation'foo)=>"Foo."
The Common Lisp version must be told what type of documentation to
return, such as function or variable (defvar, defconst).
(documentation'foo'function)=>"Foo."
As it might be expected, this is setf-able! It’s possible to update
or modify documentation strings without needing to redefine the
function.
Unfortunately it’s not setf-able in Elisp. Instead you can set the
function-documentationproperty of the symbol. The documentation
function will prefer this over the string stored in the function
itself.
One of the users of my Emacs java extensions
e-mailed me with a question/suggestion about viewing .class files in
Emacs. Emacs has automatic compression, encryption, and archive modes
which allow certain non-text files to be viewed within Emacs in a
sensible text form. He wanted to do the same with Java byte-compiled
.class files: when opening a .class file, Emacs should automatically
and transparently decompile the bytecode into Java source.
He mentioned
[JAD](http://en.wikipedia.org/wiki/JAD_(JAva_Decompiler%29)
specifically, a popular, proprietary, but unmaintained and outdated
Java bytecode decompiler. I’ve never used it and honestly I see no
reason to start using it. Unfortunately there are no other decompilers
in the Debian package archives and I know nothing else about Java
decompiling, so this left me kind of stuck. Instead I decided to build
a proof-of-concept using javap, the Java disassembler, which comes
with JDKs.
Here it is: javap-handler.el. With
these forms evaluated, try opening a .class file in Emacs. Rather than
a screen full of junk, you’ll (hopefully) be presented with a
read-only buffer containing detailed information about the class.
(add-to-list'file-name-handler-alist'("\\.class$".javap-handler))(defunjavap-handler(op&restargs)"Handle .class files by putting the output of javap in the buffer."(cond((eqop'get-file-buffer)(let((file(carargs)))(with-current-buffer(create-file-bufferfile)(call-process"javap"nil(current-buffer)nil"-verbose""-classpath"(file-name-directoryfile)(file-name-sans-extension(file-name-nondirectoryfile)))(setqbuffer-file-namefile)(setqbuffer-read-onlyt)(set-buffer-modified-pnil)(goto-char(point-min))(java-mode)(current-buffer))))((javap-handler-realopargs))))(defunjavap-handler-real(operationargs)"Run the real handler without the javap handler installed."(let((inhibit-file-name-handlers(cons'javap-handler(and(eqinhibit-file-name-operationoperation)inhibit-file-name-handlers)))(inhibit-file-name-operationoperation))(applyoperationargs)))
This was harder to do than I thought it would be. To make a new
“magic” file mode requires the use of a half-documented, hackish
file-name-handler API. There’s
a page on it
in the GNU Emacs Lisp Reference Manual but I mostly figured it out
by reading the source code around auto-compression-mode and
auto-encryption-mode.
It works by installing a handler function in file-name-handler-alist
— similar to auto-mode-alist. The handler has complete control over
how a particularly-named class of files is handled by Emacs. For
example, the most useful part is instead of actually providing the
contents of a file, the handler can present any contents it wants. In
this case, rather than read in the actual bytecode, the handler
executes javap on the file and uses the output for the buffer
content.
The hackish part is when the handler wants to let Emacs handle an
operation the normal way, which is pretty much every case except for
get-file-buffer. The handler has to disable itself by temporarily
setting a dynamically-scoped variable (one of the many legacy areas
that prevents Emacs from being lexically-scoped by default), then ask
Emacs to try the operation again.
As I said, this is just a proof-of-concept so there are two issues
remaining. The first was something requested specifically: viewing
.class files inside .jar archives. It could do this if it was just a
little bit smarter about the classpath. I leave that as an exercise to
the reader. :-)
The second is finding a well-behaved, reasonable decompiler (GUI-less
Unix filter) and replacing javap with it. Given that assumption,
this should be as simple as replacing a couple of strings in the
call-process.
This is interesting enough that, if I were to fix it up for
correctness sometime, I may include it as part of java-mode-plus
someday.
At work, this has been The Year of Presentations for me so far. I’ve
prepared and performed three hour-long presentations so far this year,
and I will continue to do more. The presentations I’ve done before
haven’t been too serious; I’d just slap a few slides together in
whatever was handy and talk in front of them. However, with these more
serious presentations, I was making much more use of the associated
software. I haven’t been happy with any of them. They violate my
preference for precision, after all.
The first one I went with KPresenter, part of KOffice. It had been
years since I last used KOffice, so I thought I’d give it a shot. One
the good side, I liked the templates. However, it crashed on me a lot,
which was very frustrating. The GUI is lacking in a lot of places. For
example, I wanted to re-arrange my slides, and dragging and dropping
them feels like the natural choice. The mouse cursor even suggests it
by switching to a hand icon. Nope, dragging and dropping does
nothing. Overall, it felt like using a crummy version of Inkscape. The
presentation was a mess when viewed by other presentation software, so
I had to export it to a PDF to use it.
For the second one, I used LibreOffice’s Impress. It’s better than
KPresenter, but it still feels clunky. It took some wrestling to get
it to do what I wanted. As to be expected, I still had the same
feeling of uneasiness I have about any WYSIWYG tool.
For the third one I used PowerPoint, as provided by my employer. The
main reason for this was that I was stealing borrowing some
important slides from a couple of other people’s presentations, so I
had little choice. It was also an opportunity to compare it to the
others. Overall I’d say it’s on the same level as Impress, with some
slightly nicer GUI behavior.
Fortunately, I recently discovered what may become my preferred
presentation tool! It’s deck.js.
With deck.js, I’ll be writing my presentations in HTML 5, something
with which I’m already comfortable and experienced. Most importantly,
I’ll be able to create my presentations with Emacs and version them
with Git. That allows for easy collaboration on presentations
without all the stupid e-mailing documents back and forth — though
the other person would need to be comfortable with using deck.js,
too. That leaves … well, just Brian I
guess. So, in theory, this could make collaboration easier.
The downside to deck.js is that it requires a lot of boilerplate,
especially if you want to use the extensions, a couple of which are
absolutely essential in my opinion. Creating a new presentation
requires going through this setup phase, and then working around all
the boilerplate the rest of the time. I’ve successfully used Git to
work around this problem with Java, so I’ve done
the same here, with a little bit of help from
Jekyll.
What I’ve done is used Jekyll as a default layout for deck.js. It
hides away all of the deck.js boilerplate so that I can focus on my
presentation. It also makes it trivial to start a new
presentation. All I have to do is clone this repository and I’m ready
to go.
Jekyll almost opens up the opportunity to really take deck.js to the
next level: presentations written in Markdown! That would be
wonderful. Unfortunately, the HTML output is a little bit too
demanding for Jekyll (i.e. Maruku) to manage. It’s not quite
extensible enough to pull it off. So it’s just HTML5 for now, which is
unfortunately bulky when it comes to lists — a common element of
presentations. Oh well. I do still get syntax highlighting with
Pygments!
I haven’t used it for anything serious yet, so it’s still untried. In
my experimentation I found it enjoyable to work with, so I really look
forward to making use of it in the future. Feel free to use it
yourself, of course, and tell me how it goes.
In my experience, software developers generally prefer some flavor of
programmer’s tools when it comes to getting things done. We like plain
text, text editors, command line programs, source control, markup, and
shells. In contrast, non-developer computer users generally prefer
WYSIWYG word processors and GUIs. Developers often have somewhere
between a distaste and a
revulsion
to WYSIWYG editors.
Why is this? What are programmers looking for that other users aren’t?
What I believe it really comes down to is one simple idea: clean
state transformations. I’m talking about modifying data, text or
binary, in a precise manner with the possibility of verifying the
modification for correctness in the future.
Think of a file produced by a word processor. It may be some
proprietary format, like a Word’s old .doc format, or, more likely as
we move into the future, it’s in some bloated XML format that’s dumped
into a .zip file. In either case, it’s a blob of data that requires a
complex word processor to view and manipulate. It’s opaque to source
control, so even merging documents requires a capable, full word
processor.
For example, say you’ve received such a document from a colleague by
e-mail, for editing. You’ve read it over and think it looks good,
except you want to italicize a few words in the document. To do that,
you open up the document in a word processor and go through looking
for the words you want to modify. When you’re done you click save.
The problem is did you accidentally make any other changes? Maybe you
had to reply to an email while you were in the middle of it and you
accidentally typed an extra letter into the document. It would be easy
to miss and you’re probably not set up to easily to check what changes
you’ve made.
I am aware that modern word processors have a feature that can show
changes made, which can then be committed to the document. This is
really crude compared to a good source control management system. Due
to the nature of WYSIWYG, you’re still not seeing all of the
changes. There could be invisible markup changes and there’s no way to
know. It’s an example of a single program trying to do too many
unrelated things, so that it ends up do many things poorly.
With source code, the idea of patches come up frequently. The program
diff, given two text files, can produce a patch file describing
their differences. The complimentary program is patch, which can
take the output from diff and one of the original files, and use it
to produce the other file. As an example, say you have this source
file example.c,
intmain(){printf("Hello, world.");return0;}
If you change the string and save it as a different file, then run
diff -u (-u for unified, producing a diff with extra context), you
get this output,
This is very human readable. It states what two files are being
compared, where they differ, some context around the difference
(beginning with a space), and shows which lines were removed
(beginning with + and -). A diff like this is capable of
describing any number of files and changes in a row, so it can all fit
comfortably in a single patch file.
If you made changes to a codebase and calculated a diff, you could
send the patch (the diff) to other people with the same codebase and
they could use it to reproduce your exact changes. By looking at it,
they know exactly what changed, so it’s not some mystery to them. This
patch is a clean transformation from one source code state to
another.
More than that: you can send it to people with a similar, but not
exactly identical, codebase and they could still likely apply your
changes. This process is really what source control is all about: an
easy way to coordinate and track patches from many people. A good
version history is going to be a tidy set of patches that take the
source code in its original form and add a feature or fix a bug
through a series of concise changes.
On a side note, you could efficiently store a series of changes to a
file by storing the original document along with a series of
relatively small patches. This is called delta encoding. This is how
both source control and video codecs usually store data on disk.
Anytime I’m outside of this world of precision I start to get
nervous. I feel sloppy and become distrustful of my tools, because I
generally can’t verify that they’re doing what I think they’re
doing. This applies not just to source code, but also writing. I’m
typing this article in Emacs and when I’m done I’ll commit it to
Git. If I make any corrections, I’ll verify that my changes are what I
wanted them to be (via Magit)
before committing and publishing them.
One of my longterm goals with my work is to try to do as much as
possible with my precision developer tools. I’ve already got
basic video editing and
GIF creation worked out. I’m still working out a
happy process for documents (i.e. LaTeX and friends) and
presentations.
Update January 2013: I’ve learned more about Java dependency
management and no longer use my old .ant repository. As a result, I
have deleted it, so ignore any references to it below. The only thing
I keep in $HOME/.ant/lib these days is an up-to-date ivy.jar.
With Emacs being a breeze to synchronize between my computers, I
noticed a new bottleneck emerged: my .ant
directory. Apache Ant puts everything in
$ANT_HOME/lib and $HOME/.ant/lib into its classpath. So, for
example, if you wanted to use JUnit with Ant,
you’d toss junit.jar in either of those directories. $ANT_HOME
tends to be a system directory, and I prefer to only modify system
directories indirectly through apt, so I put everything in
$HOME/.ant/lib. Unfortunately, that’s another directory to keep
track of on my own. Fortunately, I already know how to deal with
that. It’s now another Git repository,
With that in place, settling into a new computer for development is
almost as simple as cloning those two repositories. Yesterday I took
the step to eliminate the only significant step that remained:
setting up java-docs. Before you could really
take advantage of my Java extension, you really needed to have a
Javadoc directory scanned by Emacs. The results of that scan not only
provided an easy way to jump into documentation, but also provided the
lists for class name completion. Now, java-docs now automatically
loads up the core Java Javadoc, linking to the official website, if
the user never sets it up.
So if you want to see exactly how my Emacs workflow with Java
operates, it’s just a few small steps away. This should work for any
operating system suitable for Java development.
Let’s start by getting Java set up. First, install a JDK and Apache
Ant. This is trivial to do on Debian-based systems,
sudo apt-get install openjdk-6-jdk ant
On Windows, the JDK is easy, but Ant needs some help. You probably
need to set ANT_HOME to point to the install location, and you
definitely need to add it to your PATH.
Next install Git. This should be straightforward; just make sure its
in your PATH (so Emacs can find it).
Clone my .ant repository in your home directory.
cd
git clone https://github.com/skeeto/.ant.git
Except for Emacs, that’s really all I need to develop with Java. This
setup should allow you to compile and hack on just about any of my
Java projects. To test it out, anywhere you like clone one of my
projects, such as my
example project.
If that works, you’re ready to set up Emacs. First, install Emacs. If
you’re not familiar with Emacs, now would be the time to go through
the tutorial to pick up the basics. Fire it up and type CTRL + h and
then t (in Emacs’ terms: C-h t), or select the tutorial from the
menu.
Move any existing configuration out of the way,
mv .emacs .old.emacs
mv .emacs.d .old.emacs.d
Clone my configuration,
git clone https://github.com/skeeto/.emacs.d.git
Then run Emacs. You should be greeted with a plain, gray window: the
wombat theme. No menu bar, no toolbar, just a minibuffer, mode line,
and wide open window. Anything else is a waste of screen real
estate. This initial empty buffer has a great aesthetic, don’t you
think?
Now to go for a test drive: open up that Java project you cloned, with
M-x open-java-project. That will prompt you for the root directory
of the project. The only thing this does is pre-opens all of the
source files for you, exposing their contents to dabbrev-expand and
makes jumping to other source files as easy as changing buffers — so
it’s not strictly necessary.
Switch to a buffer with a source file, such as
SampleJavaProject.java if you used my example project. Change
whatever you like, such as the printed string. You can add import
statements at any time with C-x I (note: capital I), where
java-docs will present you with a huge list of classes from which to
pick. The import will be added at the top of the buffer in the correct
position in the import listing.
Without needing to save, hit C-x r to run the program from Emacs. A
*compilation-1* buffer will pop up with all of the output from Ant
and the program. If you just want to compile without running it, type
C-x c instead. If there were any errors, Ant will report them in the
compilation buffer. You can jump directly to these with C-x `
(that’s a backtick).
Now open a new source file in the same package (same directory) as the
source file you just edited. Type cls and hit tab. The boilerplate,
including package statement, will be filled out for you by
YASnippet. There are a bunch of completion snippets available. Try
jal for example, which completes with information from java-docs.
When I’m developing a library, I don’t have a main function, so
there’s nothing to “run”. Instead, I drive things from unit tests,
which can be run with C-x t, which runs the “test” target if there
is one.
To see your changes, type C-x g to bring up Magit and type M-s in
the Magit buffer (to show a full diff). From here you can make
commits, push, pull, merge, switch branches, reset, and so on. To
learn how to do all this, see the
Magit manual. You
can type q to exit the Magit window, or use S-<arrow key> to move
to an adjacent buffer in any direction.
And that’s basically my workflow. Developing in C is a very similar
process, but without the java-docs part.
I finally got my entire Emacs configuration into source control. My
previous solution was to copy around my .emacs.d/ to each computer I
use. This works well enough with two computers, but beyond that it’s
difficult to propagate any changes I make. Counting all my VMs, I have
around a dozen systems where I use Emacs. This is the exact problem
that source control exists to fix.
If you move your .emacs and .emacs.d/ out of the way, clone my
repository right into your home directory, clone the submodules, and
then run Emacs 23 or greater, you’ll see my exact Emacs setup, theme
and all.
Notice there’s an init.el in there. Emacs tries to load ~/.emacs
first, but if that doesn’t exist it loads ~/.emacs.d/init.el. That’s
why you need to move your own .emacs out of the way to see my
stuff. I do still make use of a .emacs file. That’s my
system-specific configuration, where, for example, I tell Emacs where
to find Javadoc files. At the top of this file I
make sure to load my other init file.
;; Load standard configuration(load-file"~/.emacs.d/init.el")
One reason I didn’t use source control right away was the submodule
problem — my configuration is largely made up of other repositories.
Git has good support for putting foreign Git repositories within your
own repository, but a couple of repositories I was using were
Subversion and CVS. I managed to cut down to just Git repositories
and one Subversion repository, for which I now maintain a Git
mirror, making these *all* Git repositories
. (Update November
2011: YASnippet has moved to Git.)
I also trimmed down a bit, cutting out some things I noticed I wasn’t
using (breadcrumbs, pabbrev) or things that didn’t need to be in
there, such as Slime. I now use Quicklisp
to manage my Slime installation, which I connect with my configuration
in my system-specific .emacs. Using source control will help better
track what I’m using and not using, keeping the whole thing more
tidy. Removing an experimental addition should be a simple revert
commit.
It works like so (see example.el for detailed information on this
code),
(require'fakespace)(defpackageexample(:useclido)(:exportexample-mainexample-vareq-hellohello))(defvarmy-var100"A hidden variable.")(defvarexample-varnil"A public variable.")(defunmy-func()"A private function."my-var)(defunexample-main()"An exported function. Notice we can access all the private
variables and functions from here."(interactive)(list(list(my-func)my-var)example-var(ido-completing-read"New value: "(list"foo""bar"))))(defuneq-hello(sym)(eqsym'hello))(end-package)
Notice end-package at the end, which is not needed in Common
Lisp. That’s part of what makes it crude.
If you run those functions and try changing the assignment of
non-exported symbols, you’ll see the namespace separation in
action. my-var and my-func are a completely different symbols than
the ones you’re seeing after end-package.
It’s really simple in how it works (it’s 40 lines of code). The
defpackage macro takes a snapshot of the symbol table. Then new
symbols get interned through various function and variable
definitions. Finally end-package compares the current symbol table
to the snapshot and uninterns any new symbols. These symbols will be
unaccessible to other code, effectively giving them their own
namespace.
Snapshots are pushed onto a stack, so it’s safe to create a new
package within another package, as long as end-package is used
properly. This is necessary when one namespaced package depends on
another, because the dependency will tend to be loaded in the middle
of defining the current package.
in-package is not provided, so there’s no way to get the symbols
back to where they can be accessed. It’s impossible to modify a
package using fake namespacing. Worst of all, implementing
in-package is currently (and will likely always be) impossible. When
symbols are uninterned they would need to be stored in a package
symbol table for future re-interning. in-package’s job would be to
unintern and store away the current package’s symbols and then place
the new package’s symbols into the main symbol table.
However, symbols cannot be re-interned. This is because it’s
impossible for a symbol to exist in two different obarrays at the same
time, so the functionality is intentionally not provided. An obarray
is an Elisp vector containing symbols. It’s treated like a hash table:
the symbol is hashed to choose a location in the vector. If the slot
is already taken, the symbol is invisibly chain behind the residing
symbol by an inaccessible linked list. If the symbol was in two
obarrays at once, it would need to be able to chain to two different
symbols at the same time.
Providing access to symbols through a colon-specificed namespace
(my-package:my-symbol) is also currently impossible — without
hacking in C anyway.
There’s a neat trick to the :export list. The defpackage macro
definition actually ignores that list altogether, because it works
automatically. By the time defpackage is invoked, the listed symbols
have already been interned by the reader, so they get stored in the
snapshot.
I doubt I’ll ever make use of this for my own packages. This was
mostly a fun exercise in toying with Elisp.
During my recent Elisp hacking I've run into the situation enough
times where I really wanted function composition that I officially
implemented it for myself. While there is
an apply-partially
function, Elisp does not currently come with a compose
function. Here's an Elisp definition,
;; ID: f0c736a9-afec-3e3f-455c-40997023e130(defuncompose(&restfuns)"Return function composed of FUNS."(lexical-let((lex-funsfuns))(lambda(&restargs)(reduce'funcall(butlastlex-funs):from-endt:initial-value(apply(car(lastlex-funs))args)))))
Here it is in action with three functions.
(funcall(compose'prin1-to-string'random*'exp)10)
I'll be using this in later posts (and linking back here when I do).
There's an extension to Emacs
called JDEE which tries to
turn Emacs into a heavyweight IDE for Java. I've never had any success
with it, and I don't know anyone else who has either. It's difficult
to set up, the dependencies are even worse, poorly documented, and
then it doesn't seem to work very well anyway. I think it's too
divorced from Emacs'
core
composable functionality to be of much use. I may as well be using
a big IDE.
So, instead, as I've
postedaboutover time, I've started with the basic
Emacs Java functionality and tweaked my way up from there. I've
extended it enough that I decided to package it up on it's own, and
hopefully others will find it useful too. I call
it java-mode-plus!
It provides a hook into java-mode that creates a bunch of
new bindings. It also creates some new globally-available functions
like open-java-project. It's all heavily Ant-based since
that's what I like to use. It wouldn't be very hard to modify it to
use Maven, if that's what your thing.
My very thorough documentation is in a large header comment in the
source file itself. I cover my whole workflow from top to
bottom. If you're interested in making Emacs more Java-friendly take a
look at it. It's not a lot of code, but each line has been
thoughtfully added after hours and hours of Java development.
Update January 2013: this package has been refined and formally
renamed
to javadoc-lookup.
The user interface is essentially the same — under different function
names — but with some extra goodies. It's available for install from
MELPA.
I keep running to either a search engine or, when offline, manually
browsing to Java API documentation when I need to look something
up. When I'm using Emacs this is stupid, so I fixed it. I put together
a java-docs package that let's me quickly jump to
documentation from within Emacs.
Unfortunately it launches it in a web browser right now because there
doesn't seem to be a reasonable way to render the
documentation inside Emacs itself. So
you'll need to
have browse-url set up properly in your
configuration.
I strongly recommend you use this
with
Ido, which comes with Emacs. If you do, you'll want to load
it after you enable ido-mode, which will enable
the Ido minibuffer completion in java-docs.
So, after you requirejava-docs, you give it
a list of places to look for documentation.
It will scan these locations and build up an index of
classes. If you're using a recent enough
version of Emacs it will cache that index for faster loading in
the future, since on certain systems it can needlessly take a
bit of time.
After that you can jump to documentation with C-h j
(java-docs-lookup). It will ask you what you want to look
up and offer completion with your preferred completion function.
If you don't want to open it up in an external browser, you can set
Emacs to run a text-based browser inside itself.
When I'm coding, I maximize Emacs and enable
winner-mode, turning my display into something much
like a
tiling window manager. Then I try not to leave Emacs until it's
necessary. It's a really nice way to work: no mouse touching needed.
At work they gave me a nice 24" monitor, 1920 pixels across. That's
just about enough to fit three Emacs' windows side-by-side at 78
columns each. The leftmost one contains my active work buffer where I
do most of my typing. The center one is usually split
horizontally. The top half is the *compilation* buffer
and the bottom half is either Emacs
calculator or an *ansi-term* buffer. The rightmost
buffer contains something more static, like some sort of reference
material.
However, I like my main editing window to be 80 columns wide. 78
columns cuts just too short. For awhile I was creating 80 dashes
(C-u 80 -) and adjusting the window width manually to
size. After doing it a few times I decided to extend Emacs to do it
instead. First define a function to set the current window width.
(defunset-window-width(n)"Set the selected window's width."(adjust-window-trailing-edge(selected-window)(-n(window-width))t))
Wrap it with an interactive function and bind it.
(defunset-80-columns()"Set the selected window to 80 columns."(interactive)(set-window-width80))(global-set-key"\C-x~"'set-80-columns)
The problem with this, though, is the dynamically generated function
doesn't have a name or a docstring. Someone
using describe-key would have little information to go
on.
Here's another bit of code I started using recently. I often find
myself wanting to open — or reopen
after kill-matching-buffers — all the files under a
specific point in the file system. I'm using it at work now to open up
all the source files in a deep Java source tree on small-ish
project. Once it's all open I can switch to any file quickly
with
ido's fuzzy matching, flattening out the directory structure a
bit. (And the ridiculous "security" software at work imposes a
3-second I/O block when opening files, so I get to pay this all up
front at once rather than having it later
break my
flow.)
This just recursively travels down the sub-directories opening a
buffer for everything it comes across. It ignores dot-files, like the
ones your source control might litter.
;; ID: 72dc0a9e-c41c-31f8-c8f5-d9db8482de1e(defunfind-all-files(dir)"Open all files and sub-directories below the given directory."(interactive"DBase directory: ")(let*((list(directory-filesdirt"^[^.]"))(files(remove-if'file-directory-plist))(dirs(remove-if-not'file-directory-plist)))(dolist(filefiles)(find-file-noselectfile))(dolist(dirdirs)(find-file-noselectdir)(find-all-filesdir))))
One caveat: if you have a symbolic link that creates a file system
loop, this will probably get hung on it.
For those not familiar with extending Emacs, when you create a
function in Elisp it cannot be called directly by the user
("interactively") without declaring the function interactive. The
simplest way to do this is by adding (interactive) to the
top of the function definition. The interactive call can
be made more complex, if needed, to ask the user interactively for
input.
There are some handy higher-order functions in Elisp, such
as compose and apply-partially. Today I
wanted to bind the output of apply-partially to a key. My
situation was this: I use revert-buffer often enough that
it needs a binding. Also because I use it so much, I wanted it to
stop asking me for confirmation. (Yes,
there are other
ways to do this including revert-without-query, but I
wanted a general solution.) Using apply-partially I could
supply the needed function arguments at keybind time.
The problem is that you can only bind interactive functions, and the
output of apply-partially is not interactive. A quick way
to work around this is to wrap it in an anonymous function, which also
takes away the need for apply-partially.
(lambda()(interactive)(revert-buffernilt))
I'd rather there be another higher-order function that takes a
non-interactive function and creates an interactive version. Here it is,
;; ID: c7db6dec-e7ab-3b0f-bf26-0fa268674c6c(defunexpose(function)"Return an interactive version of FUNCTION."(lexical-let((lex-funcfunction))(lambda()(interactive)(funcalllex-func))))
I got an Elisp idea today and even went as far as implementing a proof
of concept for it: distributed computing with Emacs Lisp. As usual for
me the idea takes advantage of Lisp features to make the task pretty
simple, very specifically Elisp's implementation. In this case it's
the Lisp reader, printer, and the fact that Elisp functions have a
printed representation, both byte-compiled and not.
A central server listens for TCP connections. Clients offering their
CPU for use connect to the server and await instructions. The server
sends a single, no-argument, anonymous function to the client. The
client calls the function, returning the resulting form back to the
server. In order to transmit the function it's encoded into a string
using the Lisp printer, and the client turns it back into an
executable function with the Lisp reader.
For some simple security there is a shared password between the client
and server. When the server sends a function it includes a signature,
and the client only runs code that matches the signature. To create a
signature the string encoded version of the function is appended with
the password (both strings) and hashed with a secure hashing
algorithm. Only someone who knows the password — including other
clients — can create the signature.
(defunsign-sexp(passwordsexp)"Return signature of the given s-exp."(sha1(format"%s%s"passwordsexp))
To make it easy for the client to read in both the signature and the
function we just cons them together before encoding them as text.
(defunencode(passwordsexp)"Encode a s-exp for transmission to client."(prin1-to-string(cons(sign-sexppasswordsexp)sexp)))
The client calls the Lisp reader on the string, then checks the
signature in the car cell against the s-expression in
the cdr cell. This will return the function if it's
legitimate, otherwise nil.
(defundecode(passwordstr)"Decode string into s-exp, checking the signature in the process."(let*((cons(readstr))(sig(carcons))(sexp(cdrcons)))(if(equalsig(sign-sexppasswordsexp))sexpnil)))
And that's the core of it. It just needs some network code to move the
string between computers. That part can be found in the linked source
above.
To demo this, I'll use the whiten function from
my previous post. I'll run it with
three different strings on three different computers. Assume we
started the dist-emacs server (dist-start) and connected
three clients (dist-connect) from three computers to
it. The clients were fired up from scratch so there's
no whiten function on them yet, but there is one
defined on the server. First we'll send the function definition to the
clients. The dist-dist function takes a list of functions
and passes each one to a client. Ideally I'd want this function to be
more intelligent, managing a work queue so that an arbitrary length
list of functions will be fed one at a time to each client. That's not
the case here.
Also like in the previous post, this is an abstraction leak with the
Emacs implementation. But I like this trick so I'm going to use it
anyway. :-) Next we call it on each client with a different string.
The way I have it set up for my proof of concept the results are just
spit back into the server's *Messages* buffer. If we
watch that buffer we can see each results come back in one at a time
as each machine finishes. I can watch Emacs saturate the CPU on every
client machine simultaneously as it works.
This isn't the same order as the clients, but the order in which the
jobs were completed.
As for the practicality, I doubt there really is one. It's really only
a neat concept (or maybe not even that). For almost the exact same
reasons as my distributed JavaScript
idea, this is a solution looking for a problem. The problem needs to
be able to be broken into small computation units, because Emacs has
no threading, and it has to be low bandwidth, because it has to be
parsed all at once from a string. If you want to pass large data sets
it needs to be done out-of-band, which probably defeats the
purpose. There seem to be few to no problems that fit these
limitations.
Memoization is something I think
should be packaged as a standard function for just about every
language. That's not generally the case, but luckily this is easy to
fix in Lisps. I needed memoization recently for an Elisp project I'm
working on. I could have hand-written one but a generic memoization
function would have worked just fine. Since I didn't find any generic
Elisp memoization on-line I wrote my own.
Just put it in your path
and (require 'memoize) it. Here's the core
function.
;; ID: 83bae208-da65-3e26-2ecb-4941fb310848(defunmemoize-wrap(func)"Return the memoized version of FUNC."(lexical-let((table(make-hash-table:test'equal)))(lambda(&restargs)(let((value(gethashargstable)))(ifvaluevalue(puthashargs(applyfuncargs)table))))))
The hash table is stored inside the fake closure provided by
lexical-let. In a previous version of this function, I
stored it in an uninterned symbol, which is what is going on behind
the scenes of lexical-let.
Note that in the full code it keeps the original function
documentation intact. I want the memoization wrapper to be an
unobtrusive as possible.
Here's a demo of it in action. This whiten function is
computationally expensive: it performs key whitening. It repeats a
hash function thousands of times to produce an expensive value. This
isn't something you generally want to memoize, but stick with me.
(defunwhiten(key)"Perform key whitening with the md5 hash function."(dotimes(i100000key)(setqkey(md5key))))(whiten"password"); takes a couple of seconds
On my laptop that takes a couple of seconds to run. Increase that
counter if it's quick on your computer. My memoize package provides
a memoize function which will create a new function that
wraps the original, then installs the new function in place of the old
one if we give it the function symbol.
(memoize'whiten)
The first time you run it after memoization it will be slow, but after
that the memoization kicks in for a quick return.
There are two Elisp specific issues at hand. First is that memoizing
an interactive function will produce a non-interactive function. It
would be easy to fix this problem when it comes to non-byte-compiled
functions, but recovering the interactive definition from a
byte-compiled function is more complex than I care to deal
with. Besides, interactive functions are always used for their side
effects so there's no reason to memoize them.
Second is a limitation of Elisp hash tables. There's no way to
distinguish a nil value and no value. The hash table returns nil for
both. This means you cannot memoize nil returns. But a computationally
expensive function shouldn't be returning nil anyway.
Update: As of August 2012, me and several other people have
gotten good mileage out of this function! It's an essential part of my
Emacs dotfiles.
A feature unique to some Lisps is the ability to compile functions
individually at any time. This could be to a bytecode or native code,
depending on the dialect and implementation. In a Lisp implementations
where compilation matters (such as
CLISP), there are typically two forms in which code can be
evaluated: a slower, unoptimized uncompiled form and a fast, efficient
compiled form. The uncompiled form would have some sort of advantage,
even if it's merely not having to spend time on compilation.
In Emacs Lisp, the uncompiled form of a function is just a lambda
s-expression. The only thing that gives it a name is the symbol it's
stored in. The compiled form is a (special) vector, with the actual
byte codes stored in a string as the second element. Constants, the
docstring, and other things are stored in this function vector as
well. The Elisp function to compile functions
is byte-compile. It can be given a lambda function or a
symbol. In the case of a symbol, the compiled function is installed
over top of the s-expression form.
The compiler will not only convert the function to bytecode and expand
macros, but also perform optimizations such as removing dead code,
evaluating safe constant forms, and inline functions. This provides a
nice performance boost (testing using my
measure-time macro),
Most of the installed functions in a typical Emacs instance are
already compiled, since they are loaded already compiled. But a number
of them aren't compiled. So, I thought, why not spend a few
seconds to do this?
In Common Lisp, there is a predicate for testing whether a function
has been compiled or not: compiled-function-p. For
whatever reason, there is no equivalent predefined in Elisp, so I
wrote one,
(defunbyte-compiled-p(func)"Return t if function is byte compiled."(cond((symbolpfunc)(byte-compiled-p(symbol-functionfunc)))((functionpfunc)(not(sequencepfunc)))(tnil)))
My idea was to iterate over every interned symbol and, if the function
slot contains an uncompiled function, using the test above, I would
call byte-compile on it. Well, it turns out
that byte-compile is very flexible and will ignore
symbols with no function and symbols with already compiled functions.
So next, how do we iterate over every interned symbol? There is
a mapatoms function for this. Provide it a function and
it calls it on every interned symbol. Well, that's simple and
anticlimactic.
(mapatoms'byte-compile)
That's it! It will take only a few seconds and spew a lot of
warnings. I haven't found a way to disable those warnings, so this
isn't something you'd want to have run automatically, unless you like
having an extra window thrown in your face. I've only discovered this
recently, so I'm not sure what sort of bad things this may do to your
Emacs session. Not every function was written with compilation in
mind. There are interactions with macros to consider.
I doubt there will be a noticeable performance difference. Like I said
before, most everything is already compiled, and those are the
functions that get used the most. There's just something nice about
knowing all your functions are compiled and optimized.
ParEdit is a
powerful extension to Emacs that I've just begun using recently. It's
a minor mode that forces all parenthesis, square brackets, and quotes
to be balanced at all times. While it's useful for any programming
language it's especially suited for Lisps, because it's designed for
manipulating nested parenthesis — i.e. s-expressions. It's not
currently part of Emacs so you have to drop the script in
your load-path somewhere.
I've frequently thought that a Lisp-based shell would be an
interesting and powerful tool, much like a normal Lisp REPL. Programs
would be treated like Lisp functions. For example,
But typing all those parenthesis all the time would be quite the
nuisance. I know this from experience typing at Lisp REPLs. I imagined
something that works exactly like ParEdit would be needed to make all
that work go away. To save even more time each prompt would begin with
a nested pair, with the cursor placed between them. Then typing a
quick command is no different than a normal shell.
wellons@luna:~$ ()
Well, in Emacs we have both ParEdit and REPLs, so we can compose these
features together with just a little advice. Here's how to do it with
the Interactive Emacs-Lisp Mode (IELM) REPL. First tell IELM to use
ParEdit,
The function in IELM that spits out the next prompt
is ielm-eval-input, so we give it the advice to call the
ParEdit function afterwards to insert a parenthesis pair.
(defadviceielm-eval-input(afterielm-pareditactivate)"Begin each IELM prompt with a ParEdit parenthesis pair."(paredit-open-round))
And that's it! Note that the first IELM prompt is not placed by this
function so it won't appear until the second prompt.
*** Welcome to IELM *** Type (describe-mode) for help.
ELISP>
ELISP> ()
If you want to enter a single atom and don't need parenthesis, just
hit backspace once. This is much less common so it gets the extra
keystroke.
This can be done for inferior-lisp
and SLIME to enhance those REPLs as
well. You just have to figure out which defun to advise.
With this hash tables can be printed and read as part of normal
s-expressions with the standard lisp reader and printer functions. It
seems heavy, having to write out "hash-table" in there,
but I think it's because the #s notation will be used to
create printed forms of other lisp objects that currently do not have
one.
I was inspired by an item
in
Luke's
Tumblr blog last night. It was a screenshot of a program
called PawSense, which
monitors a computer's keyboard for cat activity. (I don't know if it's
any good, but it's funny.) As anyone with cats knows, it's not unusual
to leave a computer only to come back later to see garbage typed in by
a wandering cat. I wrote a version for Emacs today.
Put it (cat-safe.el) somewhere in
your load-path (like ~/.emacs.d/) and put
this line in your .emacs file,
(require'cat-safe)
This only monitors Emacs itself; it should help protect your buffers
but not your web browser. When cat interference is detected Emacs
switches focus to a junk buffer and lets the cat make a mess there
instead. In case your
cat happens
to type out some Shakespeare you will be able to read it in the
junk buffer. Just kill the junk buffer to return to work.
It could still use some improvement. Right now it looks for a single
key being help down, excepting keys humans tend to hold down like
backspace, delete, and space. If you play around with it you'll notice
if you press several keys at once Emacs will sometimes create a
pattern with them. I need to figure out a good way to detect this.
I'm going to run it at home for awhile to make sure it remains
transparent, but still does its job. It will probably incur a
performance penalty on frequently repeated keyboard macros.
Update August 2011: Things have changed again, which has always been
the problem with Slime, and the reason I originally wrote
this. Currently, I think the best way to install Slime is
with Quicklisp
using quicklisp-slime-helper.
Common Lisp is possibly the most advanced programming language. Think
of pretty much any programming language feature and Common Lisp
probably has it. Since lisp is the programmable programming language,
when someone invents a new language feature it can probably be added
to Common Lisp without even touching the language core.
However, if you're interested in digging into Common Lisp to try it
out, you may find yourself quickly running into walls just getting
started. It's a lot different than other programming environments you
may be used to. The Common Lisp tutorials generally skip this step,
assuming the user has an environment, or leaving that setup for the
"vendor" to handle. So, here's a guide to setting up a great Common
Lisp environment with
Emacs and
SLIME. It should work with any Common Lisp implementation and any
operating system that can run Emacs (i.e. most of them). Even a much
less capable one like Windows.
First, you need to pick a Common Lisp implementation and install
it. Ideally, it should end up in your PATH. Like C, the language is
defined solely by its standardized specification, rather than some
canonical implementation. Steel Bank
Common Lisp (SBCL) is currently
the
highest
performing implementation, it's Free Software, and it runs on a
wide variety of platforms, so take a look at that one if you're not
sure.
Next, install Emacs. We're using Emacs not just because it's the best
text editor ever created. :-D It's because that's what
SLIME is written for, and Emacs is a lisp-aware editor. Really, Emacs
is a lisp interpreter that happens to be geared towards
text-editing. It's accused of breaking the rules of unix by being a
single, monolithic program, but it's really a whole bunch of small
lisp programs. You can even have a lisp REPL in Emacs
(ielm), similar to what we will have once we're done
here. It's plays very well with Common Lisp.
If you're unfamiliar with Emacs, you should stop here and familiarize
yourself with it a bit. Really, you could spend a decade learning
Emacs and still have more to learn. The tutorial should be good enough
for now. Fire up Emacs and run the tutorial by pressing
control+h then t. In Emacs notation,
that's C-h t. C-h is the help/documentation
prefix, which can be used to look up variables/symbols
(v), functions (f), key bindings
(k), info manuals (i), the current mode
(m), and apropos (searching) (a). In the
info manuals, you should be able to find the full Emacs manual, Elisp
reference, and Elisp tutorial, since they are generally installed
alongside Emacs these days. Nearly anything you might need to know can
be found inside the included documentation.
Next, install SLIME. I'll be a bit more specific for this one. Make
a .emacs.d directory in your home directory (whatever
your HOME environmental variable is set to). This is a common place to
put user-installed Emacs extensions. You will be putting
your slime directory in here. There are two basic ways to
obtain SLIME, as indicated right on their main page. You can do a CVS
checkout of the SLIME repository, which allows you to follow it and
run the latest version. Or you can grab a snapshot of the repository,
which is provided, and dump it in there. Since I like you so much,
I'll give you a third option. Here's a Git repository, maintained by
someone very kind, that follows SLIME's CVS repository,
git clone git://git.boinkor.net/slime.git
Ultimately, you should have a directory ~/.emacs.d/slime/
that contains a bunch of SLIME source files directly inside.
Now, we tell Emacs where SLIME is and how to use it. Make
a .emacs file in your home directory, if you haven't
already, and put this in it,
Once it's saved, either restart Emacs, or simply evaluate those lines
by putting the cursor after each them in turn and typing C-x
C-e. If you did everything right so far, you shouldn't have any
errors. (If you did, go back up and see what you did wrong.) If your
Common Lisp installation didn't end up in your PATH as
"lisp" (not uncommon) for some reason, you may need to
tell Emacs where it is. For example, I can point directly to my SBCL
installation with this line,
(setqinferior-lisp-program"/usr/bin/sbcl")
If everything is set up right, fire up SLIME with "M-x
slime". It should compile the back-end, called swank, and run a
Common Lisp REPL as an inferior process to Emacs. You should end up
with a nice prompt like this,
CL-USER>
At this line, you can start evaluating lisp expressions as you
please. But this isn't where the true power of SLIME comes in
yet. I'll give you an example: make a new file with
a .lisp extension and open it. Throw some lisp in there,
(defunadder(x)(lambda(y)(+xy)))
Type C-x C-k and it will send the current buffer over to
be compiled and loaded. This code here uses a closure, so you know you
aren't accidentally using Emacs lisp, as it doesn't have closures. At
the REPL you can call it,
CL-USER>(funcall(adder5)6)
Which will print the return value, 11. That's all there
is to it. You write code in the buffer, then with a simple keystroke
send it to the Common Lisp system to be evaluated and loaded. Because
the SLIME key bindings eclipse the Emacs lisp key bindings, you can
type this same line in the lisp source buffer place the cursor at the
end, and type C-x C-e, which will send it out to Common Lisp to be
evaluated. Look at the mode help (C-h m) to see all the
key bindings made available.
This is a great programming environment that makes Common Lisp all the
more fun to use. You run a single, continuous instance if your program
growing it gradually. (This is exactly how I
built my Emacs web server with elisp.)
You can test your code as soon as soon as it's written.
The setup can get even more advanced. The Common Lisp REPL need not be
running on the same computer. It can be running on another computer,
as long as SLIME is able to connect to it over the network. Several
developers could even share a single Common Lisp process running on a
common machine. Lots of possibilities.
If you don't have a Common Lisp book yet,
there's Practical Common
Lisp, which you can read at no cost online
or download
for reading offline. It's based on an Emacs and SLIME setup, so you'll
be right on track.
Update: This is now part of my
java-mode-plus Emacs extension.
Developing C in Emacs is a real joy, and it's mostly thanks to the
compile command. Once you have your Makefile — or SConstruct or
whatever build system you like — setup and you want to compile your
latest changes, just run M-x compile, which will run your
build system in a buffer. You can then step through the errors and
warnings with C-x `, and Emacs will take you to them.
It's a very nice way to write code.
I use the compile command so much that I bound it to C-x
C-k (C-k tends to be part of compile key
bindings),
(global-set-key"\C-x\C-k"'compile)
Until recently, I didn't have as nice of a setup for Java. Since they
generally force offensive IDEs onto me at work this wasn't something I
needed yet anyway, but I get to choose my environment on a new
project this time. If you're using Makefiles for some reason when
building your Java project, it still works out fairly well because
they're usually called recursively. It gets more complicated with Ant, where there is only one
top-level build file. Emacs' compile command only runs the build
command in the buffer's current directory.
I know three solutions to this problem. One is to provide the build
file's absolute path when compile asks for the command
with the -buildfile (-f) option. You only
need to type it once per Emacs session, so that's not too bad.
ant -emacs -buildfile /path/to/build.xml
It's not well documented, but there is a -find option
that can be given to Ant that will cause it to search for the build
file itself. This is even nicer than the previous solution. Just
remember to place it last, unless you give it the build filename
too. For example, if you wanted to run the clean target,
ant -emacs clean -find
To keep the actual call as simple as possible, I wrote a wrapper for
compile, and put a hook in java-mode to
change the local binding. The wrapper, ant-compile,
searches for the build file the same way -find would do.
(defunant-compile()"Traveling up the path, find build.xml file and run compile."(interactive)(with-temp-buffer(while(and(not(file-exists-p"build.xml"))(not(equal"/"default-directory)))(cd".."))(call-interactively'compile)))
So I can transparently keep using my muscle memory compile binding, I
set up the key binding in a hook,
Remember that Emacs web server I wrote
back in May? Well, I got an e-mail last night from Chunye Wang
containing a patch with a variant of my dynamic lisp idea, called
"servlets" (not to be confused with Java servlets). Chunye had similar
concept for an Emacs web server for a long time, but never implemented
because Emacs lacked network functionality until recently
(Specifically, make-network-process in Emacs 22.1, June 2007). This
led Chunye to find my implementation.
Again, you can clone/view the code here. I turned the patch into a
series of commits,
The servlets are simply functions installed under an
"httpd/" namespace, where the trailing slash represents
the server root. So, the function httpd/example-servlet
will be executed when "/example-servlet" is requested from the
server. The servlet runs on a temporary buffer, whose contents are
served when the servlet function returns.
To assist in HTML generation, Chunye also wrote a function to turn an
S-expression
into HTML, similar to the one I described in the web server previous
post. Symbols are converted into strings, alists are attributes, and
the elisp symbol indicates code to be executed, and the
results used to generate HTML. For a simple hello word,
(defunhttpd/roll-die(uri-queryrequri-path)"Rolls a die with the requested number of sides (default 6)."(let((sides(1-(string-to-number(or(cadr(assoc"sides"uri-query))"6")))))(httpd-generate-html'(html(head(title"Die Roll Servlet"))(body(h1"Die Roll Servlet")"You rolled a "(b(elisp(list(number-to-string(1+(randomsides)))))))))))
That one would be accessed from the browser with with
"/roll-die" or "/roll-die?sides=100".
Chunye provided some sample servlets that list the buffers, with links
that serve them up. There is also another servlet that will switch the
current buffer, which I find compelling. All of Emacs' functionality
is available to the servlet.
Now, to write a servlet that runs the Emacs psychiatrist ...
Did you know that Emacs comes with a
calculator? Woop-dee-doo! Call the presses! Wow, a whole
calculator! Sounds a bit lame, right?
Actually, it's much more than just a simple calculator. It's a computer
algebra system! It is officially called a calculator, which isn't
fair. It's an understatement, and I am sure has caused many people to
overlook it. I finally ran into it during a thorough (re)reading of
the Emacs manuals and almost skipped over it myself.
Ever see that demonstration by Will Wright for the game Spore
several years ago? The player starts as a single-cell organism and
evolves into a civilization with interstellar presence. When he
started the demo he showed a cell through what looked like a
microscope. No one had any idea yet what the game was about, so every
time he increased the scope, from bacteria to animal, animal to
civilization, civilization to space travel, interplanetary travel to
interstellar travel, there was a huge reaction from the audience. It
was like those infomercials: "But that's not all!!!"
As I made my way through the Emacs calc manual I was continually
amazed by its power, with a similar constant increase in scope. Each
new page was almost saying, "But that's not all!!!"
Like an infomercial I'm going to run through some of its features. See
the calc manual for a real thorough introduction. It has practice
exercises that shows some gotchas and interesting feature
interactions.
Fire it up with C-x * c or M-x calc. There
will be two new windows (Emacs windows, that is), one with the
calculator and the other with usage history (the "trail").
First of all, the calculator operates on a stack and so its basic use
is done with RPN. The stack builds vertically, downwards. Type in
numbers and hit enter to push them onto the stack. Operators can be
typed right after the number, so no need to hit enter all the
time. Because negative (-) is reserved for subtraction an
underscore _ is used to type a negative number. An
example stack with 3, 4, and 10,
3: 3
2: 4
1: 10
.
10 is at the "top" of the stack (indicated by the "1:"), so if we type
a * the top two elements are multiplied. Like so,
2: 3
1: 40
.
The calculator has no limitations on the size of integers, so you work
with large numbers without losing precision. For example, we'll
take 2^200.
But that's not all!!! It has a complex number type, which is entered
in pairs (real, imaginary) with parenthesis. They can be operated on
like any other number. Take -1 + 2i minus 4 +
2i,
2: (-1, 2)
1: (4, 2)
.
Subtract with -,
1: -5
.
Then take the square root of that using Q, the square
root function.
1: (0., 2.2360679775)
.
We can set the calculator's precision with p. The default
is 12 places, showing here 1 / 7.
Numbers can be displayed in various notations, too, like fixed-point,
scientific notation, and engineering notation. It will switch between
these without losing any information (the stored form is separate from
the displayed form).
But that's not all!!! We can represent rational numbers precisely with
ratios. These are entered with a :. Push
on 1/7, 3/14, and 17/29,
3: 1:7
2: 3:13
1: 17:29
.
And multiply them all together, which displays in the lowest form,
1: 51:2842
.
There is a mode for working in these automatically.
But that's not all!!! We can change the radix. To enter a number with
a different radix, which prefix it with the radix and a
#. Here is how we enter 29 in base-2,
2#11101
We can change the display radix with d r. With 29 on the
stack, here's base-4,
1: 4#131
.
Base-16,
1: 16#1D
.
Base-36,
1: 36#T
.
But that's not all!!! We can enter algebraic expressions onto the
stack with apostrophe, '. Symbols can be entered as part
of the expression. Note: these expressions are not entered in RPN.
1: a^3 + a^2 b / c d - a / b
.
There is a "big" mode (d B) for easier reading,
2
3 a b a
1: a + ---- - -
c d b
.
We can assign values to variables to have the expression evaluated. If
we assign a to 10 and use the "evaluates-to" operator,
2
3 a b a 100 b 10
1: a + ---- - - => 1000 + ----- - --
c d b c d b
.
But that's not all!!! There is a vector type for working with vectors
and matrices and doing linear algebra. They are entered with
brackets, [].
Any matrix and vector operator you could probably think of is
available, including map and reduce (and you can define your own
expression to apply).
We can use this to solve a linear system. Find x
and y in terms of a and b,
x + a y = 6
x + b y = 10
Enter it (note we are using symbols),
2: [6, 10]
1: [ [ 1, a ]
[ 1, b ] ]
.
And divide,
4 a 4
1: [6 + -----, -----]
a - b b - a
.
But that's not all!!! We can create graphs if gnuplot is installed. We
can give it two vectors, or an algebraic expression. This plot
of sin(x) and x cos(x) was made with just a
few keystrokes,
But that's not all!!! There is an HMS type for handling times and
angles. For 2 hours, 30 minutes, and 4 seconds, and some others,
3: 2@ 30' 4"
2: 4@ 22' 13"
1: 1@ 2' 56"
.
Of course, the normal operators work as expected. We can add them all up,
1: 7@ 55' 13"
.
We can convert between this and radians, and degrees, and so on.
But that's not all!!! The calculator also has a date type, entered
inside angled brackets, <> (in algebra entry
mode). It is really flexible on input dates. We can insert the current
date with t N.
1: <6:59:34pm Tue Jun 23, 2009>
.
If we add numbers they are treated as days. Add 4,
1: <6:59:34pm Sat Jun 27, 2009>
.
It works with the HMS format from before too. Subtract 2@ 3'
15".
1: <4:56:32pm Sat Jun 27, 2009>
.
But that's not all!!! There is a modulo form for performing modulo
arithmetic. For example, 17 mod 24,
1: 17 mod 24
.
Add 10,
1: 3 mod 24
.
This is most useful for forms such as n^p mod M, which
this will handle efficiently. For example, 3^100000 mod
24. The naive way would be to find 3^100000 first,
then take the modulus. This involves a computationally expensive
middle step of calculating 3^100000, a huge number. The
modulo form does it smarter.
But that's not all!!! The calculator can do unit conversions. The
version of Emacs (22.3.1) I am typing in right now knows about 159
different units. For example, I push 65 mph onto the stack,
1: 65 mph
.
Convert to meters per second with u c,
1: 29.0576 m / s
.
It is flexible about mixing type of units. For example, I enter 3
cubic meters,
3
1: 3 m
.
I can convert to gallons,
1: 792.516157074 gal
.
I work in a lab without Internet access during the day, so when I need
to do various conversions Emacs is indispensable.
The speed of light is also a unit. I can enter 1 c and
convert to meters per second,
1: 299792458 m / s
.
But that's not all!!! As I said, it's a computer algebra system so it
understands symbolic math. Remember those algebraic expressions from
before? I can operate on those. Let's push some expressions onto the
stack,
3: ln(x)
2 a x
2: a x + --- + c
b
1: y + c
.
Multiply the top two, then add the third,
2 a x
1: ln(x) + (a x + --- + c) (y + c)
b
.
Expand with a x, then simplify with a s,
2 a x y 2 a c x 2
1: ln(x) + a y x + ----- + c y + a c x + ----- + c
b b
.
Now, one of the coolest features: calculus. Differentiate with respect
to x, with a d,
1 a y a c
1: - + 2 a y x + --- + 2 a c x + ---
x b b
.
Or undo that and integrate it,
3 2 3 2
a y x a x y a c x a c x 2
1: x ln(x) - x + ------ + ------ + c x y + ------ + ------ + x c
3 2 b 3 2 b
.
That's just awesome! That's a text editor ... doing calculus!
So, that was most of the main features. It was kind of exhausting
going through all of that, and I am only scratching the surface of
what the calculator can do.
Naturally, it can be extended with some elisp. It provides a
defmath macro specifically for this.
I bet (hope?) someday it will have a functions for doing Laplace and
Fourier transforms.
Update: It looks like all these wishes, except the last one,
may actually be coming
true!
Guile can run Elisp better than Emacs! The idea is that the Elisp
engine is replaced with Guile — the GNU project's Scheme
implementation designed to be used as an extension language — and
written in Scheme is an Elisp compiler that targets Guile's VM. The
extension language of Emacs then becomes Scheme, but Emacs is still
able to run all the old Elisp code. At the same time Elisp itself,
which I'm sure many people will continue to use, gets an upgrade of
arbitrary precision, closures, and better performance.
I've been using elisp a lot lately, but unfortunately it's missing a
lot of features that one would find in a more standard lisp. The
following are some features I wish elisp had. Many of these could be
fit into a generic "be more like Scheme or Common Lisp". Some of these
features would break the existing mountain of elisp code out there,
requiring a massive rewrite, which is likely the main reason they are
being held back.
Closures, and maybe continuations. Closures are one of the
features I miss the most when writing elisp. They would allow the
implementation of Scheme-style lazy evaluation with delay
and force, among other neat tools. Continuations would
just be a neat thing to have, though they come with a performance
penalty.
Closures would also pretty much require Emacs switch to lexical
scoping.
Arbitrary precision. Really, any higher order language's
numbers should be bignums. Emacs 22 does come with the Calc
package which provides arbitrary precision via
defmath. Perl does something like this with the bignum
module.
Packages/namespaces. Without namespaces all of the Emacs
packages prefix their functions and variables with its name
(i.e. dired-). Some real namespaces would be useful for
large projects.
C interface. This is something GNU Emacs will never have
because Richard Stallman considers Emacs shared libraries support to
be
a GPL threat. If Emacs could be dynamically extended some useful
libraries could be linked in and exposed to elisp.
Concurrency. If some elisp is being executed Emacs will lock
up. This is a particular problem for Gnus. Again, Emacs would really
need to switch to lexical scoping before this could happen. Threading
would be nice.
Speed. Emacs lisp is pretty slow, even when compiled. Lexical
scoping would help with performance (compile time vs. run time
binding).
Regex type. I mention this last because I think this would be
really cool, and I am not aware of any other lisps that do it. Emacs
does regular expressions with strings, which is silly and
cumbersome. Backslashes need extra escaping, for example. Instead, I
would rather have a regex type like Perl and Javascript have. So
instead of,
(string-match "\\w[0-9]+" "foo525")
we have,
(string-match /\w[0-9]+/ "foo525")
Naturally there would be a regexpp predicate for checking
its type. There could also be a function for compiling a regexp from a
string into a regexp object. As a bonus, I would also like to use it
directly as a function,
(/\w[0-9]+/ "foo525")
I think a regexp price would really give elisp an edge, and would be
entirely appropriate for a text editor. It could also be done without
breaking anything (keep string-style regexp support).
I wanted an elisp macro that could measure the running time of a block
of code. Specifically, I wanted it to work like this,
(measure-time
...
body
...)
And it would return the running time as seconds in floating
point. Well, here's a macro that does it!
;; ID: 6a3f3d99-f0da-329a-c01c-bb6b868f3239(defmacromeasure-time(&restbody)"Measure and return the running time of the code block."(declare(indentdefun))(let((start(make-symbol"start")))`(let((,start(float-time))),@body(-(float-time),start))))
It's only good for up to around 18 hours, then the time integer
overflows. If only Emacs had arbitrary precision numbers. Here it is
in action using my binomial function from
last week.
(measure-time(nck2010)(nck307))
Which, just now, returned 3.643713 seconds when executed.
To all other non-emacsen text editors, can your text editor do that?!
Ha! Even though elisp is a slow, closure-less, dynamically scoped,
ugly cousin of more popular lisps, it's still a lot of fun to write.
To fire it up, load it into Emacs and run the extended command
(M-x) httpd-start. By default it will serve
files from "~/public_html". To change this, change the
variable httpd-root to the desired web root. You can stop
the server with httpd-stop.
It's about 200 lines of code and can serve static websites made of
small, static files. I say small files because it serves files from
buffers, meaning it has to read the entire file in first.
For a simple, text editor based server it can hold up to a pretty
decent load. At one point I hit it with 8 wget instances all making
rapid recursive downloads and my manual navigation wasn't slowed down
noticeably. Despite running in the slow elisp interpreter, I think it
can have much better performance by caching commonly served files in
buffers.
It should run, unmodified, anywhere a modern Emacs can run, so
I expect that it's already very portable. I can imagine it being
useful in a situation where someone needs to temporarily host some
files but there isn't a web server on the machine. Just grab this
script and throw it at Emacs.
Well, it only does IPv4 right now, though I expect IPv6 only requires
changing one number (namely, 4 to 6). I don't have any IPv6 systems to
test it on.
When writing it I also had security in mind so, as far as I know, it
should be safe to use. It cleans up the GET from the
client so that no files underneath the serving root can be accessed.
The server log is lisp itself. Here is an example log starting the
server, serving one request, and halting,
The log is alists of alists, making a hierarchical tree structure that
can be explored with some simple lisp functions. Normally this sort of
thing is done with XML, but lisp already has its own structured
format: lists!
When GET is a directory, it looks for
"index.html" and serves that if it exists. More indexes
can be added to the variable httpd-indexes. This can
actually be done in a special ".htaccess.el" file.
If a ".htaccess.el" exists in the directory from which a
file is being served, Emacs will first load/execute it. You see, it's
just a lisp program. If you wanted to add a new index file name, the
hypertext access file could contain this,
(add-to-list'httpd-indexes"0001.html")
It's a bit like a .emacs file.
But I think one of the coolest things about having a lisp-based server
is that the server can be modified in place without disrupting or
restarting it. In my Emacs web server, the only change that requires a
restart is changing the server port. In fact, I wrote most of it while
the server was running and tested my changes from a browser right as I
made them — all on the same instance of the server.
If you want to look into the AI side of this, the server could modify
its own code in response to its use.
I also had the idea of creating dynamic websites with elisp, in the
same way PHP or Perl does. If a .el file (or
.elc) is accessed, the server would pass the
GET/POST arguments as an alist to a function
in the elisp file. The server would also provide some nifty HTML
generation macros. A dynamic script might look like this,
(defunscript(get)(html(head(title"My Script"))(body(h1"Your Query")(p(concat"Your query was "(html-sanitize(cdr(assoc"q"get))"."))))))
However, this is not (yet?) implemented. Just an idea.
I will continue to work on it, though I don't expect to add much more
to it. I will mostly improve the code and documentation.
In latest versions of Cedet support of GNU Global was introduced, and very useful command - semantic-symref, was implemented. It allows to find places in source code (for C & C++ now) where given function is used. And if GTAGS database wasn't found, then this command tries to find occurrences with find-grep command. As result, user gets something like this...
Please note that planet.emacslife.com aggregates blogs, and blog authors might mention or link to nonfree things. To add a feed to this page, please e-mail the RSS or ATOM feed URL to sacha@sachachua.com . Thank you!
The picture above shows that I spent 30 minutes editing and there are 254 words in this section.
I was inspired by an item
in 
However, if you're interested in digging into Common Lisp to try it
out, you may find yourself quickly running into walls just getting
started. It's a lot different than other programming environments you
may be used to. The Common Lisp tutorials generally skip this step,
assuming the user has an environment, or leaving that setup for the
"vendor" to handle. So, here's a guide to setting up a great Common
Lisp environment with 
Org Social
