Irreal: Emacs and MacOS Catalina

Last night I upgraded my MacBook Pro to macOS Catalina (macOS 10.15). I’d been holding off to see if there were reports of problems with Emacs on the new OS. That happened last time with the upgrade to Mojave because Apple stopped supporting the old display model and you couldn’t use Emacs in GUI mode until the Emacs devs generated a fix.

This time, there were reports of Emacs not being able to access (any of) the file system. The solution seemed pretty simple. The TL;DR is that Emacs was being loaded by a Ruby script so it was Ruby that needed to be given permissions. I hadn’t heard of the Ruby script before but assumed it was something new in Emacs 27. When I checked the above Stack Exchange entry, though, it said the poster was seeing the problem in Emacs 26.1.

In any event, after the upgrade Emacs worked just fine without me doing anything. I’m using the same binary (Emacs 26.3 compiled from source) as before so I haven’t tried recompiling Emacs yet but the old binary is working as it was previously without having to adjust permissions or anything else. My conclusion is that there’s no reason not to upgrade as far as Emacs is concerned. If you do experience problems with accessing the file system, take a look at the above link. Every report I saw about the problem said that the solution provided by Stack Exchange got things working again.

Added in editing:
Here’s a tweet that also describes the problem. The Stack Exchange link it contains is different from the one above so you should look at it too if you’re having problems.

-1:-- Emacs and MacOS Catalina (Post jcs)--L0--C0--October 17, 2019 04:31 PM

Grant Rettke: Use all of Emacs’s modifiers on macOS with various keyboards.

When non-Emacs users observe Emacs users laboring over their key binding configuration their typical range of reactions include finding it to be interesting, surprising, remarkable, strange, weird, incredible, striking, bizarre, deviant, eccentric, freakish, monstrous, incomprehensible, inconceivable, incredible, unimaginable, and unthinkable. With that range reactions in mind it is self-evident that if you are reading this … Continue reading "Use all of Emacs’s modifiers on macOS with various keyboards."
-1:-- Use all of Emacs’s modifiers on macOS with various keyboards. (Post grant)--L0--C0--October 17, 2019 03:56 AM

emacspeak: Meta-Programming In Emacs Using Defadvice


Meta-Programming In Emacs Using defadvice

1 Introduction

This blog article Meta Programming In Python reminded me to write up
the equivalent for Emacs Lisp.
Decorators in Python enable you to modify the vehavior of existing
functions without changing the original function; Advice in Lisp
which traces back to the 80's enables similar functionality —
incidentally, Advice was the inspiration behind Aspect Oriented
Programming in Java.


2 Advice And Emacs Lisp

Advice came to Emacs lisp in the summer of 1994, when module
advice.el shipped as part of Emacs 19.23. A few weeks later, a
colleague at work (Win Treese) wrote up a simple example that added
a before advice to the vc-checkin functions of the time to create
an RCS subdirectory if one did not exist. This was such a neat
trick that all of us in our Lab adopted it — and having random
RCS*,v* files lying around in the working directory became history.


3 Emacspeak And Advice — Fall 1994

And then I decided to make Emacs speak to me.
Module advice.el provided the ability to add before, after, or
around advice. In brief, Emacspeak used these three classes of
advice as follows:


  1. Speak line moved to after next-line and previous-line — add
    an after advice that called function emacspeak-speak-line.
  2. Speak character being deleted — Add a before advice that
    speaks the character under point before it is deleted.
  3. Get various types of completion spoken — Add an around advice
    that remembers the current context, calls the original function
    being adviced, and finally speak the current context that now
    reflects the completion.

The rest of the story was written up a few years ago as Emacspeak At
Twenty
. Fast-forwarding 25 years, present versions of Emacs still
include module advice.el as well as an arguably simplified front-end
implemented by module nadvice.el that enables the definition of
custom functions that are then attached to existing code via advice.


4 References

  1. Emacspeak At Twenty, Looking Back, Looking Forward.
  2. emacspeak: The Complete Audio Desktop. Chapter from the book
    entitled Beautiful Code, OReilly.

-1:-- Meta-Programming In Emacs Using Defadvice (Post T. V. Raman (noreply@blogger.com))--L0--C0--October 16, 2019 09:33 PM

Grant Rettke: macOS 12 Sierra Brew Users: Read This Before Updating Anything

When I update Brew I minimize the terminal window so I can work on something else while it is running. Every so often I check to see if it finished. Then I read through the history to see if there is any work I need to perform by hand. For example sometimes Brew can’t link … Continue reading "macOS 12 Sierra Brew Users: Read This Before Updating Anything"
-1:-- macOS 12 Sierra Brew Users: Read This Before Updating Anything (Post grant)--L0--C0--October 16, 2019 07:52 PM

(or emacs: Ivy 0.13.0 is out

Intro

Ivy is a completion method that's similar to Ido, but with emphasis on simplicity and customizability.

Overview

The current release constitutes of 183 commits and 3 months of progress since 0.12.0. Many issues ranging from #2153 to #2278 were fixed. The number of people who contributed code as grown to form 148 to 160. Thanks, everyone!

Details on changes

Changelog.org has been a part of the repository since 0.6.0, you can get the details of the current and past changes:

Highlights

Many improvements are incremental and don't require any extra code to enable. I'll go over a few selected features that require a bit of information to make a good use of them.

New bindings

  • counsel-find-file
    • ~~ to move to the local home directory from remote or /sudo::.
    • / RET ~ achieves the same thing, but is longer.
    • M-o R calls find-file-read-only.
  • counsel-git-grep
    • C-x C-d to switch the current directory.
  • swiper-isearch
    • M-o w to copy the current line and go back to where you started.

New features

counsel-package

The idea of counsel-package is to install and remove packages with a single binding:

(global-set-key (kbd "C-c P") 'counsel-package)

But if the package contents weren't up to date, a separate M-x package-refresh-contents had to be triggered, which was an annoying overhead. Now counsel-package will look at the time stamps of the appropriate archive-contents files, and call package-refresh-contents if the timestamp is outdated by more than 4 hours.

counsel-M-x

Some commands are intended to be called only via their key binding. Make them disappear from counsel-M-x like this:

(put 'counsel-find-symbol 'no-counsel-M-x t)

counsel-rg

The default setting of ivy-case-fold-search-default is 'auto, which means:

  • the input "emacs" matches "emacs", "EMACS", and "Emacs"
  • the input "Emacs" matches only "Emacs"

This now also applies to counsel-rg: Ivy will pass the -i flag to ripgrep appropriately, based on ivy-case-fold-search-default. You should remove the -S flag from counsel-rg-base-command if you customized it.

ivy-update-candidates

This is a new API for asynchronous calls. To use it, pass to ivy-read: :dynamic-collection t, and a collection function that takes a user input string, starts some asynchronous process based on that input, and returns 0. The 0 return result tells Ivy that no candidates were returned; instead, ivy-update-candidates is used in the async callback.

See counsel-google for a reference implementation.

ivy-use-virtual-buffers

You can now choose between: recent files, or bookmarks, or both, or none. Don't forget that counsel-set-variable makes it very easy to set customization options.

New Commands

I have put these separately so they don't get lost in the crowd. Be sure to try them out.

  • counsel-buffer-or-recentf - list buffers visiting files (highlighted) then the recentf file list.
  • counsel-fonts - show a list of all supported font families for a particular frame.
  • counsel-google - asynchronously query the Google predictive search API.
  • counsel-major - switch major-mode.
  • counsel-slime-repl-history - browse SLIME REPL history.

Outro

Again, thanks to all the contributors. Happy hacking!

PS. Thanks to everyone who supports me on Liberapay and Patreon!

I am now also on Github sponsors, which is an interesting new effort by Github. The cool thing is that's more integrated with Github, there are less transaction fees, and Github matches every donation for up to $5000 for a whole year. Please consider joining there, since every $1 per month that you donate is doubled by Github.

-1:-- Ivy 0.13.0 is out (Post)--L0--C0--October 15, 2019 10:00 PM

sachachua: 2019-10-14 Emacs news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, planet.emacslife.com, YouTube, the Emacs NEWS file and emacs-devel.

-1:-- 2019-10-14 Emacs news (Post Sacha Chua)--L0--C0--October 15, 2019 02:55 PM

Irreal: Generating Slides and a PDF From the Same Org File

Karl Voit has a revealing post in his UOMF series that explains how to generate a slide deck and supporting PDF from the same Org file. That has some obvious benefits, not least that it provides a single source of truth so you don’t have to worry about the slides and PDF getting out of sync.

It turns out to be pretty easy to mix LaTeX and HTML source in the same file without having them interfere with each other. For example, anything in a NOTES block will appear only in the PDF, not on the slides.

One of the interesting things about Voit’s method is that he likes to maintain just a few Org files that cover all his work so the slides and PDF are just one node of a large Org file that contains other material. If you like that idea, be sure to take a look at his post to see how he does it.

If your work requires you to give presentations and you like to use slides accompanied by a PDF for a handout that fills in the blanks, be sure to read Voit’s post. It really makes sense to keep everything together in a single file and Voit shows you how to do that.

-1:-- Generating Slides and a PDF From the Same Org File (Post jcs)--L0--C0--October 14, 2019 03:24 PM

Saurabh Kukade: Meta-programming In Python

“Perfection is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” – Antoine de Saint-Exupery Recently I encountered with very fascinating concept which is “Meta-programming in Python”. I would like to share my findings about this topic in this article....
-1:-- Meta-programming In Python (Post)--L0--C0--October 13, 2019 06:30 PM

Timo Geusch: [HOWTO] Installing Emacs 26.3 on Ubuntu or XUbuntu 19.04

My previous instructions for installing a newer Emacs version on Ubuntu still work. Ubuntu (and in my case, XUbuntu) 19.04 ships with Emacs 26.1 out of the box. As usual I want to run the latest version – Emacs 26.3 Read More

The post [HOWTO] Installing Emacs 26.3 on Ubuntu or XUbuntu 19.04 appeared first on The Lone C++ Coder's Blog.

-1:-- [HOWTO] Installing Emacs 26.3 on Ubuntu or XUbuntu 19.04 (Post Timo Geusch)--L0--C0--October 13, 2019 04:19 PM

Marcin Borkowski: Challenge accepted - a Node.js grep

An old friend of mine posted a challenge on Twitter to implement a grep-like utility in one’s language of choice. Instead of complaining that he’s got an unfair advantage – he is a Pythonista, and Python is almost as well-suited to that kind of tasks as Perl – I decided to accept the challenge. Of course, I had to start with Emacs Lisp.
-1:-- Challenge accepted - a Node.js grep (Post)--L0--C0--October 12, 2019 11:58 AM

sachachua: 2019-10-07 Emacs news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, planet.emacslife.com, YouTube, the Emacs NEWS file and emacs-devel.

-1:-- 2019-10-07 Emacs news (Post Sacha Chua)--L0--C0--October 08, 2019 05:44 PM

Manuel Uberti: Straight ahead

I have never found Emacs appealing without external packages. From the colour theme to the best Git UI one could ask for, I can’t imagine using Emacs without external packages. The built-in package.el has served me well for a long time. Configured with the usual package archives, it has offered me easy access to new packages and updates for the currently installed ones.

However, digging a little deeper reveals some of its shortcomings. For instance, downgrading has to be done manually, because package.el gives access only to the latest version of a package. This also means it is impossible to get a specific version of said package without getting the source and loading it by yourself.

As you install and configure more and more tools to enhance your Emacs experience, you are adding layers of complexity. Dependencies are pulled in and new releases could bring breaking changes: either you never upgrade your packages, or you learn to live on the bleeding edge. I love that edge as much as your next Emacs hacker, but to approach it with steadiness I need more control over the packages I use.

This is where straight.el fits in. I have to admit that Radon Rosborough played dirty with my feelings: by describing straight.el as purely functional he knew perfectly well a developer in love with functional programming could not resist. But there is more to it, of course.

First of all, I love the documentation. The effort put into explaining the rationale behind it, the comparisons with other package managers, and the basics to get started is simply amazing. Even the process of reproducing bugs is documented. It may seem like a lot to take in, but it’s clear and it’s something I want to know before changing one of the key features of my text editor.

Moreover, it takes a few lines of code to bootstrap straight.el. It would probably take fewer lines to set up package.el, that’s true, but straight.el is far more customisable. It also works seamlessly with use-package, so I didn’t have to reinvent my Emacs configuration just to try a new package manager.

Finally, you can freeze your package state at any time with straight-freeze-versions. This creates a lockfile with the current versions of your packages, meaning you can reproduce your exact configuration with a simple straight-thaw-versions. Stability is just a couple of commands away.

There must (should?) be a moment when the Emacs hacker knows they exploited their beloved editor as much as possible, the eureka light bulb leading the way through a thick fog of setq and defun. Valhalla is there, Emacs hacker, you know it. As for me, I decided to accept the murkiness for now, and look for the bulb that will get me out of here eventually another time.

-1:-- Straight ahead (Post)--L0--C0--October 04, 2019 12:00 AM

Anything goes: Fast Scrolling in Emacs

I work with some files that have some complex fontification rules. There is a significant lag when scrolling quickly through these files. Although I am using jit-lock-mode as my font-lock-support-mode, the lag is unbearable. One solution to this is to use fast-scroll. https://github.com/ahungry/fast-scroll. fast-scroll temporarily disables font-lock when scrolling. This has worked out nicely for me.
-1:-- Fast Scrolling in Emacs (Post tsengf (noreply@blogger.com))--L0--C0--October 03, 2019 07:26 AM

Raimon Grau: Migrating from vim (proficent) to emacs

In HN's thread about 26.3 being released (Congrats!), there's this guy  explaining that being already quite confortable with vim, it's too much of a commitment to move to emacs.

I remember the frustration when coming to emacs from an advanced vimmer POV: no "yyp"? no ":%s/foo/bar"? imap? But here's what I answered him:

As this is not an overnight conversion and you are quite proficent with vim already, my advice is to:
- Get used to type "emacs file.txt" instead of "vim file.txt" in your console.
- Have a function in emacs that opens the current file in vim for those moments where you just want your trusted environment. Writing it by yourself is a good focused learning experience.
This way you'll decide (and balance) how much you want to learn every day, and little by little you'll find yourself using that function less and less.

And I think this goes very well with my other "bootstrap your emacs lisp learning".

It's the same approach to most of my development (and life) efforts.  It's a function of how bad do you need it, how fast do you want it, the compound interest of starting early, and how much it slows you down (or blocks you for doing other things).
-1:-- Migrating from vim (proficent) to emacs (Post Raimon Grau (noreply@blogger.com))--L0--C0--September 03, 2019 12:06 AM

Chen Bin (redguardtoo): Aspell 0.60.8 will have direct support for camelCase words

Kevin Atkinso told me this good news.

Currently I'm using Emacs Lisp to check camel case words.

The new option --camel-case from aspell will definitely speed up the whole process.

Minimum Emacs setup,

(setq ispell-program-name "aspell")
(setq-default ispell-extra-args '("--sug-mode=ultra"
                                  "--lang=en_US"))
;; Make sure new aspell is installed
(when (string-match-p "--camel-case"
                      (shell-command-to-string (concat ispell-program-name " --help")))
  (push "--camel-case" ispell-extra-args))

Optionally, you could read What's the best spell check setup in emacs.

-1:-- Aspell 0.60.8 will have direct support for camelCase words (Post Chen Bin)--L0--C0--September 01, 2019 01:13 PM

Marcin Borkowski: A simple tip with overlays and diffs

A few days ago I had an interesting problem. I had to resolve a particlarly nasty Git merge. It involved several files with lines’ lengths in the triple digits and a fair number of very small changes. Seeing those changes (in smerge-mode), even after refining the diffs, was tricky – there were many very small patches (sometimes two, sometimes four characters) of changed text and I was quite afraid that I would miss some of them. I searched for a command to “go to the next patch of changes”, but to no avail. Then I decided to write my own.
-1:-- A simple tip with overlays and diffs (Post)--L0--C0--August 31, 2019 11:17 AM

Manuel Uberti: A pen and a notebook

Cal Newport’s Digital Minimalism has really changed my perspective on a lot of things in my life. Since I started my decluttering process I’ve noticed several benefits. In particular, I’m less prone to distractions and leisure has become time for engaging activities.

This new approach has prompted me to reconsider how I organise my days, weeks, and months. For a couple of years I’ve been using my custom variation of GTD, tracking everything with Org files in Emacs, and using Syncthing and Orgzly to keep it all synced with my phone. However, as I already explained when I wrote about Cal Newport’s book, my smartphone is not my best friend, and so the whole setup is actually of little use. And let’s be frank: sometimes I can leave Emacs alone.

I also write a lot, everyday. I write about films, note my ideas on the music I listen to and the books I read, and I write for this blog too. Hence I need a place which can be useful both for organising my life and for my basic urge to write. Enter Bullet Journal.

I recently rediscovered the pleasure of pen and paper when in Leuven for Heart of Clojure. Since I left the laptop at home on purpose, I spent almost every spare moment on my notebook, and much of the article on the conference came straight from its pages. As Ryder Carroll, the man behind Bullet Journal, points out in his book, the connection our brain establishes with pen and paper is completely different from the one with keyboard and monitor. For me it’s mostly been a feeling I sensed when writing on my notebook. My thoughts just came out with a spontaneity I’ve rarely enjoyed with the digital counterparts.

I won’t bother you with the details about Bullet Journal. The website has tutorials, videos, there is a book about it, so everything you need to know is out there. I picked this system because I find it simple and clear, modular and practical. Starting next month, I am going to fully migrate my GTD setup to Bullet Journal and once again step back to distance myself from pervasive technology consumption.

-1:-- A pen and a notebook (Post)--L0--C0--August 25, 2019 12:00 AM

Rubén Berenguel: SyncTeX and pdf-view-mode for emacs

Or, destiny is cruel
Back in the days of yore, when I was switching between my Windows machine and a Linux machine, I remember having SyncTeX active in my Windows machine. It was a wonderful experience: SyncTeX lets you click anywhere on a generated file from LaTeX and gets back to your editor, to the place generating where you clicked. This was extremely useful, specially later on when you need to adjust many formulas to fit and you need a bit of back-and-forth-ing.

Then I got a Mac, and since Preview is so handy I slowly forgot about SyncTeX. Time went on, and I merrily kept on editing LaTeX files as usual. I even managed to deliver my PhD dissertation a couple weeks ago, the formal speech will be in a month or two (come at your own risk). AucTeX’s preview saved most of the days, so I slowly even forgot SyncTeX existed. Shame on me indeed.

The other day I got an annotated PDF from one of my advisors, and I just couldn’t open the annotations. I tried all programs I had for Mac, and no luck: annotations weren’t showing, just saw the icons. Surveying for some command-line tool to extract annotations (just in case) I found pdf-tools, a replacement for DocView based on Poppler. It had the awesome ability of actually displaying annotations, with it it was pretty clear the annotations were broken in that PDF. I got a new set of PDFs from my advisor with the annotations in place, though. While waiting for it to arrive…

I saw SyncTeX was an option of pdf-tools. I had been using that, hadn’t I? So, I activated SyncTeX in AucTeX (it is TeX-source-correlate-method, see here) and indeed: I could have two frames, one with the actual LaTeX sources and the other with a PDF, and go from one to the other. Even hyperreferences in PDF work! See (well, click on the full-screen mode to see it larger or you won't see anything)!



Getting pdf-tools to work wasn’t incredibly tricky (given the hoops you need for some packages, sometimes). Just

brew install pdf-tools

and after reading

brew info pdf-tools

I was told to run

  emacs -Q --batch --eval "(package-install-file 
\"/usr/local/Cellar/pdf-tools/0.60/pdf-tools-0.60.tar\")"

and this does the trick (well, change emacs for your actual emacs, which likely is /Applications/Emacs.app/Contents/MacOS/Emacs) You’ll also need to add to your .emacs file (or temporarily in your *scratch* buffer)


(setenv "PKG_CONFIG_PATH"

(concat

"/usr/local/Cellar/zlib/1.2.8/lib/pkgconfig"

":"

"/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig"))


(getenv "PKG_CONFIG_PATH")


and run

(pdf-tools-install)

as advised in the package’s README. And that's it, open a PDF and activate pdf-view-mode to check everything is in place. Well worth it!
-1:-- SyncTeX and pdf-view-mode for emacs (Post Ruben Berenguel (noreply@blogger.com))--L0--C0--August 15, 2019 09:15 AM

Sanel Zukan: ido-mode with eshell, shell, sql and more...

ido-mode doesn't provide completion for eshell, shell or sql-mode out of the box, and I wasn't been able to find any package with this option that is lightweight enough (Helm, I'm looking at you). So, after few attempts, here is the code for that:

(defun ido-eshell-comint-history ()
  "eshell & comint history with ido."
  (interactive)
  (if (or (member major-mode '(eshell-mode sql-interactive-mode))
          (derived-mode-p 'comint-mode))
     (let ((ring (if (eq major-mode 'eshell-mode)
                   eshell-history-ring
                   comint-input-ring)))
       (insert
        (ido-completing-read "History: "
                             (delete-dups
                              (ring-elements ring)))))
    (message "Unsupported mode")))

After evaluating the code, in eshell simply execute M-x ido-eshell-comint-history and magic will happen - you will get full eshell history managed by ido-mode. To make it more authentic, map it to C-c C-l, which is default sequence for completion in eshell.

The same can be applied for shell and sql-mode as well.

But, the things gets interesting here: (ido-eshell-comint-history) will pick up history for all modes inherided from comint-mode (shell is using comint-mode), so you will get history completion for python, lua, lisp and other inferior processes for free!

Just to add (commenting above code), although sql-mode is using comint-mode, it doesn't derive it so additional check is necessary. Also to enable history in sql-mode you will need to set variable sql-input-ring-file-name as well.

-1:-- ido-mode with eshell, shell, sql and more... (Post)--L0--C0--August 06, 2019 10:00 PM

Shae Erisson: Open Source Hardware Hearing Aid Part 1

Why pay thousands for a hearing aid when you could pay $300?
-1:-- Open Source Hardware Hearing Aid Part 1 (Post Shae Erisson)--L0--C0--August 03, 2019 12:00 AM

Alex Schroeder: Use of Emacs

Two Years With Emacs as a CEO (and now CTO) is the followup to A CEO's Guide to Emacs. The author talks about life with Emacs.

So many cool quotes in that article!

On using a tool that isn’t designed to do a task (an application) but on using a tool help you do your task. Emacs is a tool, not an app.

Emacs is a portal into the power of the computer itself, and that is a rabbit hole worth descending. Its idioms are paths to discovering and creating your own, and that for me is the definition of creativity. One of the sad things about modern computing is that it is largely made up of black boxes with shiny interfaces that provide momentary gratification rather than real satisfaction. This makes us into consumers rather than creators/makers of technology. I don’t care who you are or what your background is; you can understand your computer, and you can make things with it.

On not using Emacs for shared todo items and calendar stuff:

I gave up on using Org-mode for to dos and the like, as I have to coordinate many meetings and calls every day with dozens of people, and I cannot ask the rest of the world to adapt to my choice of tools, nor do I have the time to transcribe or automate moving things to Org.

On not using Emacs for takin notes during meetings because it’s often better to use a pen (unless the point of the meeting is to write a document).

I also use a plain old pen for note-taking during meetings, as I find laptop/keyboard use in meetings to be rude and limiting to my ability to listen and think.

Ah yes. At work, I usually joke that Emacs is just my IRC client... 🙂

Tags:

-1:-- Use of Emacs (Post)--L0--C0--August 02, 2019 10:09 PM

emacsninja: Code Conversion Language

Update: I forgot that I did a brief analysis on this many years ago, using ROT13 as example.

Update: Noam Postavsky pointed out on #emacs that CCL is not turing-complete after all as a full simulation (as opposed to just interpreting a single line) requires an Emacs Lisp loop. This loop cannot be done in CCL itself as it doesn’t allow feeding its output back in as input. The I/O restrictions most likely put it into the weaker category of finite-state transducers.

Emacs is most famously, a re-imagination of a Lisp machine, with the Emacs Lisp byte-code interpreter being at its core. A lesser-known fact is that there’s two more byte-code interpreters in its C sources, one for compiled regular expressions and another designed for encoding and decoding text, known as Code Conversion Language (CCL). This blog post will focus on the latter as it’s largely gone unnoticed and hasn’t seen too much experimentation.

The CCL implementation is split into the byte-code interpreter (ccl.c) and compiler (ccl.el) parts. There is no official documentation other than comments and docstrings found in these files. From this I’ve learned that CCL programs are represented as integer vectors and that there’s a higher-level language compiling to them, described in the ccl-define-program docstring. By reading that information I’ve deduced the following:

  • The VM has eight integer-sized registers r0 to r7 and an instruction counter ic
  • Register r7 is used as a status register and may be clobbered at any time by an arithmetic operation
  • A CCL program can either be run on a string and return a string, alternatively it can be run standalone for side effects
  • The former mode requires you to provide a nine-element status vector representing the registers and instruction counter, the latter an eight-element status vector representing the registers only
  • As a side-effect, the status vector contains the new state of the registers and instruction counter after executing the program
  • The VM supports the standard C arithmetic, comparison and assignment operators
  • The language translates several control flow statements to equivalent goto statements, such as if, branch (look-up table) and loop with repeat inside
  • Statements are grouped by surrounding them with parentheses
  • When operating on a string, they are read in and written out in a serial fashion, no random access whatsoever
  • It’s possible to do a look-up on an array, translation table or hash table
  • There is a call operator, but no stack to save/restore arguments to/from, so you’ll have to come up with a calling convention fitting the available registers
  • Each CCL program specifies a magnification factor which determines the ratio between output and input string size

Armed with that knowledge I wrote some boiler plate code for experimentation:

;; -*- lexical-binding: t; -*-

(require 'ccl)

(defvar ccl-status (make-vector 8 0))
(defvar ccl-status+ic (make-vector 9 0))

(defun ccl-init-status (status args)
  (let ((i 0))
    (fillarray status 0)
    (dolist (arg args)
      (aset status i arg)
      (setq i (1+ i)))))

(defun ccl-run (program string &rest args)
  (let ((status ccl-status+ic))
    (ccl-init-status status args)
    (ccl-execute-on-string program status string)))

(defun ccl-run-pure (program &rest args)
  (let ((status ccl-status))
    (ccl-init-status status args)
    (ccl-execute program status)
    status))

There will be some benchmark numbers, none of these are to be taken seriously. Do your own benchmarks before using mine for decisions.

Hello World!

For starters I’ll focus on processing strings. The easiest possible program that still does something useful reads in output and writes it out as is:

(define-ccl-program ccl-identity
  '(1
    (loop
     (read r0)
     (write r0)
     (repeat))))

(ccl-run 'ccl-identity "Hello World!") ;=> "Hello World!"

Let’s go through that program carefully. The S-Expression starts with a magnification factor of 1, meaning that the output buffer should be as large as the input buffer. If it were zero, no I/O would be permitted in the first place, whereas a factor greater than one would allocate enough space to produce a string larger than the input.

The magnification factor is followed by a s-expression that’s executed until it’s done or an error occurred, such as there being no more input. It may be followed by another s-expression that’s executed after the main one, no matter whether it failed with an error or not.

ccl-identity uses a pattern that will come up a few more times in this blog post. It enters a loop, reads a character into the r0 register, writes out a character from the r0 register and jumps to the beginning of the loop. If there are no more characters left, the read operation fails and terminates the loop. Let’s spice things up by adding an extra processing step before writing out the character:

(define-ccl-program ccl-xor
  '(1
    (loop
     (read r1)
     (r1 ^= r0)
     (write r1)
     (repeat))))

(ccl-run 'ccl-xor "Secret" 42) ;=> "yOIXO^"
(ccl-run 'ccl-xor "yOIXO^" 42) ;=> "Secret"

XOR is the bread and butter operator in modern cryptography. A text can be encrypted by replacing each character with the result of XORing it against a secret byte, similarly it can be decrypted by applying the same transformation again. To pass the secret byte as an argument, I’ve placed it in the r0 register and read the string into the r1 register instead. On each iteration of the loop r1 is set to r1 ^ r0 and written out again.

More on translation

In the real world translating characters isn’t as simple as applying some arithmetic to them. Suppose I wanted to challenge the upcase built-in:

(define-ccl-program ccl-upcase
  '(1
    (loop
     (read r0)
     (if (r0 >= ?a)
         (if (r0 <= ?z)
             (r0 -= 32)))
     (write r0)
     (repeat))))

The processing step is a bit more involved this time. If the read-in character appears to be between the a and z characters, transform it by subtracting 32. Why 32? Take a look at an ASCII table and you’ll see that this is the distance between uppercase and lowercase letters. Unfortunately this implementation cannot challenge upcase as it fails to translate non-ASCII characters correctly and is slower than the real deal:

(ccl-run 'ccl-upcase "Hello World!") ;=> "HELLO WORLD!"
(ccl-run 'ccl-upcase "Mötley Crüe") ;=> "MöTLEY CRüE"
(benchmark 100000 '(ccl-run 'ccl-upcase "Hello World!"))
;; => "Elapsed time: 0.165250s (0.072059s in 1 GCs)"
(benchmark 100000 '(upcase "Hello World!"))
;; => "Elapsed time: 0.119050s (0.072329s in 1 GCs)"

Let’s try again with a different text transformation where I actually have a chance to win, ROT13:

(define-ccl-program ccl-rot13
  '(1
    (loop
     (read r0)
     (if (r0 >= ?a)
         (if (r0 <= ?z)
             ((r0 -= ?a)
              (r0 += 13)
              (r0 %= 26)
              (r0 += ?a))))
     (if (r0 >= ?A)
         (if (r0 <= ?Z)
             ((r0 -= ?A)
              (r0 += 13)
              (r0 %= 26)
              (r0 += ?A))))
     (write r0)
     (repeat))))

This time the program needs to recognize two different character ranges to process, lowercase and uppercase ASCII characters. In either case they’re translated to their position in the alphabet, rotated by 13, then translated back to ASCII again. Surprisingly enough, this is enough to beat both rot13-string and rot13-region:

(ccl-run 'ccl-rot13 "Hello World!") ;=> "Uryyb Jbeyq!"
(ccl-run 'ccl-rot13 (ccl-run 'ccl-rot13 "Hello World!"))
;; => "Hello World!"
(benchmark 100000 '(ccl-run 'ccl-rot13 "Hello World!"))
;; => "Elapsed time: 0.248791s (0.072622s in 1 GCs)"
(benchmark 100000 '(rot13-string "Hello World!"))
;; => "Elapsed time: 6.108861s (2.360862s in 32 GCs)"
(with-temp-buffer
  (insert "Hello World!")
  (benchmark 100000 '(rot13-region (point-min) (point-max))))
;; => "Elapsed time: 1.489205s (1.017631s in 14 GCs)"

I then tried to use translation tables for a final example of a “Vaporwave” converter, but failed. Funnily enough this mirrors my overall experience with Emacs, it’s easy to write fun things, but the moment one tries to write something useful, you discover it’s not fun and sometimes not even up to the task. At least it’s possible to salvage the translation tables and use them with translate-region instead, the built-in used by rot13-string and rot13-region:

(defvar ccl-vaporwave-table
  (make-translation-table-from-alist
   (cons '(?\s . 12288)
         (mapcar (lambda (i) (cons i (+ i 65248)))
                 (number-sequence 33 126)))))

(defun vaporwave-it (string)
  (with-temp-buffer
    (insert string)
    (translate-region (point-min) (point-max) ccl-vaporwave-table)
    (buffer-string)))

(vaporwave-it (upcase "aesthetic")) ;=> "AESTHETIC"

Edging towards general-purpose computing

All examples so far have worked on text. If you limit yourself to numbers, you can solve some basic arithmetic problems. Here’s a classic, calculating the factorial of a number:

(define-ccl-program ccl-factorial
  '(0
    ((r1 = 1)
     (loop
      (if r0
          ((r1 *= r0)
           (r0 -= 1)
           (repeat)))))))

(defun factorial (n)
  (let ((acc 1))
    (while (not (zerop n))
      (setq acc (* acc n))
      (setq n (1- n)))
    acc))

While the regular version is more concise, the logic is nearly the same in both. Here’s some numbers:

(aref (ccl-run-pure 'ccl-factorial 10) 1) ;=> 3628800
(factorial 10) ;=> 3628800
(benchmark 100000 '(ccl-run-pure 'ccl-factorial 10))
;; => "Elapsed time: 0.069063s"
(benchmark 100000 '(factorial 10))
;; => "Elapsed time: 0.080212s"

This isn’t nearly as much of a speed-up as I’ve hoped for. Perhaps CCL pays off more when doing arithmetic than for looping? Another explanation is that the Emacs Lisp byte-code compiler has an edge over CCL’s rather simple one. Here’s a more entertaining example, printing out the lyrics of 99 Bottles of Beer on the Wall:

(define-ccl-program ccl-print-bottle-count
  '(1
    (if (r0 < 10)
        (write (r0 + ?0))
      ((write ((r0 / 10) + ?0))
       (write ((r0 % 10) + ?0))))))

(define-ccl-program ccl-99-bottles
  '(1
    (loop
     (if (r0 > 2)
         ((call ccl-print-bottle-count)
          (write " bottles of beer on the wall, ")
          (call ccl-print-bottle-count)
          (write " bottles of beer.\n")
          (write "Take one down and pass it around, ")
          (r0 -= 1)
          (call ccl-print-bottle-count)
          (write " bottles of beer on the wall.\n\n")
          (repeat))
       ((write "2 bottles of beer on the wall, 2 bottles of beer.\n")
        (write "Take one down and pass it around, 1 bottle of beer on the wall.\n\n")
        (write "1 bottle of beer on the wall, 1 bottle of beer.\n")
        (write "Take one down and pass it around, no more bottles of beer on the wall.\n\n")
        (write "No more bottles of beer on the wall, no more bottles of beer.\n")
        (write "Go to the store and buy some more, 99 bottles of beer on the wall.\n"))))))

(defun 99-bottles ()
  (with-output-to-string
    (let ((i 99))
      (while (> i 2)
        (princ (format "%d bottles of beer on the wall, %d bottles of beer.\n" i i))
        (princ (format "Take one down and pass it around, %d bottles of beer on the wall.\n\n" (1- i)))
        (setq i (- i 1))))
    (princ "2 bottles of beer on the wall, 2 bottles of beer.\n")
    (princ "Take one down and pass it around, 1 bottle of beer on the wall.\n\n")
    (princ "1 bottle of beer on the wall, 1 bottle of beer.\n")
    (princ "Take one down and pass it around, no more bottles of beer.\n\n")
    (princ "No more bottles of beer on the wall, no more bottles of beer.\n")
    (princ "Go to the store and buy some more, 99 bottles of beer on the wall.\n")))

This example shows a few more interesting things, generating text of unknown length is rather hard, so I’m using the standard magnification factor of 1 and estimate how big the buffer will be to create an appropriately sized input string. call is useful to not repeat yourself, at the cost of having to carefully plan register usage. Printing out the bottle count can be done if you’re limiting yourself to whole numbers up to 100, a generic solution is going to be hard without random access to the output string. The performance numbers for this one are somewhat surprising:

(let ((input (make-string 15000 ?\s)))
  (benchmark 1000 '(ccl-run 'ccl-99-bottles input 99)))
;; => "Elapsed time: 0.301170s (0.217804s in 3 GCs)"
(benchmark 1000 '(my-99-bottles))
;; => "Elapsed time: 1.735386s (0.507231s in 7 GCs)"

This doesn’t make much sense. Is using format that expensive? It’s hard to tell in advance whether CCL will make a noticable difference or not.

But is it Turing-complete?

My experimentation so far left me wondering, is this language turing-complete? You can perform arithmetics, there’s goto, but the I/O facilities, amount of registers and memory access are limited. The easiest way of proving this property is by implementing another known turing-complete system on top of your current one. I researched a bit and found the following candidates:

  • Brainfuck: A classic, however it requires writable memory. Registers could be used for this, but you don’t have many to play with. You’d need the branch instruction to simulate the data pointer.
  • subleq: Implementing subleq looks easy, but suffers from the same problem as Brainfuck, it requires you to modify an arbitrary memory location. I’ve found a compiler from a C subset to subleq that generates code operating beyond the handful of registers, so that’s not an option either.
  • Rule 110: It’s basically Game of Life, but one-dimensional and can be implemented in a serial fashion. With some tricks it doesn’t require random access either. The proof of it being turing-complete looks painful, but whatever, I don’t care. It’s perfect. There are more elementary cellular automata, so I’ll try to implement it in a generic fashion and demonstrate it on Rule 90 which produces the Sierpinski triangle.
(defmacro define-ccl-automaton (n)
  (let ((print-sym (intern (format "ccl-rule%d-print" n)))
        (rule-sym (intern (format "ccl-rule%d" n))))
    `(progn
       (define-ccl-program ,print-sym
         '(1
           ((r4 = 0)
            (if (r0 == ?1)
                (r4 += 4))
            (if (r1 == ?1)
                (r4 += 2))
            (if (r2 == ?1)
                (r4 += 1))
            (branch r4
                    (write ,(if (zerop (logand n 1)) ?0 ?1))
                    (write ,(if (zerop (logand n 2)) ?0 ?1))
                    (write ,(if (zerop (logand n 4)) ?0 ?1))
                    (write ,(if (zerop (logand n 8)) ?0 ?1))
                    (write ,(if (zerop (logand n 16)) ?0 ?1))
                    (write ,(if (zerop (logand n 32)) ?0 ?1))
                    (write ,(if (zerop (logand n 64)) ?0 ?1))
                    (write ,(if (zerop (logand n 128)) ?0 ?1))))))
       (define-ccl-program ,rule-sym
         '(1
           ((r6 = ,n)
            (r0 = 0)
            (read r1)
            (read r2)
            (loop
             (call ,print-sym)
             (read r3)
             (r0 = r1)
             (r1 = r2)
             (r2 = r3)
             (repeat)))
           ((r0 = r1)
            (r1 = r2)
            (r2 = r5)
            (call ,print-sym)))))))

(define-ccl-automaton 30)
(define-ccl-automaton 90)
(define-ccl-automaton 110)

(defun ccl-sierpinski ()
  (with-output-to-string
    (let ((line "0000000001000000000"))
      (dotimes (_ 20)
        (princ line)
        (terpri)
        (setq line (ccl-run 'ccl-rule90 line))))))

The macro may look scary, but all it does is defining two CCL programs. What an elementary cellular automaton does is looking at the two cells around the current cell, map them to a cell depending to a rule and emit it. There are two edge cases with this for the first and last cell, in my implementation the first cell assumes the previous one was a zero and the last cell uses the first cell. Since there’s no random access, it’s stored into a spare register at the beginning and accessed in a S-Expression after the main loop terminated due to no more input. The surrounding and current cell are stored in three registers and rotated every time a new cell is read in. The mapping is done in the print program by summing up the ones and zeroes, then using the branch instruction to apply the rule to it. If you find this hard to follow, here’s an Emacs Lisp version of it using random access and less limited arithmetic to do the job:

(defun rule--evolve (prev cur next n)
  (let ((acc (+ (if (= prev ?1) 4 0)
                (if (= cur ?1) 2 0)
                (if (= next ?1) 1 0))))
    (if (zerop (logand n (ash 1 acc))) ?0 ?1)))

(defun rule-evolve (line n)
  (let ((out (make-string (length line) ?0)))
    (dotimes (i (length line))
      (cond
       ((zerop i)
        (aset out i (rule--evolve ?0 (aref line i) (aref line (1+ i)) n)))
       ((= i (1- (length line)))
        (aset out i (rule--evolve (aref line (1- i)) (aref line i) (aref line 0) n)))
       (t
        (aset out i (rule--evolve (aref line (1- i)) (aref line i) (aref line (1+ i)) n)))))
    out))

(defun sierpinski ()
  (with-output-to-string
    (let ((line "0000000001000000000"))
      (dotimes (_ 20)
        (princ line)
        (terpri)
        (setq line (rule-evolve line 90))))))

One more benchmark run, this time with less surprising performance numbers:

(benchmark 1000 '(ccl-sierpinski))
;; => "Elapsed time: 0.365031s (0.071827s in 1 GCs)"
(benchmark 1000 '(sierpinski))
;; => "Elapsed time: 0.545512s (0.071829s in 1 GCs)"

If you want to see it in action, try evaluating (progn (princ (sierpinski)) nil) in M-x ielm.

Now for a big letdown, despite everything what I’ve demonstrated, this system is not turing-complete after all. While it’s capable of processing a single line of input, the proof of Rule 110 being turing-complete relies on feeding its output in as input over and over again, however that part has been done in Emacs Lisp as it’s impossible to do in CCL. I’m not 100% sure what CCL’s computing power is, Noam Postavsky suggested on #emacs that it’s most likely a finite-state transducer.

Final words

You may ask yourself now whether you should rewrite all of your slow code to CCL programs. I don’t believe that’s the way to go for a number of reasons:

  • Speedups vary wildly, somewhere between -40% to 450%. There’s no obvious way of predicting whether the optimization is worth it.
  • It’s hard to write an equivalent program or sometimes even impossible, especially if it requires you to use many variables or random access read/write operations.
  • It’s hard to debug a CCL program. While the compiler does a good job at catching syntax errors, runtime errors are far harder to figure out if you can only stare at the registers. Maybe a debugger could be built that uses the “continue” facility and instruction counter register…
  • It’s hard to maintain a CCL program. Not to mention, there’s hardly people who know how to write them. Most of the examples I’ve found online do text encoding/decoding. The only exception to this is pgg-parse-crc24-string which lives in a file marked as obsolete.
  • This leads me to my last point, CCL may be obsoleted as well. Granted, it will take time, but so far I haven’t seen people enthusiastic about it being a thing.

If you still believe that despite of this it’s worth giving CCL a try, please let me know, especially if you’re doing something non-standard with it, like advanced cryptography or number crunching. Likewise, if you’re not convinced that it’s turing-complete or that I could be doing some things far better than presented, send me a message.

-1:-- Code Conversion Language (Post Vasilij Schneidermann)--L0--C0--July 23, 2019 06:37 PM

Chen Bin (redguardtoo): Javascript code navigation in counsel-etags

Javascript code navigation is supported by counsel-etags out of box.

It can support new javascript syntax like arrow function easily because counsel-etags is just frontend.

It reads tags file created by backend CLI program Ctags. Ctags uses regular expression to extract tag name from source code.

But there are some syntax which regular expression could not help.

For example, json object path can't be extracted by regular expression.

Given an object a in file A,

var a = {
  b: {
    c: 3,
  }
};

File B has code let v1 = a.b.c;, how can we jump to the definition of the field c from json path a.b.c?

The solution is to use Lisp to parse code in file A and generate extra navigation data which could be appended to tags file generated by Ctags.

The algorithm is simple,

  • Traverse all the field of object a in file A. Use API js2-print-json-path from js2-mode to get json path of current field.
  • The json path could be regarded as tags name. We've already got file name and line number. So there is enough information to create navigation data for tags file. Here is tags file format.

Necessary utilities are already provided by counsel-etags v1.8.7,

  • After tags files is generated by Ctags, the hook counsel-etags-after-update-tags-hook is executed. Users can append tags file in this hook
  • (counsel-etags-tag-line code-snippet tag-name line-number byte-offset) return a line which could be appended into tags file

My current project uses a technology called styled-components which has an advanced feature theming.

It could dynamically change web application's appearance and is a critical technology for our application to support multiple customer. Application's theme is basically a file containing a huge json object. So it's important that developers can jump to the corresponding json object's field by json path.

Screencast

counsel-etags-plus-json-path.gif

Code

(require 'counsel-etags)

(defun my-manual-update-tag-file (code-file tags-file)
  (let* ((dir (file-name-directory tags-file))
         (path (concat dir code-file))
         curline
         jp
         tagstr)
    (unless (featurep 'js2-mode) (require 'js2-mode))
    (with-temp-buffer
      (insert-file-contents path)
      (js2-init-scanner)
      (js2-do-parse)
      (goto-char (point-min))
      ;; find all js object property names
      (while (re-search-forward "\"?[a-z][a-zA-Z0-9]*\"?:" (point-max) t)
        (when (setq jp (js2-print-json-path))
          (setq curline (string-trim (buffer-substring-no-properties (line-beginning-position)
                                                                     (line-end-position))))
          (setq tagstr (concat tagstr
                               (counsel-etags-tag-line curline
                                                       jp
                                                       (count-lines 1 (point))
                                                       (point)))))
        ;; move focus to next search
        (goto-char (line-end-position))))
    (when tagstr
      (with-temp-buffer
        (insert-file-contents tags-file)
        (goto-char (line-end-position))
        (insert (format "\n\014\n%s,%d\n%s" code-file 0 tagstr))
        (write-region (point-min) (point-max) tags-file nil :silent)))))

(defun counsel-etags-after-update-tags-hook-setup (tags-file)
    (my-manual-update-tag-file "frontend/theming/themes/darkTheme.js" tags-file)
    (my-manual-update-tag-file "frontend/theming/themes/lightTheme.js" tags-file))
(add-hook 'counsel-etags-after-update-tags-hook 'counsel-etags-after-update-tags-hook-setup)
-1:-- Javascript code navigation in counsel-etags (Post Chen Bin)--L0--C0--July 22, 2019 12:07 PM

Shae Erisson: Hyper and Super on my keyboard?

Stranger Strings or the new-ish way to redefine keyboard layouts in the X Window System
-1:-- Hyper and Super on my keyboard? (Post Shae Erisson)--L0--C0--July 21, 2019 12:00 AM

emacsninja: Code Conversion Language

Update: I forgot that I did a brief analysis on this many years ago, using ROT13 as example.

Emacs is most famously, a re-imagination of a Lisp machine, with the Emacs Lisp byte-code interpreter being at its core. A lesser-known fact is that there’s two more byte-code interpreters in its C sources, one for compiled regular expressions and another designed for encoding and decoding text, known as Code Conversion Language (CCL). This blog post will focus on the latter as it’s largely gone unnoticed and hasn’t seen too much experimentation.

The CCL implementation is split into the byte-code interpreter (ccl.c) and compiler (ccl.el) parts. There is no official documentation other than comments and docstrings found in these files. From this I’ve learned that CCL programs are represented as integer vectors and that there’s a higher-level language compiling to them, described in the ccl-define-program docstring. By reading that information I’ve deduced the following:

  • The VM has eight integer-sized registers r0 to r7 and an instruction counter ic
  • Register r7 is used as a status register and may be clobbered at any time by an arithmetic operation
  • A CCL program can either be run on a string and return a string, alternatively it can be run standalone for side effects
  • The former mode requires you to provide a nine-element status vector representing the registers and instruction counter, the latter an eight-element status vector representing the registers only
  • As a side-effect, the status vector contains the new state of the registers and instruction counter after executing the program
  • The VM supports the standard C arithmetic, comparison and assignment operators
  • The language translates several control flow statements to equivalent goto statements, such as if, branch (look-up table) and loop with repeat inside
  • Statements are grouped by surrounding them with parentheses
  • When operating on a string, they are read in and written out in a serial fashion, no random access whatsoever
  • It’s possible to do a look-up on an array, translation table or hash table
  • There is a call operator, but no stack to save/restore arguments to/from, so you’ll have to come up with a calling convention fitting the available registers
  • Each CCL program specifies a magnification factor which determines the ratio between output and input string size

Armed with that knowledge I wrote some boiler plate code for experimentation:

;; -*- lexical-binding: t; -*-

(require 'ccl)

(defvar ccl-status (make-vector 8 0))
(defvar ccl-status+ic (make-vector 9 0))

(defun ccl-init-status (status args)
  (let ((i 0))
    (fillarray status 0)
    (dolist (arg args)
      (aset status i arg)
      (setq i (1+ i)))))

(defun ccl-run (program string &rest args)
  (let ((status ccl-status+ic))
    (ccl-init-status status args)
    (ccl-execute-on-string program status string)))

(defun ccl-run-pure (program &rest args)
  (let ((status ccl-status))
    (ccl-init-status status args)
    (ccl-execute program status)
    status))

There will be some benchmark numbers, none of these are to be taken seriously. Do your own benchmarks before using mine for decisions.

Hello World!

For starters I’ll focus on processing strings. The easiest possible program that still does something useful reads in output and writes it out as is:

(define-ccl-program ccl-identity
  '(1
    (loop
     (read r0)
     (write r0)
     (repeat))))

(ccl-run 'ccl-identity "Hello World!") ;=> "Hello World!"

Let’s go through that program carefully. The S-Expression starts with a magnification factor of 1, meaning that the output buffer should be as large as the input buffer. If it were zero, no I/O would be permitted in the first place, whereas a factor greater than one would allocate enough space to produce a string larger than the input.

The magnification factor is followed by a s-expression that’s executed until it’s done or an error occurred, such as there being no more input. It may be followed by another s-expression that’s executed after the main one, no matter whether it failed with an error or not.

ccl-identity uses a pattern that will come up a few more times in this blog post. It enters a loop, reads a character into the r0 register, writes out a character from the r0 register and jumps to the beginning of the loop. If there are no more characters left, the read operation fails and terminates the loop. Let’s spice things up by adding an extra processing step before writing out the character:

(define-ccl-program ccl-xor
  '(1
    (loop
     (read r1)
     (r1 ^= r0)
     (write r1)
     (repeat))))

(ccl-run 'ccl-xor "Secret" 42) ;=> "yOIXO^"
(ccl-run 'ccl-xor "yOIXO^" 42) ;=> "Secret"

XOR is the bread and butter operator in modern cryptography. A text can be encrypted by replacing each character with the result of XORing it against a secret byte, similarly it can be decrypted by applying the same transformation again. To pass the secret byte as an argument, I’ve placed it in the r0 register and read the string into the r1 register instead. On each iteration of the loop r1 is set to r1 ^ r0 and written out again.

More on translation

In the real world translating characters isn’t as simple as applying some arithmetic to them. Suppose I wanted to challenge the upcase built-in:

(define-ccl-program ccl-upcase
  '(1
    (loop
     (read r0)
     (if (r0 >= ?a)
         (if (r0 <= ?z)
             (r0 -= 32)))
     (write r0)
     (repeat))))

The processing step is a bit more involved this time. If the read-in character appears to be between the a and z characters, transform it by subtracting 32. Why 32? Take a look at an ASCII table and you’ll see that this is the distance between uppercase and lowercase letters. Unfortunately this implementation cannot challenge upcase as it fails to translate non-ASCII characters correctly and is slower than the real deal:

(ccl-run 'ccl-upcase "Hello World!") ;=> "HELLO WORLD!"
(ccl-run 'ccl-upcase "Mötley Crüe") ;=> "MöTLEY CRüE"
(benchmark 100000 '(ccl-run 'ccl-upcase "Hello World!"))
;; => "Elapsed time: 0.165250s (0.072059s in 1 GCs)"
(benchmark 100000 '(upcase "Hello World!"))
;; => "Elapsed time: 0.119050s (0.072329s in 1 GCs)"

Let’s try again with a different text transformation where I actually have a chance to win, ROT13:

(define-ccl-program ccl-rot13
  '(1
    (loop
     (read r0)
     (if (r0 >= ?a)
         (if (r0 <= ?z)
             ((r0 -= ?a)
              (r0 += 13)
              (r0 %= 26)
              (r0 += ?a))))
     (if (r0 >= ?A)
         (if (r0 <= ?Z)
             ((r0 -= ?A)
              (r0 += 13)
              (r0 %= 26)
              (r0 += ?A))))
     (write r0)
     (repeat))))

This time the program needs to recognize two different character ranges to process, lowercase and uppercase ASCII characters. In either case they’re translated to their position in the alphabet, rotated by 13, then translated back to ASCII again. Surprisingly enough, this is enough to beat both rot13-string and rot13-region:

(ccl-run 'ccl-rot13 "Hello World!") ;=> "Uryyb Jbeyq!"
(ccl-run 'ccl-rot13 (ccl-run 'ccl-rot13 "Hello World!"))
;; => "Hello World!"
(benchmark 100000 '(ccl-run 'ccl-rot13 "Hello World!"))
;; => "Elapsed time: 0.248791s (0.072622s in 1 GCs)"
(benchmark 100000 '(rot13-string "Hello World!"))
;; => "Elapsed time: 6.108861s (2.360862s in 32 GCs)"
(with-temp-buffer
  (insert "Hello World!")
  (benchmark 100000 '(rot13-region (point-min) (point-max))))
;; => "Elapsed time: 1.489205s (1.017631s in 14 GCs)"

I then tried to use translation tables for a final example of a “Vaporwave” converter, but failed. Funnily enough this mirrors my overall experience with Emacs, it’s easy to write fun things, but the moment one tries to write something useful, you discover it’s not fun and sometimes not even up to the task. At least it’s possible to salvage the translation tables and use them with translate-region instead, the built-in used by rot13-string and rot13-region:

(defvar ccl-vaporwave-table
  (make-translation-table-from-alist
   (cons '(?\s . 12288)
         (mapcar (lambda (i) (cons i (+ i 65248)))
                 (number-sequence 33 126)))))

(defun vaporwave-it (string)
  (with-temp-buffer
    (insert string)
    (translate-region (point-min) (point-max) ccl-vaporwave-table)
    (buffer-string)))

(vaporwave-it (upcase "aesthetic")) ;=> "AESTHETIC"

Edging towards general-purpose computing

All examples so far have worked on text. If you limit yourself to numbers, you can solve some basic arithmetic problems. Here’s a classic, calculating the factorial of a number:

(define-ccl-program ccl-factorial
  '(0
    ((r1 = 1)
     (loop
      (if r0
          ((r1 *= r0)
           (r0 -= 1)
           (repeat)))))))

(defun factorial (n)
  (let ((acc 1))
    (while (not (zerop n))
      (setq acc (* acc n))
      (setq n (1- n)))
    acc))

While the regular version is more concise, the logic is nearly the same in both. Here’s some numbers:

(aref (ccl-run-pure 'ccl-factorial 10) 1) ;=> 3628800
(factorial 10) ;=> 3628800
(benchmark 100000 '(ccl-run-pure 'ccl-factorial 10))
;; => "Elapsed time: 0.069063s"
(benchmark 100000 '(factorial 10))
;; => "Elapsed time: 0.080212s"

This isn’t nearly as much of a speed-up as I’ve hoped for. Perhaps CCL pays off more when doing arithmetic than for looping? Another explanation is that the Emacs Lisp byte-code compiler has an edge over CCL’s rather simple one. Here’s a more entertaining example, printing out the lyrics of 99 Bottles of Beer on the Wall:

(define-ccl-program ccl-print-bottle-count
  '(1
    (if (r0 < 10)
        (write (r0 + ?0))
      ((write ((r0 / 10) + ?0))
       (write ((r0 % 10) + ?0))))))

(define-ccl-program ccl-99-bottles
  '(1
    (loop
     (if (r0 > 2)
         ((call ccl-print-bottle-count)
          (write " bottles of beer on the wall, ")
          (call ccl-print-bottle-count)
          (write " bottles of beer.\n")
          (write "Take one down and pass it around, ")
          (r0 -= 1)
          (call ccl-print-bottle-count)
          (write " bottles of beer on the wall.\n\n")
          (repeat))
       ((write "2 bottles of beer on the wall, 2 bottles of beer.\n")
        (write "Take one down and pass it around, 1 bottle of beer on the wall.\n\n")
        (write "1 bottle of beer on the wall, 1 bottle of beer.\n")
        (write "Take one down and pass it around, no more bottles of beer on the wall.\n\n")
        (write "No more bottles of beer on the wall, no more bottles of beer.\n")
        (write "Go to the store and buy some more, 99 bottles of beer on the wall.\n"))))))

(defun 99-bottles ()
  (with-output-to-string
    (let ((i 99))
      (while (> i 2)
        (princ (format "%d bottles of beer on the wall, %d bottles of beer.\n" i i))
        (princ (format "Take one down and pass it around, %d bottles of beer on the wall.\n\n" (1- i)))
        (setq i (- i 1))))
    (princ "2 bottles of beer on the wall, 2 bottles of beer.\n")
    (princ "Take one down and pass it around, 1 bottle of beer on the wall.\n\n")
    (princ "1 bottle of beer on the wall, 1 bottle of beer.\n")
    (princ "Take one down and pass it around, no more bottles of beer.\n\n")
    (princ "No more bottles of beer on the wall, no more bottles of beer.\n")
    (princ "Go to the store and buy some more, 99 bottles of beer on the wall.\n")))

This example shows a few more interesting things, generating text of unknown length is rather hard, so I’m using the standard magnification factor of 1 and estimate how big the buffer will be to create an appropriately sized input string. call is useful to not repeat yourself, at the cost of having to carefully plan register usage. Printing out the bottle count can be done if you’re limiting yourself to whole numbers up to 100, a generic solution is going to be hard without random access to the output string. The performance numbers for this one are somewhat surprising:

(let ((input (make-string 15000 ?\s)))
  (benchmark 1000 '(ccl-run 'ccl-99-bottles input 99)))
;; => "Elapsed time: 0.301170s (0.217804s in 3 GCs)"
(benchmark 1000 '(my-99-bottles))
;; => "Elapsed time: 1.735386s (0.507231s in 7 GCs)"

This doesn’t make much sense. Is using format that expensive? It’s hard to tell in advance whether CCL will make a noticable difference or not.

But is it Turing-complete?

My experimentation so far left me wondering, is this language turing-complete? You can perform arithmetics, there’s goto, but the I/O facilities, amount of registers and memory access are limited. The easiest way of proving this property is by implementing another known turing-complete system on top of your current one. I researched a bit and found the following candidates:

  • Brainfuck: A classic, however it requires writable memory. Registers could be used for this, but you don’t have many to play with. You’d need the branch instruction to simulate the data pointer.
  • subleq: Implementing subleq looks easy, but suffers from the same problem as Brainfuck, it requires you to modify an arbitrary memory location. I’ve found a compiler from a C subset to subleq that generates code operating beyond the handful of registers, so that’s not an option either.
  • Rule 110: It’s basically Game of Life, but one-dimensional and can be implemented in a serial fashion. With some tricks it doesn’t require random access either. The proof of it being turing-complete looks painful, but whatever, I don’t care. It’s perfect. There are more elementary cellular automata, so I’ll try to implement it in a generic fashion and demonstrate it on Rule 90 which produces the Sierpinski triangle.
(defmacro define-ccl-automaton (n)
  (let ((print-sym (intern (format "ccl-rule%d-print" n)))
        (rule-sym (intern (format "ccl-rule%d" n))))
    `(progn
       (define-ccl-program ,print-sym
         '(1
           ((r4 = 0)
            (if (r0 == ?1)
                (r4 += 4))
            (if (r1 == ?1)
                (r4 += 2))
            (if (r2 == ?1)
                (r4 += 1))
            (branch r4
                    (write ,(if (zerop (logand n 1)) ?0 ?1))
                    (write ,(if (zerop (logand n 2)) ?0 ?1))
                    (write ,(if (zerop (logand n 4)) ?0 ?1))
                    (write ,(if (zerop (logand n 8)) ?0 ?1))
                    (write ,(if (zerop (logand n 16)) ?0 ?1))
                    (write ,(if (zerop (logand n 32)) ?0 ?1))
                    (write ,(if (zerop (logand n 64)) ?0 ?1))
                    (write ,(if (zerop (logand n 128)) ?0 ?1))))))
       (define-ccl-program ,rule-sym
         '(1
           ((r6 = ,n)
            (r0 = 0)
            (read r1)
            (read r2)
            (loop
             (call ,print-sym)
             (read r3)
             (r0 = r1)
             (r1 = r2)
             (r2 = r3)
             (repeat)))
           ((r0 = r1)
            (r1 = r2)
            (r2 = r5)
            (call ,print-sym)))))))

(define-ccl-automaton 30)
(define-ccl-automaton 90)
(define-ccl-automaton 110)

(defun ccl-sierpinski ()
  (with-output-to-string
    (let ((line "0000000001000000000"))
      (dotimes (_ 20)
        (princ line)
        (terpri)
        (setq line (ccl-run 'ccl-rule90 line))))))

The macro may look scary, but all it does is defining two CCL programs. What an elementary cellular automaton does is looking at the two cells around the current cell, map them to a cell depending to a rule and emit it. There are two edge cases with this for the first and last cell, in my implementation the first cell assumes the previous one was a zero and the last cell uses the first cell. Since there’s no random access, it’s stored into a spare register at the beginning and accessed in a S-Expression after the main loop terminated due to no more input. The surrounding and current cell are stored in three registers and rotated every time a new cell is read in. The mapping is done in the print program by summing up the ones and zeroes, then using the branch instruction to apply the rule to it. If you find this hard to follow, here’s an Emacs Lisp version of it using random access and less limited arithmetic to do the job:

(defun rule--evolve (prev cur next n)
  (let ((acc (+ (if (= prev ?1) 4 0)
                (if (= cur ?1) 2 0)
                (if (= next ?1) 1 0))))
    (if (zerop (logand n (ash 1 acc))) ?0 ?1)))

(defun rule-evolve (line n)
  (let ((out (make-string (length line) ?0)))
    (dotimes (i (length line))
      (cond
       ((zerop i)
        (aset out i (rule--evolve ?0 (aref line i) (aref line (1+ i)) n)))
       ((= i (1- (length line)))
        (aset out i (rule--evolve (aref line (1- i)) (aref line i) (aref line 0) n)))
       (t
        (aset out i (rule--evolve (aref line (1- i)) (aref line i) (aref line (1+ i)) n)))))
    out))

(defun sierpinski ()
  (with-output-to-string
    (let ((line "0000000001000000000"))
      (dotimes (_ 20)
        (princ line)
        (terpri)
        (setq line (rule-evolve line 90))))))

One more benchmark run, this time with less surprising performance numbers:

(benchmark 1000 '(ccl-sierpinski))
;; => "Elapsed time: 0.365031s (0.071827s in 1 GCs)"
(benchmark 1000 '(sierpinski))
;; => "Elapsed time: 0.545512s (0.071829s in 1 GCs)"

If you want to see it in action, try evaluating (progn (princ (sierpinski)) nil) in M-x ielm.

Final words

You may ask yourself now whether you should rewrite all of your slow code to CCL programs. I don’t believe that’s the way to go for a number of reasons:

  • Speedups vary wildly, somewhere between -40% to 450%. There’s no obvious way of predicting whether the optimization is worth it.
  • It’s hard to write an equivalent program or sometimes even impossible, especially if it requires you to use many variables or random access read/write operations.
  • It’s hard to debug a CCL program. While the compiler does a good job at catching syntax errors, runtime errors are far harder to figure out if you can only stare at the registers. Maybe a debugger could be built that uses the “continue” facility and instruction counter register…
  • It’s hard to maintain a CCL program. Not to mention, there’s hardly people who know how to write them. Most of the examples I’ve found online do text encoding/decoding. The only exception to this is pgg-parse-crc24-string which lives in a file marked as obsolete.
  • This leads me to my last point, CCL may be obsoleted as well. Granted, it will take time, but so far I haven’t seen people enthusiastic about it being a thing.

If you still believe that despite of this it’s worth giving CCL a try, please let me know, especially if you’re doing something non-standard with it, like advanced cryptography or number crunching. Likewise, if you’re not convinced that it’s turing-complete or that I could be doing some things far better than presented, send me a message.

-1:-- Code Conversion Language (Post Vasilij Schneidermann)--L0--C0--July 20, 2019 09:31 PM

(or emacs: Ivy 0.12.0 is out

Intro

Ivy is a completion method that's similar to Ido, but with emphasis on simplicity and customizability.

Overview

The current release constitutes of 398 commits and 6 months of progress since 0.11.0. Many issues ranging from #1904 to #2151 were fixed. The number of people who contributed code as grown to 148. Thanks, everyone!

Details on changes

Changelog.org has been a part of the repository since 0.6.0, you can get the details of the current and past changes:

Highlights

Many improvements are incremental and don't require any extra code to enable. I'll go over a few selected features that require a bit of information to make a good use of them.

New bindings

  • counsel-descbinds
    • M-o x execute action.
  • counsel-file-jump
    • M-o d dired.
  • counsel-find-file
    • M-o c copy file.
    • ` bookmarks: efficiently jump between recent directories.
    • $ directories stored in environment variables.
    • C-DEL go up directory. Customize: counsel-up-directory-level.
    • RET open file. Customize: counsel-find-file-extern-extensions.
    • // when on remote, cd to remote root.
    • / C-j select local root.
    • ~ when on remote, cd to remote home.
    • / C-j ~ cd to local home from remote.
  • counsel-git-log
    • M-o v open the current commit in magit.
  • counsel-rg
    • C-x C-d change the current directory for grep.
  • ivy-avy
    • C-v to scroll down.
    • M-v to scroll up.
  • ivy-read C-o
    • m mark and move down.
    • u unmark and move down.
    • DEL move up and unmark.
    • t toggle marks.
    • d perform the action on all marked elements.
  • ivy-switch-buffer
    • C-k kill buffer.
    • M-o x open buffer file externally.
  • ivy-reverse-i-search
    • C-k remove item from the history.

New Commands extensions

These commands are new variants and adaptations of existing commands.

Thing at point variants:

  • swiper-all-thing-at-point.
  • swiper-isearch-thing-at-point.
  • swiper-thing-at-point.

Search variants that go backwards:

  • swiper-backward.
  • counsel-grep-or-swiper-backward.
  • swiper-isearch-backward.

A variant of ivy-switch-buffer with live preview:

  • counsel-switch-buffer.
  • counsel-switch-buffer-other-window.

And finally:

  • counsel-dired - like counsel-find-file, but open dired.
  • swiper-isearch-toggle - toggle between swiper and isearch.

New Commands

I have put these separately so they don't get lost in the crowd. Be sure to try them out.

  • counsel-compile - completion for compile.
  • counsel-register - completion for registers.
  • counsel-minor - completion for minor modes.
  • swiper-isearch - a faster swiper that's not line-based.

Outro

Again, thanks to all the contributors. Happy hacking!

-1:-- Ivy 0.12.0 is out (Post)--L0--C0--July 19, 2019 10:00 PM

Alex Schroeder: Gopher Clients for Emacs

“Always two there are...” – Yoda

I’ve been using the Gopher app on my iOS devices for a while, now, as well as VF-1 on the command line. For Emacs, I was using gopher.el. But yesterday, @gcupc mentioned a second Gopher client for Emacs: Elpher. Indeed, there are at least two implementations of everything in Emacs! 😓

The number one thing I’ve thought when I gave Elpher a try was: there’s no browsing history! There’s no way to say “back”, or “next” (if reading a text file in a menu). When I read the History and Caching entry on GitHub, however, it seems that the up command does what I’d think a back command would do.

And something else I’ve noticed: there’s no TLS support. gophers://alexschroeder.ch:7443 can’t be accessed. I’ve added that to gopher.el, I might as well try and add it to Elpher, I guess?

UTF-8 works. 👍

Tags:

-1:-- Gopher Clients for Emacs (Post)--L0--C0--June 21, 2019 01:16 PM

Jonathan Bennett: Python and Emacs Pt. 1

No Description
-1:-- Python and Emacs Pt. 1 (Post)--L0--C0--June 20, 2019 12:00 AM

Jonathan Bennett: Emacs.org ~ May 2019

No Description
-1:-- Emacs.org ~ May 2019 (Post)--L0--C0--June 18, 2019 12:00 AM

William Denton: Mapping Acid Mothers Temple tours

I just posted Mapping Acid Mothers Temple tours with Org, R and Geonames, which shows how I use Org mode, R and Geonames to go from data printed on the back of a concert t-shirt, like this:

2016 concert shirt 2016 concert shirt

To a map of the band’s recent North American tours:

Map of North American tours Map of North American tours

Some people like RStudio or Jupyter notebooks to work through a problem, but for me it’s Org in Emacs.

-1:-- Mapping Acid Mothers Temple tours (Post William Denton)--L0--C0--June 06, 2019 03:55 PM