Marcin Borkowski: Batch Org agenda

Org-mode agenda is a notoriously complex system. It is really great from a user’s perspective, but a nightmare for a programmer wanting to tap into its generation. A long time ago I have had an idea for a productivity tool which would compute my “agenda score” (for lack of a better name), where each task not yet done would contribute to a total of some kind of “penalty” points – and more of them depending on how long the task is overdue. However, I am probably too stupid to be able to understand what is going on under the hood of org-agenda, and I started to look for better ways. There do exist some tools which might help with that – org-super-agenda being one of them, but there are probably others. Still, I felt this is not the best way. I decided to drop my idea (at least until I have more time and motivation to get back to it). After some time, however, I came back to org-agenda, this time from another direction.
-1:-- Batch Org agenda (Post)--L0--C0--July 15, 2019 06:31 PM

Irreal: Is Emacs an Editor Framework?

Over at the Emacs sub-reddit, yep808 says that he thinks of Emacs not as an editor but as an editor framework that allows you to build the editor you really want. He asks if others share this view. It is, of course, a widely held perspective but as Irreal oldtimers know, not one that I think is precisely accurate.

My view is neatly captured by rhabarba’s comment which posits that, “GNU Emacs is a Lisp machine emulator which can (but does not have to) start a text editor application by default.” I’ve long held that the real power of Emacs comes from that fact: that Emacs is, in fact, a light-weight Lisp Machine.

Sadly, it’s nowhere nearly as powerful (a software system) as the original Lisp Machines but it does allow us to do all sorts of non-editing things like using it as an RSS reader, or music player, or Email client, or any number of other things. And, of course, we can easily build our own applications in Emacs Lisp.

It might seem as if it doesn’t matter how you view Emacs but I believe thinking of it as a Lisp Machine has a definite benefit: it encourages the view that Emacs is a programmable environment in which you can build all sorts of applications that are only peripherally, if at all, related to text editing. Once you adopt that point of view, you can start really tapping into the power of Emacs by writing code to solve your own problems.

-1:-- Is Emacs an Editor Framework? (Post jcs)--L0--C0--July 15, 2019 05:00 PM

Irreal: Long Lines in Emacs

One of the (legitimate) complaints about Emacs is that it doesn’t do very well with excessively long lines. Happily, this isn’t a problem for me. I write my blog posts with visual line mode enabled and fill disabled so Emacs sees each paragraph as a single line. Despite that, I never run into problems so it’s not a issue for many (or perhaps even most) users.

Still, if it does effect you, it can be a real annoyance: Emacs can take a considerable amount of time to render the text. Happily, help is on the way. A fix for the long lines problem has been added to the 27.1 Master branch. Some early 27.1 adopters are reporting excellent results with the fix.

If you follow the link at the top of reddit link above, you will see the documentation for the changes. It’s worth reading that to get a feel for how the changes will affect working with long lines.

This change should—but probably won’t—put to rest the notion that Emacs is an old, moribund editor still alive only because a few gray beards refuse to let go. It shows that development is strong and on-going and even long standing issues are being worked on and resolved.

-1:-- Long Lines in Emacs (Post jcs)--L0--C0--July 14, 2019 02:58 PM

sachachua: Tweaking Emacs on Android via Termux: xclip, xdg-open, syncthing conflicts

I’m planning to leave my laptop at home during our 3-week trip, so I’ve been shifting more things to my phone in preparation. Orgzly is handy for reviewing my agenda and adding items on the go, but nothing beats Emacs for full flexibility. Runnng Emacs via Termux works pretty well. I can resize by pinch-zooming, scroll by swiping, and even enter in more text by swiping right on the row of virtual keyboard buttons.

Here’s what I’ve configured in ~/.termux/termux-properties:

extra-keys = [['ESC','/','-','HOME','UP','END','PGUP','F5'],['TAB','CTRL','ALT','LEFT','DOWN','RIGHT','PGDN','F6']]

and here’s what that looks like:

Screenshot of Emacs

I patched my Termux to allow the use of function keys in the extra keys shortcut bar. It’s been merged upstream, but the new version hasn’t been released yet, so I’m still running the one I compiled from source. It would be nice to fix accidental keypresses when swiping extra keys to get to the input field, but that can wait a bit.

I set up Syncthing to synchronize files with my server and laptop, Termux:API to make storage accessible, and symlinks in my home directory to replicate the main parts of my setup. I set up Orgzly to auto-sync with a local repository (synchronized with my server and laptop via Syncthing) and the same Org files set up in my agenda in Emacs on Termux. That way, I can hop between Orgzly and Emacs as quickly as I want. Here are a few tweaks that made Emacs even better.

First, a little bit of code for phone-specific config, taking advantage of the weird automatic username I have there.

(defun my/phone-p ()
  (and (equal (system-name) "localhost") 
       (not (equal user-login-name "sacha"))))

For Emacs News, I wanted to be able to easily open Org Mode links to webpages by tapping on them. This code makes that work (and in general, anything that involves opening webpages):

(setq browse-url-browser-function 'browse-url-xdg-open)

This piece of code cleans up my Inbox.org file so that it’s easier to skim in Orgzly. It archives all the done tasks and sorts them.

     (defun my/org-clean-up-inbox ()
       "Archive all DONE tasks and sort the remainder by TODO order."
       (interactive)
       (with-current-buffer (find-file my/org-inbox-file)
         (my/org-archive-done-tasks 'file)
         (goto-char (point-min))
         (if (org-at-heading-p) (save-excursion (insert "\n")))
         (org-sort-entries nil ?p)
         (goto-char (point-min))
         (org-sort-entries nil ?o)
         (save-buffer)))

     (defun my/org-archive-done-tasks (&optional scope)
       "Archive finished or cancelled tasks.
SCOPE can be 'file or 'tree."
       (interactive)
       (org-map-entries
        (lambda ()
          (org-archive-subtree)
          (setq org-map-continue-from (outline-previous-heading)))
        "TODO=\"DONE\"|TODO=\"CANCELLED\"" (or scope (if (org-before-first-heading-p) 'file 'tree))))

I also sometimes wanted to copy and paste between Termux and Emacs by using the keyboard, so I submitted a patch for xclip.el so that it would detect and work with termux-clipboard-get/set. That code is now in xclip 1.9 in ELPA, so if you M-x package-install xclip, you should be able to turn on xclip-mode and have it copy and paste between applications. In my config, that looks like:

(when (my/phone-p)
  (use-package xclip :config (xclip-mode 1)))

Because I use Orgzly and Termux to edit my Org files and I also edit the files on my laptop, I occasionally get synchronization errors. I came across this handy bit of code to find Syncthing conflicts and resolve them. I just had to change some of the code (in bold below) in order to make it work, and I needed to pkg install diffutils to solve the diff errors. Here’s the fixed code below, along with a convenience function that checks my Orgzly directory:

(defun my/resolve-orgzly-syncthing ()
  (interactive)
  (ibizaman/syncthing-resolve-conflicts "~/cloud/orgzly"))

(defun ibizaman/syncthing-resolve-conflicts (directory)
  "Resolve all conflicts under given DIRECTORY."
  (interactive "D")
  (let* ((all (ibizaman/syncthing--get-sync-conflicts directory))
        (chosen (ibizaman/syncthing--pick-a-conflict all)))
    (ibizaman/syncthing-resolve-conflict chosen)))

(defun ibizaman/syncthing-show-conflicts-dired (directory)
  "Open dired buffer at DIRECTORY showing all syncthing conflicts."
  (interactive "D")
  (find-name-dired directory "*.sync-conflict-*"))

(defun ibizaman/syncthing-resolve-conflict-dired (&optional arg)
  "Resolve conflict of first marked file in dired or close to point with ARG."
  (interactive "P")
  (let ((chosen (car (dired-get-marked-files nil arg))))
    (ibizaman/syncthing-resolve-conflict chosen)))

(defun ibizaman/syncthing-resolve-conflict (conflict)
  "Resolve CONFLICT file using ediff."
  (let* ((normal (ibizaman/syncthing--get-normal-filename conflict)))
    (ibizaman/ediff-files
     (list conflict normal)
     `(lambda ()
       (when (y-or-n-p "Delete conflict file? ")
         (kill-buffer (get-file-buffer ,conflict))
         (delete-file ,conflict))))))

(defun ibizaman/syncthing--get-sync-conflicts (directory)
  "Return a list of all sync conflict files in a DIRECTORY."
  (directory-files-recursively directory "\\.sync-conflict-"))

(defvar ibizaman/syncthing--conflict-history nil 
  "Completion conflict history")

(defun ibizaman/syncthing--pick-a-conflict (conflicts)
  "Let user choose the next conflict from CONFLICTS to investigate."
  (completing-read "Choose the conflict to investigate: " conflicts
                   nil t nil ibizaman/syncthing--conflict-history))


(defun ibizaman/syncthing--get-normal-filename (conflict)
  "Get non-conflict filename matching the given CONFLICT."
  (replace-regexp-in-string "\\.sync-conflict-.*\\(\\..*\\)$" "\\1" conflict))


(defun ibizaman/ediff-files (&optional files quit-hook)
  (interactive)
  (lexical-let ((files (or files (dired-get-marked-files)))
                (quit-hook quit-hook)
                (wnd (current-window-configuration)))
    (if (<= (length files) 2)
        (let ((file1 (car files))
              (file2 (if (cdr files)
                         (cadr files)
                       (read-file-name
                        "file: "
                        (dired-dwim-target-directory)))))
          (if (file-newer-than-file-p file1 file2)
              (ediff-files file2 file1)
            (ediff-files file1 file2))
          (add-hook 'ediff-after-quit-hook-internal
                    (lambda ()
                      (setq ediff-after-quit-hook-internal nil)
                      (when quit-hook (funcall quit-hook))
                      (set-window-configuration wnd))))
      (error "no more than 2 files should be marked"))))

If you use Emacs on Android via Termux, I hope some of these tweaks help you too!

-1:-- Tweaking Emacs on Android via Termux: xclip, xdg-open, syncthing conflicts (Post Sacha Chua)--L0--C0--July 13, 2019 08:39 PM

Shae Erisson: Part two of SMT Solvers, Integer Linear Programming

Other people already know how to do this well
-1:-- Part two of SMT Solvers, Integer Linear Programming (Post Shae Erisson)--L0--C0--July 12, 2019 12:00 AM

Shae Erisson: SMT Solvers, Integer Linear Programming

How do I get started optimizing cost/benefit with constraints?
-1:-- SMT Solvers, Integer Linear Programming (Post Shae Erisson)--L0--C0--July 10, 2019 12:00 AM

(or emacs: Ivy reverse-i-search

Introduction

I'm sure many are aware of the C-r functionality in bash (a whole lot of Emacs bindings are valid in bash and do the same thing). I also like the quirky Emacs-style prompt, that uses a backquote to open and a straight quote to close a your quoted input:

bash-reverse-i-search.png

So when you want to cd to somewhere where you were before you do C-r cd. And then the C-r "roulette" begins: you keep pressing C-r over and over, in the hopes to find what you were looking for.

Getting better history completion with Ivy

Ivy improves the "roulette" situation in two ways:

  • You get an overview of the matching candidates and their count,
  • You can quickly narrow down the candidates with fuzzy matching.

Here's the basic setup to enable C-r completion using ivy:

(define-key minibuffer-local-map
    (kbd "C-r") 'counsel-minibuffer-history)
(define-key shell-mode-map
    (kbd "C-r") 'counsel-shell-history)

The first key binding is also part of counsel-mode, while the second needs to be set up separately, after shell-mode was loaded.

And here's how counsel-shell-history looks like:

counsel-shell-history.png

The cool thing is that ivy-reverse-i-search applies to any Ivy command, not just for shell command completion. I find it especially useful for:

  • counsel-find-file
  • eval-expression

Recent improvement: delete history items

While searching with regexes is great, it's not so great when the old stuff that we won't ever need gets in the way. So now you can delete it with C-k. Since C-k also has to serve as ivy-kill-line, the history deleting behavior will only come into effect when the point is at the end of the line (so that ivy-kill-line does not make sense anyway).

ivy-reverse-i-search-kill.png

This way, your typos don't stay around to haunt you.

Outro

I hope you'll find ivy-reverse-i-search a useful boost to your completion needs. Happy hacking!

-1:-- Ivy reverse-i-search (Post)--L0--C0--July 08, 2019 10:00 PM

Marcin Borkowski: Pausing an Emacs keyboard macro

As I promised last week, I’d like to describe something I probably knew a long time ago, but completely forgot about. You can pause an Emacs keyboard macro during its execution. But wait, there’s more!
-1:-- Pausing an Emacs keyboard macro (Post)--L0--C0--July 08, 2019 09:02 PM

sachachua: 2019-07-08 Emacs news

Links from reddit.com/r/emacs, /r/orgmode, /r/spacemacs, /r/planetemacs, Hacker News, planet.emacslife.com, YouTube, the changes to the Emacs NEWS file, and emacs-devel.

-1:-- 2019-07-08 Emacs news (Post Sacha Chua)--L0--C0--July 08, 2019 12:50 PM

Manuel Uberti: Motivate yourself

Life is such an ephemeral thing. When you really think about it, the time spent on the planet we are destroying day after day is not that much, especially compared to the entire history of humankind. Luckily enough, I can allow myself to indulge in thoughts about life and time. I tend to avoid laziness as much as possible, saving spare time for gratifying leisure: morning runs and basic workout, books, cinema, collecting vinyls, occasional cooking. I like to think of myself as someone who is not set on a predictable path for the rest of his life.

Lassi Kortela’s memento-mori has recently appeared on MELPA. It’s a small package which aims to display your age in the mode-line. The age is displayed with two decimals and it gets updated after a few days once you turned on memento-mori-mode. Thus you have a reminder of how long you have been on Earth, a somehow inspirational indicator to question the time at your disposal. We are entering philosophical worlds here.

However, I don’t like having too much stuff in my mode-line. I want it simple and with only the information I need. That’s why I use minions to hide all the minor-modes. But I can still use memento-mori to display a message on Emacs startup.

(defun mu-display-memento-mori ()
  "Display my current age by leveraging `memento-mori-age-string'."
  (interactive)
  (let ((first-name (car (s-split-words user-full-name)))
        (age (s-trim memento-mori-age-string))
        (msg "%s, you are %s, don't waste your time!"))
    (run-with-timer 2 nil
                    (lambda ()
                      (message msg first-name age)
                      (run-with-timer 3 nil
                                      (lambda ()
                                        (message nil)))))))

I am using s facilities to manipulate strings, but more importantly the trick is using run-with-timer as explained on this answer on Emacs.StackExchange. After two seconds, the message is shown, and three seconds after being visible it will disappear.

Provided you set memento-mori-birth-date and activate memento-mori-mode, adding the function to after-init-hook does the rest.

Note that you need to activate Lexical Binding in your init file to use mu-display-memento-mori1.

Notes

  1. Thanks to Martin Buchmann for pointing this out. 

-1:-- Motivate yourself (Post)--L0--C0--July 06, 2019 12:00 AM

Manuel Uberti: Getting ready for Heart of Clojure

Quick announcement.

I’m going to be in Leuven from 1 to 3 August, ready to enjoy Heart of Clojure.

I’ve been following the development of the conference since the first announcements, and the awesome team behind it has been keeping me really excited along the way. The incredible variety of topics in the program is already enough to skyrocket my expectations.

As I did with Dutch Clojure Days last year, I’ll take notes diligently and report my experience here as soon as I get back.

If you fancy wise Clojure chats or wild Emacs ramblings, get in touch. There is a dedicated #heartofclojure channel on Clojurians which I’ve already joined, or you can always rely on good old emails.

See you there, Clojurists!

-1:-- Getting ready for Heart of Clojure (Post)--L0--C0--July 03, 2019 12:00 AM

(or emacs: Ivy usability improvements when dealing with directories

Introduction

When Ivy just started out as a completion framework, the functionality was supposed to be simple: select one string from a list of strings. The UI is simple enough:

  • Show the list of strings that match the entered text,
  • Use C-n and C-p to navigate them,
  • Use C-m or C-j to submit.

Emacs has three key bindings that mean "Enter": RET, C-m, and C-j. But in terminal mode, emacs -nw, RET and C-m are the same binding: Emacs can't distinguish them. So we have at most two bindings. Fortunately, the world of completion is simple at the moment, and we only need one binding.

File name completion

Enter file name completion. When you're completing file names, you are selecting not one string, but many strings in succession while moving from one directory to the next. So we need at least two key bindings:

  • Use C-m (ivy-done) to select the current candidate and exit completion.
  • Use C-j (ivy-alt-done) to change the current directory to the current candidate without exiting completion.

What to do when C-j is used on a file and not on a directory? Might as well open the file: same action as C-m. OK, we had two key bindings, and we have used them. Hopefully nothing else comes up.

Enter creating a new file, i.e. selecting something that's not on the list of strings. Suppose I call find-file, enter "do", and the only match is a directory named "doc":

  • Pressing C-m will end completion with the "doc" directory selected.
  • Pressing C-j will continue completion inside the "doc" directory.

So creating a file named "do" is the third action. Our two "Enter" keybindings are already taken by the first two different useful actions. So we need a third key binding. The one I chose is C-M-j (ivy-immediate-done). It means: I don't care that the current input is not on the list of candidate strings, submit it anyway.

Directory creation

Enter directory creation: dired-create-directory and make-directory. These built-in Emacs commands request file name completion, but what they tell Ivy is no different from what find-file tells: "I want to select a file". However, for these commands, the C-M-j action is the one that makes most sense. Here it would be nice for Ivy to take the back seat and just act like an interactive ls, since the user will enter a completely new string that's not on the list of candidates.

For a long time, you still had to use C-M-j with those commands, to much frustration of new but also existing users, including myself. But a few days ago, I looked at the prompt that dired-create-directory uses: "Create directory: ". That prompt is passed to Ivy. Using the prompt to detect the intention of the command is a bit of a hack, but I think in this case it's justifiable. So now Ivy will recognize that the intention commands that request file name completion and pass the "Create directory: " prompt is to create a directory, and all key bindings will do just that: C-m, C-j, and C-M-j will behave the same in this case.

An alternative key binding scheme

Note that C-m and C-j behave differently only for directories. But thanks to the fact that "." is always the first candidate, C-m for directories is equivalent to C-j C-j. So we can get away with just using ivy-alt-done, and bind C-m to ivy-immediate-done. Or swap the two meanings:

(define-key ivy-minibuffer-map (kbd "C-j") 'ivy-immediate-done)
(define-key ivy-minibuffer-map (kbd "C-m") 'ivy-alt-done)

The price to pay here is the extra context switch when we simply want to select a directory. We could the bind ivy-done to C-M-j and avoid the context switch, but then we're back to three bindings once more. Still, I thought that swapping the bindings is an interesting idea worth sharing.

Canceling dired-dwim-target

Setting dired-dwim-target is a nice productivity boost. It allows to use Emacs in a similar way to a two-pane file explorer, like mc(1). But it was really annoying when I was in dir-1 with the intention to copy a file to a different name in dir-1 (e.g. create a backup, or copy a template), but the current directory was set to dir-2 because of a random dired window I had open. In that case, I had to call delete-other-windows, perform the copy, and then restore the window configuration with winner-undo.

I did the above many times over many years, until I finally dug into the code of dired.el to see how dired-dwim-target worked. Turns out it was storing dir-1 in the minibuffer-defaults variable. So now Ivy will use that variable when I press M-n. All in all, it was a five minute fix. But rather than regret that I didn't do it years ago, I'm glad I did it now. It only remains to build some muscle memory to press M-n in that situation.

I'm guesstimating that dired-dwim-target works to my advantage 90% of the time when I press C in dired. For the other 10% of the times, I can now press M-n.

Outro

I hope you find the new functionality useful. I'm always open to new ideas and pull requests. Happy hacking!

-1:-- Ivy usability improvements when dealing with directories (Post)--L0--C0--June 26, 2019 10:00 PM

Alex Schroeder: Gopher Clients for Emacs

“Always two there are...” – Yoda

I’ve been using the Gopher app on my iOS devices for a while, now, as well as VF-1 on the command line. For Emacs, I was using gopher.el. But yesterday, @gcupc mentioned a second Gopher client for Emacs: Elpher. Indeed, there are at least two implementations of everything in Emacs! 😓

The number one thing I’ve thought when I gave Elpher a try was: there’s no browsing history! There’s no way to say “back”, or “next” (if reading a text file in a menu). When I read the History and Caching entry on GitHub, however, it seems that the up command does what I’d think a back command would do.

And something else I’ve noticed: there’s no TLS support. gophers://alexschroeder.ch:7443 can’t be accessed. I’ve added that to gopher.el, I might as well try and add it to Elpher, I guess?

UTF-8 works. 👍

Tags:

-1:-- Gopher Clients for Emacs (Post)--L0--C0--June 21, 2019 01:16 PM

Jonathan Bennett: Python and Emacs Pt. 1

No Description
-1:-- Python and Emacs Pt. 1 (Post)--L0--C0--June 20, 2019 12:00 AM

Jonathan Bennett: Emacs.org ~ May 2019

No Description
-1:-- Emacs.org ~ May 2019 (Post)--L0--C0--June 18, 2019 12:00 AM

William Denton: Mapping Acid Mothers Temple tours

I just posted Mapping Acid Mothers Temple tours with Org, R and Geonames, which shows how I use Org mode, R and Geonames to go from data printed on the back of a concert t-shirt, like this:

2016 concert shirt 2016 concert shirt

To a map of the band’s recent North American tours:

Map of North American tours Map of North American tours

Some people like RStudio or Jupyter notebooks to work through a problem, but for me it’s Org in Emacs.

-1:-- Mapping Acid Mothers Temple tours (Post William Denton)--L0--C0--June 06, 2019 03:55 PM

Chen Bin (redguardtoo): dianyou 0.0.3 is out

I created dianyou (电邮) to solve improve my Gnus work flow.

Search mails

Most Gnus users use gnus-group-make-nnir-group to search mails. But it's not usable in "Summary Buffer". Its IMAP SEARCH Command is hard to use.

It could be replaced by dianyou-group-make-nnir-group which supports shortcuts. For example, you can input "f John s 1y2w2d" and select "imap" when Gnus asks the question "Imap Search in". It means search mails from John since 1 year, 2 weeks and 2 days ago.

Auto complete email address

I'm too lazy to set up my contacts. It would be better that Gnus could automatically collect email addresses from all received mails. So I created the command dianyou-insert-email-address-from-received-mails. The inserted email address is also inserted into the global variable dianyou-email-address-history which could be persisted by session.

Switch Gnus buffer

One annoying thing in Gnus is pressing "q" will close opened folder. I often pressed "q" when I only intended to switch to another Gnus buffer. Now I can use dianyou-switch-gnus-buffer.

Paste image from clipboard

Use dianyou-paste-image-from-clipboard. CLI program xclip should be installed at first.

-1:-- dianyou 0.0.3 is out (Post Chen Bin)--L0--C0--June 04, 2019 12:54 PM

Saurabh Kukade: Why Emacs

Why I think Emacs is so cool - “Emacs outshines all other editing software in approximately the same way that the noonday sun does the stars. It is not just bigger and brighter; it simply makes everything else vanish.” -Neal Stephenson, “In the Beginning was the Command Line” Introduction This is my first blog post about Emacs. I...
-1:-- Why Emacs (Post)--L0--C0--June 04, 2019 07:00 AM

Nicolas Martyanoff: Reading RFC documents in Emacs

I am used to read RFC documents in Emacs by opening them as simple text files. But I was always annoyed by the impossibility to follow references to other documents. Additionally, searching for the right RFC usually involve going back to the web browser and Google while I would prefer staying in Emacs.

Since I enjoy Helm so much, I thought it would be easy to parse the index of all RFC documents, and create a browser allowing me to pick a RFC. And since I was writing some elisp, I ended up writing a major mode to highlight parts of RFC documents, including adding links when another RFC is mentioned.

So here is rfc-mode. It is quite simple for the time being, and I have a few ideas for more features, but it is already quite useful.

It is also available on MELPA.

RFC browser

RFC reader

-1:-- Reading RFC documents in Emacs (Post)--L0--C0--June 02, 2019 12:00 AM

Chris Wellons: UTF-8 String Indexing Strategies

This article was discussed on Hacker News.

When designing or, in some cases, implementing a programming language with built-in support for Unicode strings, an important decision must be made about how to represent or encode those strings in memory. Not all representations are equal, and there are trade-offs between different choices.

One issue to consider is that strings typically feature random access indexing of code points with a time complexity resembling constant time (O(1)). However, not all string representations actually support this well. Strings using variable length encoding, such as UTF-8 or UTF-16, have O(n) time complexity indexing, ignoring special cases (discussed below). The most obvious choice to achieve O(1) time complexity — an array of 32-bit values, as in UCS-4 — makes very inefficient use of memory, especially with typical strings.

Despite this, UTF-8 is still chosen in a number of programming languages, or at least in their implementations. In this article I’ll discuss three examples — Emacs Lisp, Julia, and Go — and how each takes a slightly different approach.

Emacs Lisp

Emacs Lisp has two different types of strings that generally can be used interchangeably: unibyte and multibyte. In fact, the difference between them is so subtle that I bet that most people writing Emacs Lisp don’t even realize there are two kinds of strings.

Emacs Lisp uses UTF-8 internally to encode all “multibyte” strings and buffers. To fully support arbitrary sequences of bytes in the files being edited, Emacs uses its own extension of Unicode to precisely and unambiguously represent raw bytes intermixed with text. Any arbitrary sequence of bytes can be decoded into Emacs’ internal representation, then losslessly re-encoded back into the exact same sequence of bytes.

Unibyte strings and buffers are really just byte-strings. In practice, they’re essentially ISO/IEC 8859-1, a.k.a. Latin-1. It’s a Unicode string where all code points are below 256. Emacs prefers the smallest and simplest string representation when possible, similar to CPython 3.3+.

(multibyte-string-p "hello")
;; => nil

(multibyte-string-p "π ≈ 3.14")
;; => t

Emacs Lisp strings are mutable, and therein lies the kicker: As soon as you insert a code point above 255, Emacs quietly converts the string to multibyte.

(defvar fish "fish")

(multibyte-string-p fish)
;; => nil

(setf (aref fish 2) ?ŝ
      (aref fish 3) ?o)

fish
;; => "fiŝo"

(multibyte-string-p fish)
;; => t

Constant time indexing into unibyte strings is straightforward, and Emacs does the obvious thing when indexing into unibyte strings. It helps that most strings in Emacs are probably unibyte, even when the user isn’t working in English.

Most buffers are multibyte, even if those buffers are generally just ASCII. Since Emacs uses gap buffers it generally doesn’t matter: Nearly all accesses are tightly clustered around the point, so O(n) indexing doesn’t often matter.

That leaves multibyte strings. Consider these idioms for iterating across a string in Emacs Lisp:

(dotimes (i (length string))
  (let ((c (aref string i)))
    ...))

(cl-loop for c being the elements of string
         ...)

The latter expands into essentially the same as the former: An incrementing index that uses aref to index to that code point. So is iterating over a multibyte string — a common operation — an O(n^2) operation?

The good news is that, at least in this case, no! It’s essentially just as efficient as iterating over a unibyte string. Before going over why, consider this little puzzle. Here’s a little string comparison function that compares two strings a code point at a time, returning their first difference:

(defun compare (string-a string-b)
  (cl-loop for a being the elements of string-a
           for b being the elements of string-b
           unless (eql a b)
           return (cons a b)))

Let’s examine benchmarks with some long strings (100,000 code points):

(benchmark-run
    (let ((a (make-string 100000 0))
          (b (make-string 100000 0)))
      (compare a b)))
;; => (0.012568031 0 0.0)

With using two, zeroed unibyte strings it takes 13ms. How about changing the last code point in one of them to 256, converting it to a multibyte string:

(benchmark-run
    (let ((a (make-string 100000 0))
          (b (make-string 100000 0)))
      (setf (aref a (1- (length a))) 256)
      (compare a b)))
;; => (0.012680513 0 0.0)

Same running time, so that multibyte string cost nothing more to iterate across. Let’s try making them both multibyte:

(benchmark-run
    (let ((a (make-string 100000 0))
          (b (make-string 100000 0)))
      (setf (aref a (1- (length a))) 256
            (aref b (1- (length b))) 256)
      (compare a b)))
;; => (2.327959762 0 0.0)

That took 2.3 seconds: about 2000x longer to run! Iterating over two multibyte strings concurrently seems to have broken an optimization. Can you reason about what’s happened?

To avoid the O(n) cost on this common indexing operating, Emacs keeps a “bookmark” for the last indexing location into a multibyte string. If the next access is nearby, it can starting looking from this bookmark, forwards or backwards. Like a gap buffer, this gives a big advantage to clustered accesses, including iteration.

However, this string bookmark is global, one per Emacs instance, not once per string. In the last benchmark, the two multibyte strings are constantly fighting over a single string bookmark, and indexing in comparison function is reduced to O(n^2) time complexity.

So, Emacs pretends it has constant time access into its UTF-8 text data, but it’s only faking it with some simple optimizations. This usually works out just fine.

Julia

Another approach is to not pretend at all, and to make this limitation of UTF-8 explicit in the interface. Julia took this approach, and it was one of my complaints about the language. I don’t think this is necessarily a bad choice, but I do still think it’s inappropriate considering Julia’s target audience (i.e. Matlab users).

Julia strings are explicitly byte strings containing valid UTF-8 data. All indexing occurs on bytes, which is trivially constant time, and always decodes the multibyte code point starting at that byte. But it is an error to index to a byte that doesn’t begin a code point. That error is also trivially checked in constant time.

s = "π"

s[1]
# => 'π'

s[2]
# ERROR: UnicodeError: invalid character index
#  in getindex at ./strings/basic.jl:37

Slices are still over bytes, but they “round up” to the end of the current code point:

s[1:1]
# => "π"

Iterating over a string requires helper functions which keep an internal “bookmark” so that each access is constant time:

for i in eachindex(string)
    c = string[i]
    # ...
end

So Julia doesn’t pretend, it makes the problem explicit.

Go

Go is very similar to Julia, but takes an even more explicit view of strings. All strings are byte strings and there are no restrictions on their contents. Conventionally strings contain UTF-8 encoded text, but this is not strictly required. There’s a unicode/utf8 package for working with strings containing UTF-8 data.

Beyond convention, the range clause also assumes the string contains UTF-8 data, and it’s not an error if it does not. Bytes not containing valid UTF-8 data appear as a REPLACEMENT CHARACTER (U+FFFD).

func main() {
    s := \xff"
    for _, r := range s {
        fmt.Printf("U+%04x\n", r)
    }
}

// U+03c0
// U+fffd

A further case of the language favoring UTF-8 is that casting a string to []rune decodes strings into code points, like UCS-4, again using REPLACEMENT CHARACTER:

func main() {
    s := \xff"
    r := []rune(s)
    fmt.Printf("U+%04x\n", r[0])
    fmt.Printf("U+%04x\n", r[1])
}

// U+03c0
// U+fffd

So, like Julia, there’s no pretending, and the programmer explicitly must consider the problem.

Preferences

All-in-all I probably prefer how Julia and Go are explicit with UTF-8’s limitations, rather than Emacs Lisp’s attempt to cover it up with an internal optimization. Since the abstraction is leaky, it may as well be made explicit.

-1:-- UTF-8 String Indexing Strategies (Post)--L0--C0--May 29, 2019 09:52 PM

Emacs Redux: Spell Checking Comments

I’m notorious for all the typos I make.1 Thankfully Emacs features an awesome built-in mode named flyspell to help poor typists like me. Flyspell highlights misspelled words as you type (a.k.a. on the fly) and has useful keybindings to quickly fix them.

Most people typically enable flyspell only for major modes derived from text-mode (e.g. markdown-mode, adoc-mode), but it can really help programmers as well by pointing out typos they make in comments. All you need to do is enable flyspell-prog-mode. I typically enable it for all programming modes2 like this:

(add-hook 'prog-mode-hook #'flyspell-prog-mode)

Now you’ll get instant feedback when you make some typo in a comment. To fix a word just press C-c $ (M-x flyspell-correct-word-before-point), while your cursor is behind it.3

flyspell_prog_mode.gif

That’s all I have for you today! Keep fixing those nasty typos!

  1. Especially in blog posts. 

  2. At least the well-behaved ones that derive from prog-mode that is. 

  3. There are many other ways to correct misspelled words with flyspell, but we’ll ignore them for now for the sake of simplicity. 

-1:-- Spell Checking Comments (Post Bozhidar Batsov)--L0--C0--May 24, 2019 07:07 AM

Modern Emacs: Colorful ielm - font-locking comint

If you ever used ielm, or other comint-mode derivatives, you will notice that the text you input is not highlighted according to the major-mode. If I type (setq foo bar) into ielm, the setq won't be highlighted. Why is this? And how do we change this? Naive solution Look at font-lock-keywords in ielm and it is suspiciously near-empty. We could copy over emacs-lisp's keywords: (setq-local font-lock-keywords `(,@lisp-el-font-lock-keywords-2 ,@lisp-cl-font-lock-keywords-2)) But what if I type in (princ "(setq foo bar)")?
-1:-- Colorful ielm - font-locking comint (Post)--L0--C0--May 23, 2019 12:00 AM

Rubén Berenguel: 2019#13 Readings of the week

NOTE: The themes are varied, and some links below are affiliate links. Functional programming, adtech, history. Expect a similar wide range in the future as well. You can check all my weekly readings by checking the tag here . You can also get these as a weekly newsletter by subscribing here.

Photo by Mgg Vitchakorn on Unsplash

Ancient Chinese Buildings Are Held Together With Rice

As everyone who has prepared sticky rice and forgotten to clean the pan quickly knows.

'I'd Have These Extremely Graphic Dreams': What It's Like To Work On Ultra-Violent Games

Fatality burnout.

The Artistry of China’s Ivory Puzzle Balls

I’ve seen one from really close and they are amazing

Bristol academic cracks Voynich code

There was a rebuttal of this approach close to 2 years ago, but it still sounds interesting. Also, I’d expect phys.org to be somewhat trustable.

Technical Debt? Probably not.

Pay it off. Or not.

Scala's Future.successful: Do Not Block Your Future Success

Beware the future!

The Functional Scala Concurrency Challenge

Some of the solutions look neat, and there is something to be learnt in the diversity.

Technical overview of ads viewability measurement methods

I have always been interested into how ad viewability (if an ad is seen by a user or not) works. Now I know.

GraalVM installation and setup on macOS

GraalVM is a new Java runtime and compiler, which is somewhat faster than the normal JDK, and offers some really fancy cross-language options. I had been using RC9 or 10. It was time to update. I usually use Enterprise Edition for anything local, but can switch per-terminal with several aliases I have (j8, j11, jgree and jgrce) in case something breaks or I want to try another.

Caire - a content aware image resize library

Seam carving is so cool. I wrote a cropping system once, using PILs Haar cascade based object detection. It works pretty well for automated creation of ads, but seam carving is way better for almost all other cases.

🎥 Flare - Optimizing Apache Spark for Scale-Up Architectures and Medium-Size Data

I’m scared this will become a closed source, or business-on-top. Code is not available, but results are awesome. Luckily, the papers are (why the code is not as part of the papers is another question).

🎥 helm-edit

I had used multiple-cursors (and expand-region) in emacs many times, although I had a hard time making it work properly in spacemacs for a while. helm-edit works better already in spacemacs, so, big win!

🎥 Understanding Query Plans and Spark UIs

I knew everything in this video already, but it covers lots. Give it a look.

🎥 What the ƒ is a monad

Although focused on Java, it’s very well explained for any language. Of course, if you have do syntax or for comprehensions, better.

📚 The Great Mental Models

The audiobook by Shane Parrish, of FS Blog. It was good, but I expected something longer.

📚 The art of thinking in systems

If you have read Donna Meadows Thinking in Systems, this won’t give you anything new. And I’d rather recommend TiS.

Newsletter?

These weekly posts are also available as a newsletter. These days (since RSS went into limbo) most of my regular information comes from several newsletters I’m subscribed to, instead of me going directly to a blog. If this is also your case, subscribe by clicking here.
-1:-- 2019#13 Readings of the week (Post Ruben Berenguel (noreply@blogger.com))--L0--C0--May 21, 2019 09:42 PM

Rubén Berenguel: gtags (GNU global) in emacs for Scala

Photo by thom masat on Unsplash
As you may know, I’m a heavy emacs user, and a frequent Scala developer. Scala tooling for emacs was restricted to mostly ensime until recently. Although ensime is an excellent piece of software, it made my old Macbook suffer a lot (it only had 8gb of RAM). So, most of the time I just went hardcore mode developer, and worked with no autocompletion, no jump to definition, no-nothing. A pervasive use of ripgrep and good memory were sometimes enough, but I was envious of many things I could see in my colleagues using IntelliJ. Of course, switching editors was not an option.


I looked up what solutions were available, and the only option that seemed good enough was using the classic ctags/etags/gtags I hadn’t used since my C days.

Having a tags implementation can bring an almost-decent jump-to-definition essentially for free: gtags (and the others) work by pre-analysing the data and generating symbol maps which are usually stored as files, the overhead in speed and memory is minimal.

Installing it can get somewhat unintuitive (specially on Mac), since you need very specific settings for it to work with emacs and especially, Scala code.

Start by installing GNU global:
brew install global --with-pygments --with-ctags --with-exuberant-ctags
The key parts is having pygments.
Aside from this you will need to export the following environment variables:
GTAGSCONF=/usr/local/share/gtags/gtags.conf
GTAGSLABEL=pygments
Finally, you need to install (or activate) ggtags-mode. If you use spacemacs, you only need to activate the gtags layer.
All this has become moot as soon as metals has reached a usable state: now you can have a really fast language server for Scala, written in Scala with low memory overhead.
-1:-- gtags (GNU global) in emacs for Scala (Post Ruben Berenguel (noreply@blogger.com))--L0--C0--May 21, 2019 09:42 PM

Emacs Redux: Dealing with Jekyll Post URLs

A while ago I wrote about migrating Emacs Redux from Octopress to Jekyll. While I’m really happy with Jekyll overall, there has always been one thing that frustrated me a bit - namely linking to other posts. The syntax for this is the following:


{% post_url name-of-post %}

I know this probably doesn’t seem like a problem, but posts in Jekyll are usually prefixed with a timestamp (e.g. 2019-05-21-some-post ) which makes it really hard to get the name right without consulting the list of posts first. Luckily for us it’s trivial to write an Emacs command that helps with this.


(defun jekyll-insert-post-url ()
  (interactive)
  (let* ((files (remove "." (mapcar #'file-name-sans-extension (directory-files "."))))
         (selected-file (completing-read "Select article: " files nil t)))
    (insert (format "{%% post_url %s %%}" selected-file))))

Here’s how this command looks like in action:

jekyll_post_url.gif

I don’t know you, but I’m totally loving this. You can easily extend the core idea for all sorts of similar tasks that require transforming a bit the files in the current directory. Here’s a similar helper for dealing with image URLs:

(defun jekyll-insert-image-url ()
  (interactive)
  (let* ((files (directory-files "../assets/images"))
         (selected-file (completing-read "Select image: " files nil t)))
    (insert (format "![%s](/assets/images/%s)" selected-file selected-file))))

Let’s see this in action:

jekyll_image_url.gif

That’s all I have for you today! Meta-X forever!

-1:-- Dealing with Jekyll Post URLs (Post Bozhidar Batsov)--L0--C0--May 21, 2019 03:52 PM

(with-emacs: Show matching lines when parentheses go off-screen

This posts will describe how I display lines matching off-screen parentheses at the top of the window:

demo

The variable blink-matching-paren can be used to configure the display of matching parentheses. If you look at the source, you will see that blink-paren-post-self-insert-function is added to post-self-insert-hook to do the job.

There are two things I would like to change about the way it works:

  1. When the matching paren goes off-screen, I usually look at the top of the window. Thus, this is the place where I would like to display the match information.

  2. The matching information should be displayed after movement commands as well.

First let’s take care of blink-matching-paren:

;; we will call `blink-matching-open` ourselves...
(remove-hook 'post-self-insert-hook
             #'blink-paren-post-self-insert-function)
;; this still needs to be set for `blink-matching-open` to work
(setq blink-matching-paren 'show)

As I already use show-paren-mode to visualize matching parentheses it makes sense to look for a way to hook into it. Because of that I decided to advice show-paren-function:

(let ((ov nil)) ; keep track of the overlay
  (advice-add
   #'show-paren-function
   :after
    (defun show-paren--off-screen+ (&rest _args)
      "Display matching line for off-screen paren."
      (when (overlayp ov)
        (delete-overlay ov))
      ;; check if it's appropriate to show match info,
      ;; see `blink-paren-post-self-insert-function'
      (when (and (overlay-buffer show-paren--overlay)
                 (not (or cursor-in-echo-area
                          executing-kbd-macro
                          noninteractive
                          (minibufferp)
                          this-command))
                 (and (not (bobp))
                      (memq (char-syntax (char-before)) '(?\) ?\$)))
                 (= 1 (logand 1 (- (point)
                                   (save-excursion
                                     (forward-char -1)
                                     (skip-syntax-backward "/\\")
                                     (point))))))
        ;; rebind `minibuffer-message' called by
        ;; `blink-matching-open' to handle the overlay display
        (cl-letf (((symbol-function #'minibuffer-message)
                   (lambda (msg &rest args)
                     (let ((msg (apply #'format-message msg args)))
                       (setq ov (display-line-overlay+
                                 (window-start) msg ))))))
          (blink-matching-open))))))

To create the overlay I have written the following helper function:

(defun display-line-overlay+ (pos str &optional face)
  "Display line at POS as STR with FACE.

FACE defaults to inheriting from default and highlight."
  (let ((ol (save-excursion
              (goto-char pos)
              (make-overlay (line-beginning-position)
                            (line-end-position)))))
    (overlay-put ol 'display str)
    (overlay-put ol 'face
                 (or face '(:inherit default :inherit highlight)))
    ol))

All that is left is to activate show-paren-mode with the settings you like:

(setq show-paren-style 'paren
      show-paren-delay 0.03
      show-paren-highlight-openparen t
      show-paren-when-point-inside-paren nil
      show-paren-when-point-in-periphery t)
(show-paren-mode 1)

To use the code of this post make sure to evaluate the code with lexical-scope.

-1:-- Show matching lines when parentheses go off-screen (Post clemera)--L0--C0--May 20, 2019 01:09 PM

Emacs Notes: Rectangle Commands: A handy tool for working with multi-columnar / tabular text

A Problem … and a Solution You are given this (1) a b c aaaa bb ccccc aaa bbb cccc aa bb ccc How would you generate this [ a , b , c ] [ aaaa, bb , ccccc ] [ aaa , bbb, cccc ] [ aa , bb , ccc ] A … Continue reading Rectangle Commands: A handy tool for working with multi-columnar / tabular text
-1:-- Rectangle Commands: A handy tool for working with multi-columnar / tabular text (Post Emacks)--L0--C0--May 19, 2019 02:57 PM

Endless Parentheses: What tests you shouldn’t write: an essay on negative tests

Software tests are great! I’m fortunate enough to have only worked with code-bases with reasonable-to-excellent test coverage, and I wouldn’t want to work in a world without tests. In fact, a thoroughly tested system is nothing short of liberating.

That said, tests are not free. I’m not talking about CI time, that is a cost but it’s usually reducible. Nor am I referring to the effort it takes to write the test, that’s a very real cost, but people are usually very mindful of that (it’s easy to take it into account the very real effort you’re having right now).

The cost that people tend to underestimate is the time wasted with false failures.

Let’s get the basics right. Tests are designed to fail. A test that never fails under any circumstance is a useless test. But there are good failures and bad failures. Which gets me to the entire point of this post.

Write tests with real failures

Real failures are good, false failures are bad.
You want a test to fail when you break functionality, not when you harmlessly change code.

Let’s start with a quick Ruby example. Consider a model with an attribute called value. It wouldn’t be surprising to see a test like this for such a model.

it { is_expected.to respond_to(:value) }

If you don’t know Rspec, this is roughly testing that you can call .value on the model.

But when will this test ever fail?
Under any reasonable condition, this will only fail if you rename the column in the database. Nobody will ever do that by accident!

What’s worse, this failure will never carry any useful information. It doesn’t tell the developer that this change is unexpectedly breaking something. All it ever does is give us yet another piece of code to fix while in the middle of an already long refactoring.

And how do we fix it? By editing the test to use the new name.

A false failure is one that you fix by editing the test, while a real failure is one you fix by editing the code.

False failures are unavoidable. Every test is exposed to them, and every time they happen the developer wastes some amount of time identifying and fixing the failure. That is a negative cost that every test carries and not everyone takes into account.

For most cases, we happily pay this cost because not having a test is way worse than having to fix it. Because one real failure preventing a bug from going live outweighs several false failures, adding up to a positive net effect.

But some tests (such as the example above) virtually never have real failures. Without any positive upside to them, they are strictly negative tests.

And how do we avoid negative tests?

Test function, not code

Let’s expand on our previous example.
Rails provides a helpful one-liner to validate the presence of a mandatory attribute.

validates :value, presence: true

That’s fine and good. The problem is when you see a similar one-liner testing that validation.

it { is_expected.to validate_presence_of(:value) }

Pause on that for a moment. We’ve written a spec to test a single line of code.

The only way that can fail is if we rename the attribute or remove the validation. Again, nobody is ever going to do that by accident. We’re not really testing that a specific functionality works as it should, we’re just testing that a particular line of code is written in a particular way.

That is a code test, not a function test, and code tests are negative tests.

A function test is one that verifies non-trivial functionality, functionality that could be accidentally broken by a number of reasons.

Testing the interface of a service, for instance, is basically always good. As there’s usually at least a few branching code paths inside it where one could inadvertently break a branch while editing another or while adding functionality.

Unit tests for simple functions and methods, in my opinion, are not no-brainers. People like to go nuts with them, because they’re easy to write and quick to run (so “why not?”), but a lot of them fall under the category of negative tests.

Unit tests are good when testing some reasonably complicated algorithm, as someone could actually break an edge case while trying to optimize the implementation. And even then, you shouldn’t just write a couple of mindless tests, as they will probably be negative. You should put some effort into figuring out the edge cases and testing them specifically.

Think before you test

Hopefully, you started thinking well before you wrote that first line of code, so there’s no reason to stop now just because you changed from the app/ to the specs/ directory.

Thinking and being mindful of what you’re testing will not only help you avoid negative tests, but will go a long way to making your positive tests more effective at catching the bugs they’re supposed to catch.

Comment on this.

-1:-- What tests you shouldn’t write: an essay on negative tests (Post)--L0--C0--May 19, 2019 12:00 AM

Chen Bin (redguardtoo): My favorite Richard Stallman interviews

I've watched/listened many Richard Stallman interviews. Some hosts are really bad. They know nothing about technology and are too aggressive.

But good interviewers still exist. Here are top 3 interviews I recommend:

-1:-- My favorite Richard Stallman interviews (Post Chen Bin)--L0--C0--May 13, 2019 02:33 PM

Modern Emacs: Testing Emacs programs with Buttercup

Buttercup is a testing framework for emacs-lisp. It is used by large projects like Clojure's CIDER to write clean, concise, and descriptive tests. I introduce Buttercup and build up to advanced usages with the faint, unlikely dream that some Emacs programmer decides to add tests to their library... What is Buttercup? Introduction Buttercup's entry points are: describe, it, and expect. We describe a test suite with a name. Test cases within the possibly nested suites are done with it and assertions as expect blocks within.
-1:-- Testing Emacs programs with Buttercup (Post)--L0--C0--May 13, 2019 12:00 AM