Making it Easier to Minimize Risk of Information Loss

In Emacs, when I’m writing a commit message for a repository backed by version control, and I type #123 the bug-reference package overlays that #123 with links to the remote issue. I can then “click” on #123 and jump to the issue at the remote repository; a convenient feature!

However that convenience comes with a cost, namely those terse references do two things:

• create low-level lock-in
• increase the risk of information loss

If I were to change the host of that repository or transfer ownership, the #123 becomes disconnected from what it once referenced.

I prefer, instead, to use full Uniform Resource Locators (URLs 📖). This way there is no ambiguity about what I’m referencing. Unless of course the remote service breaks links or goes away.

Adding further nuance, due to the nature of my work, I’m often referencing other repository’s issues and pull requests during the writing of a commit.

Enter Automation

I’ve been playing with Completion at Point Functions (CaPFs 📖) and decided the explore automatically creating those URLs. See Completion at Point Function (CAPF) for Org-Mode Links. The end goal is to have a full URL. For example https://github.com/samvera/hyrax/issues/6056.

I broke this into two steps:

• Create a CaPF for finding the project
• Create a CaPF for replacing the project and issue with the URL

I settled on the following feature:

Given that I have typed “/hyr”
When I then type {{{kbd(TAB)}}}
Then auto-complete options should include “/hyrax”

I went a step further in my implementation, when I select the project completion candidate I append a # to that. I end up with /hyrax# and the cursor is right after the # character. From which I then have my second CaPF.

Given that the text before <point> is "/hyrax#123"
When I type {{{kbd(TAB)}}}
Then auto-complete will convert "/hyrax#123"
     to "https://github.com/samvera/hyrax/issues/123"

Code

Create a CaPF for finding the project

First let’s look at the part for finding a project. I do this via jf/version-control/project-capf.

(defun jf/version-control/project-capf ()
  "Complete project links."
  ;; While I'm going to replace "/project" I want to make
  ;; sure that I don't have any odd hits (for example
  ;; "/path/to/file")
  (when (looking-back "[^[:word:]]/[[:word:][:digit:]_\-]+"
                      (jf/capf-max-bounds))
    (let ((right (point))
           (left (save-excursion
                     ;; First check for the project
                   (search-backward-regexp
                    "/[[:word:][:digit:]_\-]+"
                    (jf/capf-max-bounds) t)
                   (point))))
      (list left right
        (jf/version-control/known-project-names)
        :exit-function
        (lambda (text _status)
          (delete-char (- (length text)))
          (insert text "#"))
        :exclusive 'no))))

The above function looks backwards from point, using jf/capf-max-bounds as the bounds of how far back to look. If there’s a match the function then gets the left and right boundaries and calls jf/version-control/known-project-names to get a list of all possible projects that I have on my machine.

The jf/capf-max-bounds function ensures that we don’t attempt to look at a position outside of the buffer. See the below definition:

(cl-defun jf/capf-max-bounds (&key (window-size 40))
  "Return the max bounds for `point' based on given WINDOW-SIZE."
  (let ((boundary (- (point) window-size)))
    (if (> 0 boundary) (point-min) boundary)))

The jf/version-control/known-project-names leverages the projectile package to provides a list of known projects. I’ve been working at moving away from projectile but the projectile-known-projects variable just works, so I’m continuing my dependency on projectile. I want to migrate towards the built-in project package, but there are a few points that I haven’t resolved.

(cl-defun jf/version-control/known-project-names (&key (prefix "/"))
  "Return a list of project, prepending PREFIX to each."
  (mapcar (lambda (proj)
            (concat prefix (f-base proj)))
    projectile-known-projects))

I then add jf/version-control/project-capf to the completion-at-point-functions variable. I also need to incorporate that elsewhere, based on various modes. But that’s a different exercise.

(add-to-list 'completion-at-point-functions #'jf/version-control/project-capf)

The above code delivers on the first feature; namely auto completion for projects that sets me up to deliver on the second feature.

Create a CaPF for replacing the project and issue with the URL

The jf/version-control/issue-capf function below builds on jf/version-control/project-capf convention, working then from having an issue number appended to the text.

(defun jf/version-control/issue-capf ()
  "Complete project issue links."
  ;; While I'm going to replace "/project" I want to make sure that I don't
  ;; have any odd hits (for example /path/to/file)
  (when (looking-back "[^[:word:]]/[[:word:][:digit:]_\-]+#[[:digit:]]+"
          (jf/capf-max-bounds))
    (let ((right (point))
           (left (save-excursion
                   (search-backward-regexp
                     "/[[:word:][:digit:]_\-]+#[[:digit:]]+"
                     (jf/capf-max-bounds) t)
                   (point))))
      (list left right
        (jf/version-control/text)
        :exit-function
        #'jf/version-control/unfurl-issue-to-url
        :exclusive 'no))))

I continue to leverage jf/capf-max-bounds querying for all matching version control text within the buffer (via jf/version-control/text):

(defun jf/version-control/text ()
  "Find all matches for project and issue."
  (s-match-strings-all "/[[:word:][:digit:]_\-]+#[[:digit:]]+" (buffer-string)))

Once we have a match, I use jf/version-control/unfurl-issue-to-url to convert the text into a URL. I had originally tried to get #123 to automatically unfurl the issue URL for the current project. But I set that aside as it wasn’t quite working.

(defun jf/version-control/unfurl-issue-to-url (text _status)
  "Unfurl the given TEXT to a URL.

Ignoring _STATUS."
  (delete-char (- (length text)))
  (let* ((parts (s-split "#" text))
          (issue (cadr parts))
          (project (or (car parts) (cdr (project-current)))))
    (insert (format
              (jf/version-control/unfurl-project-as-issue-url-template project)
              issue))))

That function relies on jf/version-control/unfurl-project-as-issue-url-template which takes a project and determines the correct template for the project.

(cl-defun jf/version-control/unfurl-project-as-issue-url-template (project &key (prefix "/"))
  "Return the issue URL template for the given PROJECT.

Use the provided PREFIX to help compare against
`projectile-known-projects'."
  (let* ((project-path
          (car (seq-filter
                (lambda (el)
                  (or
                   (s-ends-with? (concat project prefix) el)
                   (s-ends-with? project el)))
                projectile-known-projects)))
         (remote
          (s-trim (shell-command-to-string
                   (format
                    "cd %s && git remote get-url origin"
                    project-path)))))
    (s-replace ".git" "/issues/%s" remote)))

And last, I add jf/version-control/issue-capf to my list of completion-at-point-functions.

(add-to-list 'completion-at-point-functions #'jf/version-control/issue-capf)

Conclusion

While demonstrating these functions to a co-worker, I said the following:

“The purpose of these URL unfurling functions is to make it easier to minimize the risk of losing information that might be helpful in understanding how we got here.”

In other words, information is scattered across many places, and verbose URLs are more likely to be relevant than terse short-hand references.

A future refactor would be to use the bug-reference logic to create the template; but what I have works because I mostly work on Github projects and it’s time to ship it. Also, these CaPFs are available in other contexts, which helps with writing more expressive inline comments.

Adding a Function to Help Establish a Habit

Earlier this week my team members began talking prefixing our commit title with the type of commit. The idea being that with consistent prefixing, we can more scan the commit titles to get an overview of what that looks like.

We cribbed our initial list from Udacity Nanodegree Style Guide:

feat

A new feature

fix

A bug fix

docs

Changes to documentation

style

Formatting, missing semi colons, etc; no code change

refactor

Refactoring production code

test

Adding tests, refactoring test; no production code change

chore

Updating build tasks, package manager configs, etc; no production code change

Our proposal was that at the start of next sprint we’d adopt this pattern for one sprint and then assess. We also had a conversation about the fact that those “labels” consume precious space in the 50 character or so title.

So we adjusted our recommendation to use emojis. We established the following:

🎁

feature (A new feature)

🐛

bug fix (A bug fix)

📚

docs (Changes to documentation)

💄

style (Formatting, missing semi colons, etc; no code change)

♻️

refactor (Refactoring production code)

☑️

tests (Adding tests, refactoring test; no production code change)

🧹

chore (Updating build tasks, package manager configs, etc; no production code change)

Which means we were only surrendering 2 characters instead of a possible 8 or so.

Given that we were going to be practicing this, I wanted to have Emacs prompt me to use this new approach.

The jf/version-control/valid-commit-title-prefixes defines the glossary of emojis and their meanings:

(defvar jf/version-control/valid-commit-title-prefixes
  '("🎁: feature (A new feature)"
     "🐛: bug fix (A bug fix)"
     "📚: docs (Changes to documentation)"
     "💄: style (Formatting, missing semi colons, etc; no code change)"
     "♻️: refactor (Refactoring production code)"
     "☑️: tests (Adding tests, refactoring test; no production code change)"
     "🧹: chore (Updating build tasks, package manager configs, etc; no production code change)")
  "Team 💜 Violet 💜 's commit message guidelines on <2023-05-12 Fri>.")

I then added jf/git-commit-mode-hook which is added as find-file-hook This hook is fired anytime we find a file and load it into a buffer. .

(cl-defun jf/git-commit-mode-hook (&key (splitter ":") (padding " "))
  "If the first line is empty, prompt for commit type and insert it.

Add PADDING between inserted commit type and start of title.  For
the `completing-read' show the whole message.  But use the
SPLITTER to determine the prefix to include."
  (when (and (eq major-mode 'text-mode)
          (string= (buffer-name) "COMMIT_EDITMSG")
          ;; Is the first line empty?
          (save-excursion
            (goto-char (point-min))
            (beginning-of-line-text)
            (looking-at-p "^$")))
    (let ((commit-type (completing-read "Commit title prefix: "
                         jf/version-control/valid-commit-title-prefixes nil t)))
      (goto-char (point-min))
      (insert (car (s-split splitter commit-type)) padding))))

(add-hook 'find-file-hook 'jf/git-commit-mode-hook)

The jf/git-commit-mode-hook function delivers on the following two scenarios:

Given I am editing a commit message
When I start from an empty message
Then Emacs will prompt me to select the commit type
And will insert an emoji representing that type

Given I am editing a commit message
When I start from a non-empty message
Then Emacs will not prompt me to select the commit type

Conclusion

This function took about 20 minutes to write and helps me create habits around a new process. And if we agree to stop doing it, I’ll remove the hook (maybe keeping the function).

In this practice time, before we commit as a team to doing this, I am already appreciating the improved scanability of the various project’s short-logs. Further this prompt helps remind me to write small commits.

Also, in exploring how to do this function, I continue to think about how my text editor reflects my personal workflows and conventions.

Peeling Back the Curtain of Some Blogging Wizardy

In Completing Org Links, the author mentioned the following: “I try never to link to something more than once in a single post.”

And I agree!

In a single blog post, I like all of my article’s A-tags to have unique href attributes. See <a>: The Anchor element - HTML: HyperText Markup Language And I also like to use semantic Hypertext Markup Language (HTML 📖), such as the CITE-tag See <cite>: The Citation element - HTML: HyperText Markup Language or the ABBR-tag. See <abbr>: The Abbreviation element - HTML: HyperText Markup Language

In my Org-Mode writing, I frequently link to existing Denote documents. Some of those documents do not have a public Uniform Resource Locator (URL 📖) and others do. During the export from Org-Mode to Hugo, via Ox-Hugo, linked documents that have public URLs will be written up as Hugo shortcodes. And linked documents without public URLs will be rendered as plain text.

The shortcode logic See glossary.html shortcode for implementation details. ensures that each page does not have duplicate A-tags. And in the case of abbreviations, the short code ensures that the first time I render the abbreviation, it renders as: Full Term (Abbreviation) then the next time as Abbreviation; always using the correct ABBR tag and corresponding title attribute.

I also have date links Here I add “date” to the org-link-set-parameters , which export as TIME-tags. See <time>: The (Date) Time element - HTML: HyperText Markup Language And someday, I might get around to writing a function to find the nodes that reference a date’s same year, year/month, and year/month/day.

Another advantage of multiple links in my Org-Mode is that when I shuffle my notes to different files, the backlink utility of Denote and Org-Roam will pick up these new documents

All of this means that my Org-Mode document is littered with links, but on export the resulting to my blog, things become tidier.

So yes, don’t repeat links in blog posts; that’s just a lot of clutter. But for Personal Knowledge Management (PKM 📖), spamming the links helps me ensure that I’m able to find when and where I mention things.

Which is another reason I have an extensive Glossary of Terms for Take on Rules. All in service of helping me find things.

Leveraging cape and org-element-map

I write a lot of things using Org-Mode. One function I have wanted is auto-completion of links that already exist in the current Org-Mode buffer. I have created custom links for abbreviations, epigraphs, dates, and glossary of terms.

Last night I spent a bit of time writing that function. I remembered Org-Roam’s completion functions, so I started there for inspiration.

Writing Some Emacs Lisp

I looked to org-roam-complete-link-at-point for inspiration. I need a function that returns the text of the links. Along with the text, I would need the raw-link.

Below is jf/org-links-with-text, the function I wrote. Here’s the link to jf/org-links-with-text.

(defun jf/org-links-with-text (&optional given-link)
  "Return the `distinct-' `org-mode' links in the
 `current-buffer'.

Each element of the list will be a `propertize' string where the
string value is the text of the link and the \"link\" property
will be the :raw-link.

When provided a GIVEN-LINK stop processing when we encounter the
first matching link."


  (let ((links
	 (org-element-map
	     (org-element-parse-buffer)
	     'link
	   (lambda (link)
	     (when-let* ((left (org-element-property :contents-begin link))
			 (right (org-element-property :contents-end link)))
	       (let ((returning
		      (propertize
		       (buffer-substring-no-properties left right)
		       'link (org-element-property :raw-link link))))
		 (if given-link
		     (when (string= given-link returning)
		       returning)
		   returning))))
	   nil
	   given-link)))
    ;; Ensure that we have a distinct list.
    (if (listp links)
	(-distinct links)
      (list links))))

The above loops through all link elements. Assembling a propertized string with each link it encounters. When provided a given-link it halts processing on the first match. And then returns a list of the matches. I reference Org Element API when writing the function.

Here are some examples of the propertized string section of the code:

• Given [[https://orgmode.org][Org-Mode]] then return the string Org-Mode with a 'link property of https://orgmode.org.
• Given [[denote:20230506T202945][Title of Note]] then return the string Title of Note with a 'link property of denote:20230506T202945.

In other words, the CAPF function I’m developing will handle all Org-Mode style links.

With that function, I turned to the inspiration of the org-roam-complete-link-at-point. Below is the function I wrote. Here’s the link to jf/org-capf-links.

;; Cribbed from `org-roam' org-roam-complete-link-at-point
(defun jf/org-capf-links ()
  "Complete links."
  (when (and (thing-at-point 'symbol)
          (not (org-in-src-block-p))
          (not (save-match-data (org-in-regexp org-link-any-re))))
    ;; We want the symbol so that links such performing completion on
    ;; "org-mode" will look for links with the text of org-mode and
    ;; then replace the text "org-mode" with the returned link.
    (let ((bounds (bounds-of-thing-at-point 'symbol)))
      (list (car bounds) (cdr bounds)
        ;; Call without parameters, getting a links (filtered by CAPF
        ;; magic)
        (jf/org-links-with-text)
        :exit-function
        (lambda (text _status)
          ;; We want the properties of that link.  In the case of one
          ;; match, the provided text will have the 'link property.
          ;; However if the
          (let ((link (car (jf/org-links-with-text text))))
            (delete-char (- (length text)))
            (insert "[[" (get-text-property 0 'link link) "]"
                    "[" text "]]")))
        ;; Proceed with the next completion function if the returned
        ;; titles do not match. This allows the default Org capfs or
        ;; custom capfs of lower priority to run.
        :exclusive 'no))))

The above jf/org-capf-links function has three significant parts:

The (when (and… section guards running in a context where things might get confusing.

The (bounds-of-thing-at-point 'symbol) section checks the current item; I could use either 'symbol or 'word; but 'symbol means I can complete or links that have dashes.

The :exit-function, and this is where I spent significant time. In my first round of testing, I had a simple Org-Mode buffer that had one link. When I called the Completion at Point Function (CaPF 📖) function (via TAB) the lambda’s text parameter would have the propertized value.

However, when I had multiple candidates, and selected one, the lambda’s text parameter would not have the propertized value. Hence, I had to go and find again the property.

Last, I wire this into my Org-Mode. To test the functions prior, I had already done this. I use the Corfu and Cape packages. Below is the configuration for my Org-Mode CaPFs:

  (defun jf/org-capf ()
    "The `completion-at-point-functions' I envision using for `org-mode'."
    (setq-local completion-at-point-functions
      (list (cape-super-capf
              #'jf/org-capf-links
              #'tempel-expand
              #'cape-file))))
(add-hook ‘org-mode-hook #’jf/org-capf)

Conclusion

Given that I write between one thousand and four thousand words per day in Org-Mode and I do a lot of linking to code, glossaries, and external sites, I felt it worth the time and energy to write up a CaPF that could help reduce context shifting.

Now, when I write, I can use my TAB completion to provide link candidates to insert.

Repurposing Muscle Memory at Your Fingertips

Every time you call my name
I hear the angels say
Dig my grave

Yesterday afternoon, I was watching one of Gavin Freeborn’s many Emacs videos. I mentally noted his Org-Mode 📖 code block prompting. It was different from what I had.

Last evening, I dug a bit into his Emacs configuration; and experimented with org-tempo. But it wasn’t quite what I wanted.

So, I started cleaning up my my Emacs configuration, moving some functions I no longer use into a “graveyard” file.

During this process, I was looking at a function I had commented out. It overloaded the tilde (e.g. ~) key in Org-Mode.

When I typed three consecutive tilde it would would replace those three tilde with a source block, position the cursor inside the block and then call org-edit-special to perhaps create a special editing buffer.

Enter the Overload of Back-tick/Grave

After typing the third consecutive back-tick, the following function:

• Deletes the three back-ticks.
• Prompts for the content to insert.
• Positions into the newly inserted content.
• Attempts to edit the content in Org-Mode’s special editing buffer.

(defun dig-my-grave ()
  "Three consecutive graves (e.g. “`”) at the start of the line prompts for
 inserting content.  See `dig-my-grave/templates-alist/org-mode'."
  (interactive)
  (if (or (and (> (point) 3)
            (string= (buffer-substring-no-properties
                       (- (point) 3) (point)) "\n``"))
        ;; Account for starting on the first line
        (and (= (point) 3)
          (string= (buffer-substring-no-properties
                     (- (point) 2) (point)) "``")))
    ;; We have just hit our third back-tick at the beginning of the line.
    (progn
      (delete-char -2)
      ;; I use the alist-get pattern a lot...perhaps a function?
      (let ((value (alist-get (completing-read "Special Content: "
                                  dig-my-grave/templates-alist/org-mode nil t)
                     dig-my-grave/templates-alist/org-mode nil nil #'string=)))
        (cond
          ;; Let's assume that we're dealing with registered org blocks.
          ((stringp value)
            (insert value) (forward-line -1) (org-edit-special))
          ;; Trust the function
          ((commandp value) (call-interactively value))
          ((functionp value) (funcall value))
          ((ad-lambda-p) (funcall value))
          ;; Time for a pull request
          (t (error "Unprocessable value %s for #'dig-my-grave" value)))))
    (setq last-command-event ?`)
    (call-interactively #'org-self-insert-command)))

For the above dig-my-grave function to work, I need map ` to the function. Near complete control over the environment.

(require 'org)
(define-key org-mode-map (kbd "`") #'dig-my-grave)

And below is the list of templates; They represent the vast majority of the blocks I use in Emacs 📖:

(defvar dig-my-grave/templates-alist/org-mode
  '(("Bash" . "#+begin_src bash :results scalar replace :exports both :tangle yes\n#+end_src")
    ("Blockquote" . tempel-insert-blockquote_block)
    ("Details and Summary" . "#+begin_details\n#+begin_summary\n\n#+end_summary\n#+end_details")
    ("Emacs Lisp" . "#+begin_src emacs-lisp\n#+end_src")
    ("Org Structure" . org-insert-structure-template)
    ("Plant UML" . "#+begin_src plantuml\n@startuml\n!theme amiga\n\n@enduml\n#+end_src")
    ("Ruby" . "#+begin_src ruby\n#+end_src")
    ("Update" . tempel-insert-update_block))
  "A list of `cons' cells with `car' as the label and `cdr' as
 the value that we'll insert.  Used as the collection for the
 `dig-my-grave' `completing-read'.")

There are three functions:

tempel-insert-blockquote_quote

A Tempel template to insert a blockquote block that I export to my blog post.

tempel-insert-update_block

A Tempel template to insert an update block that I export to my blog post.

org-insert-structure-template

The build-in Org-Mode function to prompt for registered templates.

You can find the above code in the .emacs.d/dig-my-grave.el file of my Emacs configuration.

Conclusion

Now I can repurpose the ingrained Markdown keystrokes for code-blocks into something useful for my Org-Mode writing that I find easy to remember.

Which has me thinking, what are other ways that I might leverage the above function. The progn function hints at generalization. I suppose I’ll hold this and let my mind wander.

In almost all settings other than Markdown, I don’t type ```; how might I bring this to those other modes. Also, I suspect that I want to add the above to a minor-mode, so that I can disable it; in those few cases where I want to type ```.

Invest in Your Tools

Yesterday afternoon, I listened to Things Your Editor Should Have with Amir Rajan on the Rubber Duck Dev Show.

Amir Rajan outlined four high-level pillars for his software development tools. He needs to be able to…

• See the function called when I press a key.
• Run the function without pressing the key.
• Trivially create my own functions.
• See the source code for any function.

I have previously written about the Principles of My Text Editor and Revisiting the Principles of My Text Editor. And when I heard Amir’s four points I nodded in deep understanding.

As I grow in experience, understanding, and processes, I want my tools to grow with me. And as my growth is unique to myself, there’s an implication that I should be able to extend the tools to match my frame.

I had spent years with Textmate 📖, Sublime Text 📖, and Atom 📖; and as Atom reached end of life, I explored Vim 📖 and Visual Studio Code (VS Code 📖). I wrote two “plugins” for Atom:

• https://github.com/jeremyf/atom-commands
• https://github.com/jeremyf/vibrant-ink-ruby

And forked a language markdown package to add some Hugo 📖 functionality. The process for writing those packages felt obtuse and cumbersome.

I contrast that to my three years working in Emacs 📖. Prior to adopting Emacs, I hadn’t programmed in Lisp 📖 and as of today, I have defined 231 functions in my Emacs configuration. I ran rg "\((cl-)?defun" | wc in the root directory of my Emacs configuration. These functions help me, in small ways, move fluidly in my day to day writing, reading, coding, and testing.

Looking at the Time Investment

Later in the interview the hosts wonder about the time investment of learning a tool versus the pay-off. I disagree with Amir’s sentiment: “If you’re career trajectory is you want to do 10 years of development and then go into lead and management, then this it’s not worth the investment.”

From where I sit, I can easily say Emacs has been worth my investment; and will continue to pay dividends for me for years to come. Because I intend to continue writing and reading.

In fact, if I were coding less, I think Emacs would become even more valuable. I think about Emacs for Writers; when Jay Dixit, a journalist, presented on Emacs for writing. Watching that, I took notes and sharpened my tools. After watching that recorded meetup, I setup my abbrev_defs to improve my auto-correction.

I write so many things in Org-Mode 📖 and use the multitude of export functions to get the right format for my audience. Using my jf/formatted-copy-org-to-html with surprising regularity, as I write notes in Emacs and then paste them into Slack.

Advice for Junior Developers

This is some great advice:

When you’re a junior developer, you don’t have the experience and intuition to make good decisions about how your code needs to be written…Do you invest in design patterns, or elsewhere? You want to invest in your fundamentals.

The fundamentals you have in your control is typing speed; being able to quickly go to code and read it; those mechanical functions.

You’re in your exploration phase. Don’t commit to VS Code; do Vim, do Sublime Text…try it all.

Conclusion

I really appreciate Amir’s four pillars enumeration. It resonates with me and draws into focus what I appreciate about Emacs and did not appreciate about VS Code when I last explored using it.

My advice is to broaden your time-frame of consideration for a “text editor”; in my experience I have bounced between developer, manager, and team lead. One constant has been my need to write. As a developer, I’m often writing code. As a manager, I’m more often writing documents. As a lead, I’m writing reviews, technical documents, and tasks breakdowns.

Post Script

This morning, I had breakfast with my dad. He’s one who’s always loved working in the shop; building things. We got to talking about the tools of our trade. And how I use my tools to make more tools.

He nodded in understanding. Knowing what your tools can do, and using them to make more tools, is a virtuous cycle.

Minor Refinements of a Recently Created Function

On Saturday I wrote about Using Built-in Emacs 29 Tree Sitter Package to Get Qualified Ruby Function Name. Today at work I used my jf/treesit/qualified_method_name function about 15 times. That function grabs the method name and it’s class/module scope.

During the day time, I encountered two edge cases that didn’t work with the implementation I originally wrote. These were self-inflicted edge-cases that related to some idiomatic Ruby 📖. The first edge case was as follows:

module A::B
  def call
  end
end

My original code returned #call.

The other edge case was as follows:

module A
  C = Struct.new do
    def call
    end
  end
end

The original code would return A#call.

I spent a bit of time this evening—five minutes or so—resolving the following test case:

module A::B
  C::D = Struct.new do
    def call
    end
  end
end

The expected result is A::B::C::D#call. Let’s look at the Abstract Syntax Tree (AST 📖):

(program
 (module module
  name: (scope_resolution scope: (constant) :: name: (constant))
  (body_statement
   (module module name: (constant) ; end)
   (assignment
    left: (scope_resolution scope: (constant) :: name: (constant))
    =
    right:
     (call method: (constant) block: . (identifier)
      (do_block do
       (body_statement
	(method def body: (identifier) end))
       body: end))))
  body: end))

I use the following two functions:

(cl-defun jf/treesit/qualified_method_name (&key (type "method"))
  "Get the fully qualified name of method at point."
  (interactive)
  (if-let ((func (treesit-defun-at-point)))
      ;; Instance method or class method?
      (let* ((method_type (if (string= type
                                       (treesit-node-type func))
                              "#" "."))
             (method_name (treesit-node-text
                           (car (treesit-filter-child
                                 func
                                 (lambda (node)
                                   (string= "identifier"
                                            (treesit-node-type
                                             node)))))))
             (module_space (s-join "::"
                                   (-flatten
                                    (jf/treesit/module_space func))))
             (qualified_name (concat module_space method_type
                                     method_name)))
        (message qualified_name)
        (kill-new (substring-no-properties qualified_name)))
    (user-error "No %s at point." type)))
(defun jf/treesit/module_space (node)
  (when-let* ((parent (treesit-parent-until
                       node
                       (lambda (n) (member (treesit-node-type n)
                                           '("class"
                                             "module"
                                             "assignment")))))
              (parent_name (treesit-node-text
                            (car
                             (treesit-filter-child
                              parent
                              (lambda (n)
                                (member (treesit-node-type n)
                                        '("constant"
                                          "scope_resolution"))))))))
    (list (jf/treesit/module_space parent) parent_name)))

The key was adding assignment to the list of parents and scope_resolution to the list of parent’s child nodes to check.

You can see my updated code here.

A Rainy Day Coding Practice Session

When I’m writing about or in Ruby 📖 code, sometimes I want to grab the qualified method name. For example, let’s say I have the following Ruby code:

module Hello
  module World
    def foo
      :bar
    end

    def self.call
      :yup
    end
  end
end

The qualified method name for the method #foo would be Hello::World#foo. The qualified method name for the singleton method .call is Hello::World.call. A Ruby documentation convention is that instance methods are prefix with a # and singleton methods are prefixed with a . or ::.

Using treesit-explore-mode, I was able to quickly refine my recursive queries. Below is treesit’s rendering of the Abstract Syntax Tree (AST 📖) of the above Ruby code:

(program
 (module module name: (constant)
  (body_statement
   (module module name: (constant)
    (body_statement
     (method def body: (identifier)
      (body_statement (simple_symbol))
      end)
     (singleton_method def object: (self) . body: (identifier)
      (body_statement (simple_symbol))
      end))
    body: end))
  body: end))

This afternoon, in a moment of dreary skies and sleeping dogs, I hacked together the following functions:

jf/treesit/qualified_method_name

Copy the qualified method name to the paste buffer (e.g. the kill-ring).

jf/treesit/module_space

Recurse up from a node to create a list of the module/class ancestors.

(require 'treesit)
(cl-defun jf/treesit/qualified_method_name (&key (type "method"))
  "Get the fully qualified name of method at point."
  (interactive)
  (if-let ((func (treesit-defun-at-point)))
      ;; Instance method or class method?
      (let* ((method_type (if (string= type
                                       (treesit-node-type func))
                              "#" "."))
             (method_name (treesit-node-text
                           (car (treesit-filter-child
                                 func
                                 (lambda (node)
                                   (string=
                                    "identifier"
                                    (treesit-node-type node)))))))
             (module_space (s-join "::"
                                   (-flatten
                                    (jf/treesit/module_space func))))
             (qualified_name (concat module_space
                                     method_type
                                     method_name)))
        (message qualified_name)
        (kill-new (substring-no-properties qualified_name)))
    (user-error "No %s at point." type)))

;; An ugly bit of code to recurse upwards from the node to the "oldest"
;; parent.  And collect all module/class nodes along the way. This will
;; return a series of nested lists.  It's on the originating caller to
;; flatten that list.
(defun jf/treesit/module_space (node)
  (when-let* ((parent (treesit-parent-until
                       node
                       (lambda (n) (member (treesit-node-type n)
                                           '("class" "module")))))
              (parent_name (treesit-node-text
                            (car (treesit-filter-child
                                  parent (lambda (n)
                                           (string=
                                            "constant"
                                            (treesit-node-type n))))))))
    (list (jf/treesit/module_space parent) parent_name)))

This is most certainly a rainy day kind of project; one that helped me learn just a bit more about the treesit package.

Postscript

The list returned by jf/treesit/module_space is '(nil ("Hello" ("World"))); which is a ugly but workable. Perhaps someone will write to me with a refactor of this code.

A Rainy Day Coding Practice Session

When I’m writing about or in Ruby 📖 code, sometimes I want to grab the qualified method name. For example, let’s say I have the following Ruby code:

module Hello
  module World
    def foo
      :bar
    end

    def self.call
      :yup
    end
  end
end

The qualified method name for the method #foo would be Hello::World#foo. The qualified method name for the singleton method .call is Hello::World.call. A Ruby documentation convention is that instance methods are prefix with a # and singleton methods are prefixed with a . or ::.

Using treesit-explore-mode, I was able to quickly refine my recursive queries. Below is treesit’s rendering of the Abstract Syntax Tree (AST 📖) of the above Ruby code:

(program
 (module module name: (constant)
  (body_statement
   (module module name: (constant)
    (body_statement
     (method def body: (identifier)
      (body_statement (simple_symbol))
      end)
     (singleton_method def object: (self) . body: (identifier)
      (body_statement (simple_symbol))
      end))
    body: end))
  body: end))

This afternoon, in a moment of dreary skies and sleeping dogs, I hacked together the following functions:

jf/treesit/qualified_method_name

Copy the qualified method name to the paste buffer (e.g. the kill-ring).

jf/treesit/module_space

Recurse up from a node to create a list of the module/class ancestors.

(require 'treesit)
(cl-defun jf/treesit/qualified_method_name (&key (type "method"))
  "Get the fully qualified name of method at point."
  (interactive)
  (if-let ((func (treesit-defun-at-point)))
      ;; Instance method or class method?
      (let* ((method_type (if (string= type
                                       (treesit-node-type func))
                              "#" "."))
             (method_name (treesit-node-text
                           (car (treesit-filter-child
                                 func
                                 (lambda (node)
                                   (string=
                                    "identifier"
                                    (treesit-node-type node)))))))
             (module_space (s-join "::"
                                   (-flatten
                                    (jf/treesit/module_space func))))
             (qualified_name (concat module_space
                                     method_type
                                     method_name)))
        (message qualified_name)
        (kill-new (substring-no-properties qualified_name)))
    (user-error "No %s at point." type)))

;; An ugly bit of code to recurse upwards from the node to the "oldest"
;; parent.  And collect all module/class nodes along the way. This will
;; return a series of nested lists.  It's on the originating caller to
;; flatten that list.
(defun jf/treesit/module_space (node)
  (when-let* ((parent (treesit-parent-until
                       node
                       (lambda (n) (member (treesit-node-type n)
                                           '("class" "module")))))
              (parent_name (treesit-node-text
                            (car (treesit-filter-child
                                  parent (lambda (n)
                                           (string=
                                            "constant"
                                            (treesit-node-type n))))))))
    (list (jf/treesit/module_space parent) parent_name)))

This is most certainly a rainy day kind of project; one that helped me learn just a bit more about the treesit package.

Postscript

The list returned by jf/treesit/module_space is '(nil ("Hello" ("World"))); which is a ugly but workable. Perhaps someone will write to me with a refactor of this code.

When One Pathway Fails, Try Another

This afternoon, I was exploring some Tree Sitter 📖 functions in Emacs 📖. I wanted to take a Ruby 📖’s method’s signature and create Yardoc stubs from the method parameters.

Requirements

Let’s say I have the following Ruby method definition:

def call(foo, bar:, baz: :default, **kwargs)
  # do stuff
end

I wanted to call a function and update the buffer as follows:

##
# @param foo [Object]
# @param bar [Object]
# @param baz [Object]
# @param kwargs [Object]
def call(foo, bar:, baz: :default, **kwargs)
  # do stuff
end

(let ((func-node (tree-sitter-node-at-point 'method))
        (params (tsc-get-child-by-field func-node ':method_parameters)))
    (message "%s" params))

The Interlude and Solution

I was encountering problems with tree-sitter functionality. The tree-sitter package is an external package. The treesit is a built-in package in Emacs 29. I prefer tree-sitter as it’s more performant in my use case. The following emacs-lisp writes a nil message:

(let ((func-node (treesit-inspect-node-at-point 'method))
      (params (tsc-get-child-by-field func-node ':method_parameters)))
  (message "%s" params))

The above, in my reading, should’ve found the node that had the method parameters.

Running into those problems, I took a different path. String parsing and regular expressions. Below is that solution:

(defun jf/ruby-mode/yardoc-ify ()
  "Add parameter yarddoc stubs for the current method."
  (interactive)
  ;; Remember where we started.
  (save-excursion
    ;; Goto the beginning of the function
    (ruby-beginning-of-defun)
    ;; Move to just after the first (
    (search-forward "(")
    ;; Move back to just before the (
    (backward-char)
    ;; Select parameters declaration
    (mark-sexp)
    ;; Copy that
    (copy-region-as-kill (point) (mark))
    ;; Split apart the parameters into their identifiers
    (let ((identifiers (mapcar (lambda (token)
                            (replace-regexp-in-string
                             "[^a-z|_]" ""
                             (car (s-split " "
                                           (s-trim token)))))
                          (s-split "," (substring-no-properties
                                        (car kill-ring))))))
      ;; Go to the beginning of the function again
      (ruby-beginning-of-defun)
      ;; Now insert the identifiers as yardoc
      (insert "##\n"
              (s-join "\n" (mapcar
                            (lambda (param)
                              (concat "# @param "
                                      param
                                      " [Object]"))
                            identifiers))
              "\n"))))

Exploring Functionality by Re-Reading Documentation

When I first adopted the Consult package, I scanned the documentation and setup a basic configuration. I then went about using the basics and hacking functionality. Tonight I spent some time reviewing Consult’s documentation.

Below is the original advice-add for my Consult configuration. The functionality was as follows: when I have text highlighted, use the highlighted text for my line search (e.g. consult-line) and file search function (e.g. consult-ripgrep).

(defun jf/consult-first-param-is-initial-text (consult-fn &rest rest)
  "Advising function around CONSULT-FN.

The CONSULT-FN's first parameter should be the initial text.

When there's an active region, use that as the first parameter
for CONSULT-FN.  Otherwise, use an empty string the first
parameter.  This function handles the REST of the parameters."
  (interactive)
  (apply consult-fn
         (when (use-region-p)
           (buffer-substring
            (region-beginning) (region-end)))
         rest))

(defun jf/consult-ripgrep-wrapper (consult-fn &optional dir given-initial)
  "Advising function around CONSULT-FN.

DIR and GIVEN-INITIAL match the method signature of `consult-wrapper'."
  (interactive "P")
  (let ((initial (list (or given-initial
                           (when (use-region-p)
                             (buffer-substring (region-beginning)
                                               (region-end)))))))
    (apply consult-fn dir initial)))
(advice-add #'consult-line
            :around #'jf/consult-first-param-is-initial-text
            '((name . "wrapper")))
(advice-add #'consult-ripgrep
            :around #'jf/consult-ripgrep-wrapper
            '((name . "wrapper"))))

After reading the Consult README, I removed my customization and added the following:

(consult-customize consult-line
                   consult-ripgrep
                   :initial (when (use-region-p)
                              (buffer-substring-no-properties
                               (region-beginning) (region-end))))

The above consult-customize duplicates many lines of code I had previously written.

There are three distinct benefits:

1. I’m following the documented pattern of customization.
2. I’m not over-riding a method and relying on stable method signatures.
3. I’m writing less code.

Put another way, Daniel Mendler has implemented Consult to be extensible.

Conclusion

I also took this time to review other functionality; learning about consult history functionality. When I run consult-ripgrep, I can invoke consult-history (which I’ve bound to C-c h). This then shows past searches and lets me pick from them.

I’ve been spending time actively thinking about how I’m using my editor:

• Shifting key bindings to ease how my fingers stretch and move.
• Reading package customization options.
• Practicing new to me navigation shortcuts (looking at you Avy 📖).

Why? Because more than ever my writing is my thinking. And improving my reflexes on moving throughout my editor helps reduce mental friction; which for me reduces fatigue and increases the chances of retention during long-running tasks.

A Quick Intro to Other Open Source Note Taking Tools

Over the last year I’ve talked with a few folks looking at improving their note-taking game. As part of those conversations, I try to find out their use-case.

Do they want to get better at writing? Are they looking to synchronize notes across their machine? Are they looking at integration with their code and text editor?

For myself, I want my note taking functionality incorporated in my text editor. That way I can further strengthen existing muscle memory short-cuts and perhaps open pathways to consider more.

Were I not using Emacs 📖 and Org-Mode 📖 (with Denote 📖) I’d consider the following:

NB 📖

a command line and local web note‑taking, book-marking, archiving, and knowledge base application.

Logseq 📖

a privacy-first, open-source knowledge base that works on top of local plain-text Markdown and Org-mode files.

Dendron 📖

an open-source, local-first, markdown-based, note-taking tool.

VimWiki 📖

A personal wiki for Vim—interlinked, plain text files written in a markup language.

Were I using Visual Studio Code 📖, I’d go with Dendron. If I used Vim 📖, I’d consider VimWiki or NB. And if I didn’t write code? I’d consider Logseq.

All of the above options are open-source, something I consider critical to my note-taking. I do not want vendor lock-in for the information I’m gathering, organizing, and referencing.

Further, each of those applications write notes to your file system in a recoverable manner. If the application goes away, you still have the raw files written in a consistent and mostly portable format.

Documenting my High Level Workflow for Bringing the Bits to the Web

As of 2023-03-10 I build Take on Rules via a Static Site Generator (SSG 📖). Hugo 📖 to be precise. I write locally, run a build process which generates static Hypertext Markup Language (HTML 📖) pages. I push those pages to a host which is then serves those via a bare-bones web-server. I cannot express how much I appreciate that my personal blog eschews a complex web application server. My web server has no “keys to the kingdom.” It simply knows about the static pages that I give it to render.

Throughout this post, I reference functions that are in my my Emacs configuration.

Preliminary Work

Throughout my days, I’m reading code, writing code, documentation, and commit messages; I’m thinking through different implementation approaches for processes; reading blog posts around my hobbies and my profession; reading books for learning and pleasure. All of which I might be writing down little pointers or notes to what I am doing.

Those little pointers begin to take shape in my mind.

Composing a Blog Post

In Emacs 📖. I start writing my blog post by calling M-x jf/denote-create--blog-posts--default. I might instead move a note into my blog-post sub-directory; which means it’s something I might make public. This prompts me to:

• Name the blog post
• Creates the Org-Mode 📖 file (using Denote 📖)
• Opens the file in a buffer

While writing I leverage several different commands, some of those are as follows:

jf/denote-link-or-create

Search for and insert a link to an existing document in my Personal Knowledge Management (PKM 📖).

org-insert-link

Pick the type of link, describe it then insert a link to the corresponding resource. Links can be dates, abbreviations, epigraphs, files, Uniform Resource Locators (URLs 📖), and many other things provided by the default Org-Mode.

org-insert-structure-template

To insert src, details, summary, marginnote, inline_comments

org-footnote-action

To create a footnote, which converts into a sidenote when exporting to my blog. See Hacking Org-Mode Export for Footnotes as Sidenotes.

jf/org-capf

Some custom completion at point functions; in particular auto-completing existing abbreviations; beware when I’m in a list the capf does not work.

Another important function is my jf/org-mode/capture/insert-content-dwim. This builds on the Org-Mode clocking function. See Clocking Work Time (The Org Manual) for more documentation on Org-Mode “clocking.” When I start a clock on a blog post, I can use jf/org-mode/capture/insert-content-dwim to quickly capture text and context from other buffers.

Leveraging HTML Semantics via Macro Substitution

Org-Mode is a great markup format, but it doesn’t map to many semantic elements of HTML. Something that I appreciate. And because I’ve chosen semantic markup, I can more readily redesign my site using Cascading Stylesheet (CSS 📖).

I also have several Org-Mode replacement macros. See Macro Replacement (The Org Manual).

cite

This will export the text into a CITE element.

idiomatic

This will export the text into a I element; once meaning italics and now meaning idiomatic.

kbd

This will export the text into a KBD element.

Each of those macros have corresponding Tempel templates which also provide completing-read functionality by looking up all references on my local file-system. See my templates file.

I have a bq template that I can use to generate a BLOCKQUOTE element with cite, url, and attribution attributes. See the bq template definition here.

About Links in My PKM

As of writing this, I have just over 3000 entries in my PKM. 858 of those entries are blog posts I’ve published. In my Denote Emacs Configuration blog post, I wrote about the different domains of my PKM:

blog-posts

the blog posts I write

epigraphs

the epigraphs and quotes I’ve collected

glossary

my link farm of terms and concepts

indices

top-level indices to organize concepts

melange

a general junk drawer

people

people I know or don’t know

scientist

my work

By linking my “to be published” blog posts to internal notes, I can look at my internal notes and see their backlinks. That is what blog posts link to what notes. This is done through the denote-link-backlinks function. See Denote’s documentation on the Backlinks buffer.

Exporting a Blog Post

I use the Ox-Hugo 📖 package for exporting my blog post. I have written several overrides to the export process:

• Configure to export back-ticks (e.g. `) for code blocks, instead of indentations.
• Override foot note generation for side notes.
• Override how I export links, to leverage my glossary and other custom mechanisms.

One advantage in overriding how my blog exports foot notes is that if when I choose to export my blog post into other formats (e.g. a Portable Document Format (PDF 📖) via LaTeX 📖), I still get foot notes.

Reviewing the Blog Post

With my blog post exported, I switch over to ~/git/takeonrules.source/, a Hugo project that builds the static site that is Take on Rules.

I run the following Hugo command: hugo serve -D. The -D flag tells Hugo to build and serve both published and draft pages. This starts a web server which can be accessed at https://localhost:1313. The default port when locally running Hugo. (the default port for Hugo).

Publishing a Blog Post

Having checked the blog post on my local machine, I run the following command: bundle exec rake one_shot_publish.

That command does many things: first, it runs audits to ensure well-formed content. Somewhere around version 0.78.0 of Hugo, I was encountering rendering snafus. If memory serves, it was around the time that the maintainers were switching from one Markdown processor to another. It also fetches data from my PKM and generates metadata for my site. The List of All Tables being one of those. Eventually it creates the public directory and pushes that to my Virtual Private Server (VPS 📖).

Socializing the Blog Post

Sometimes I’ll syndicate to other platforms (e.g. the DEV Community 📖). Other times I’ll post to the corresponding sub-reddit. I might also post a quick link up on Mastodon 📖. Once upon a time, I would also post to Twitter. But I Deactivated my Twitter Account.

Conclusion

I wrote this playbook as a meditation on what steps I take to bring things forward from my personal note taking process into blog posts. These days, the notes I take are now readily available to convert into blog posts for sharing.

Adding a Bit More Color to My Baseline Theme

In this post, I want to show how I added a bit of additional color to my theme for faces assigned by the built-in Emacs 📖 29 treesit package.

Introduction

Since adopting Emacs, I have used the Modus Themes. I appreciate Protesilaos Stavrou (Prot 📖)’s focus on accessibility for the Modus Themes as well as a commitment to accessible documentation.

When you dive deeper into the Modus Themes, you see how customizable and extensible the themes are. I suspect it could be used as a theme-building framework.

In Adding Consistent Color to Emacs Mode Line and iTerm Tab for Various Projects, I showed one way that I’m leveraging Modus Themes to set my mode-line and iTerm2 tab colors by project.

My Emacs Configuration

I’m favoring treesit modes over other modes as Tree Sitter 📖 provides a more robust syntax highlighting.

Below is my treesit and treesit-auto configuration. The treesit package is part of Emacs 29. The treesit-auto package provides automatic translations from non-treesit modes to their corresponding treesit modes. A very helpful package to help bridge other packages that assume one language mode (e.g. ruby-mode) when treesit sets it to a different mode (e.g. ruby-ts-mode).

(use-package treesit
  :custom (treesit-font-lock-level 4)
  :straight (:type  built-in))

(use-package treesit-auto
  :straight (:host github :repo "renzmann/treesit-auto")
  :config (setq treesit-auto-install 'prompt)
  (global-treesit-auto-mode))

Next is the code I use to set my modus-theme. I prefer the “tinted” variation as the day mode (e.g. “operandi”) it’s a bit gentler on my eyes. Most importantly is using modus-themes-load-theme, as that is the function that calls the modus-themes-after-load-theme-hook. More on that in a bit.

(defun jf/emacs-theme-by-osx-appearance ()
  "Set theme based on OSX appearance state."
  (if (equal "Dark" (substring
                     (shell-command-to-string
                      "defaults read -g AppleInterfaceStyle")
                     0 4))
      (modus-themes-load-theme 'modus-vivendi-tinted)
    (modus-themes-load-theme 'modus-operandi-tinted)))

(jf/emacs-theme-by-osx-appearance)

I used the describe-text-properties function to inspect the opening regular expression character. It had the font-lock-regexp-face face.

Likewise, I did the following for Ruby’s string interpolation function (e.g. #{}). That had the face of font-lock-misc-punctuation-face.

Following the Modus Themes documentation, I added the code for setting the foreground color.

(defun jf/modus-themes-custom-faces ()
  (modus-themes-with-colors
    (custom-set-faces
     `(font-lock-misc-punctuation-face
       ((,c :foreground ,green-warmer)))
     `(font-lock-regexp-face
       ((,c :foreground ,red-cooler))))))

(add-hook 'modus-themes-after-load-theme-hook
          #'jf/modus-themes-custom-faces)

Show me the Screenshots

The modus-vivendi-tinted theme with colorized regular expression and string interpolation.

The modus-operandi-tinted theme with colorized regular expression and string interpolation.

An Evening of Hacking to Provide Visual Clues for Varying Contexts

Yesterday, I was pairing with a co-worker and they were sharing their iTerm2 and Visual Studio Code 📖 tab colors. They mentioned that there are plugins for setting the iTerm tab color. Which had me wondering how I might configure my iTerm2 and Emacs 📖 to have the same colors for the same project?

I approached this problem in three steps:

• How do I adjust the mode-line color of Emacs?
• How do I adjust the tab color of iTerm2?
• How do I create a common canonical source for colors?

After an evening of hacking, I have a solution that works well for my use-case.

Adjusting the Mode-line Color of Emacs

For Emacs, I use the Modus Themes by Protesilaos Stavrou 📖. This informed my solution, as I wanted to use the named colors. I love the modus-theme-list-colours function; it was helpful to see the range that I was working with.

I also use projectile, which provides a useful function for determining the project’s root file (e.g. “Where’s the Git 📖 folder located?”)

(require 'modus-themes)
(require 'projectile)
(defvar jf/project/theme-colors/table
  '(("~/git/dotemacs/" . bg-green-subtle)
    ("~/git/dotzshrc/" . bg-green-nuanced)
    ("~/git/takeonrules.source/" . bg-magenta-subtle)
    ("~/git/org/" . bg-sage)
    ("~/git/britishlibrary/" . bg-blue-intense)
    ("~/git/adventist-dl/" . bg-yellow-intense)
    ("~/git/utk-hyku/" . bg-red-intense)
    ("~/git/bulkrax/" . bg-sage))
  "The `car' of each list item should be of begin with \"~/\" and
 end with \"/\" (so as to conform to multiple machines and
 projectile's interface.")

(cl-defun jf/project/theme-colors/current (&key (default 'bg-blue-subtle))
  "Returns a HEX color (e.g. \"#CCDDEE\") for the given project.

The DEFAULT is a named color in the `modus-themes' palette."
  (let* ((project-dir (abbreviate-file-name (or (projectile-project-root) "~/")))
         (name (alist-get project-dir
                          jf/project/theme-colors/table
                          default nil #'string=)))
    (modus-themes-get-color-value name)))

(defun jf/project/theme-colors/apply-to-buffer ()
  "Apply the the project's colors to the buffer (e.g. 'mode-line-active)"
  (unless (active-minibuffer-window)
    (progn
      (face-remap-add-relative
       'mode-line-active
       `( :background ,(jf/project/theme-colors/current)
          :foreground ,(face-attribute 'default :foreground))))))

;; I need to ensure that I'm not doing this while Emacs is initializing.  If I
;; don't have the 'after-init-hook I experience significant drag/failure to
;; initialize.
(add-hook 'after-init-hook
          (lambda ()
            (add-hook 'buffer-list-update-hook
                      #'jf/project/theme-colors/apply-to-buffer)
            (add-hook 'projectile-after-switch-project-hook
                      #'jf/project/theme-colors/apply-to-buffer)))

I did this work in three parts:

1. Change the mode-line-active color once.
2. Change the mode-line-active color based on a named Modus Theme color.
3. Change the mode-line-active color when I changed projects or buffers.

Adjusting the Tab Color of iTerm2

With Emacs resolved, I set about adjust the iTerm2 tabs.

# This function sets the tab color for iTerm based on the "term-color-get"
# results.
function auto_iterm_tag_color_cwd () {
    preline="\r\033[A"
    # Assumes format of `"#aabbcc"'
    hex=`term-color-get`

    first="${hex:0:1}"

    if [ "#" = "$first" ]; then
        hex="${hex:1:6}"
    fi

    hex_r="${hex:0:2}"
    hex_g="${hex:2:2}"
    hex_b="${hex:4:2}"

    rgb_r=`echo $((0x${hex_r}))`
    rgb_g=`echo $((0x${hex_g}))`
    rgb_b=`echo $((0x${hex_b}))`

    echo -e "\033]6;1;bg;red;brightness;$rgb_r\a"$preline
    echo -e "\033]6;1;bg;green;brightness;$rgb_g\a"$preline
    echo -e "\033]6;1;bg;blue;brightness;$rgb_b\a"$preline
}

auto_iterm_tag_color_cwd
autoload -U add-zsh-hook
add-zsh-hook chpwd auto_iterm_tag_color_cwd

In working on this, I brought the solution into two steps:

1. First get the auto_iterm_tag_color_cwd to work with a hard-coded hex value.
2. Create a term-color-get function that would echo a hex value.

Common Canonical Source for Colors

#!/usr/bin/env ruby -wU

# This command is responsible for returning a hex color code, prefixed with the # sign.  It will determine

# The following colors come from the modus tinted color palette.  The names are common across
# modus-vivendi and modus-operandi but the hex colors vary.
COLOR_LOOKUP_LIGHT = {
  "bg-red-intense" => "#ff8f88",
  "bg-green-intense" => "#8adf80",
  "bg-yellow-intense" => "#f3d000",
  "bg-blue-intense" => "#bfc9ff",
  "bg-magenta-intense" => "#dfa0f0",
  "bg-cyan-intense" => "#a4d5f9",
  "bg-red-subtle" => "#ffcfbf",
  "bg-green-subtle" => "#b3fabf",
  "bg-yellow-subtle" => "#fff576",
  "bg-blue-subtle" => "#ccdfff",
  "bg-magenta-subtle" => "#ffddff",
  "bg-cyan-subtle" => "#bfefff",
  "bg-red-nuanced" => "#ffe8f0",
  "bg-green-nuanced" => "#e0f5e0",
  "bg-yellow-nuanced" => "#f9ead0",
  "bg-blue-nuanced" => "#ebebff",
  "bg-magenta-nuanced" => "#f6e7ff",
  "bg-cyan-nuanced" => "#e1f3fc",
  "bg-ochre" => "#f0e0cc",
  "bg-lavender" => "#dfdbfa",
  "bg-sage" => "#c0e7d4"
}

COLOR_LOOKUP_DARK = {
  "bg-red-intense" => "#9d1f1f",
  "bg-green-intense" => "#2f822f",
  "bg-yellow-intense" => "#7a6100",
  "bg-blue-intense" => "#1640b0",
  "bg-magenta-intense" => "#7030af",
  "bg-cyan-intense" => "#2266ae",
  "bg-red-subtle" => "#620f2a",
  "bg-green-subtle" => "#00422a",
  "bg-yellow-subtle" => "#4a4000",
  "bg-blue-subtle" => "#242679",
  "bg-magenta-subtle" => "#552f5f",
  "bg-cyan-subtle" => "#004065",
  "bg-red-nuanced" => "#350f14",
  "bg-green-nuanced" => "#002718",
  "bg-yellow-nuanced" => "#2c1f00",
  "bg-blue-nuanced" => "#131c4d",
  "bg-magenta-nuanced" => "#2f133f",
  "bg-cyan-nuanced" => "#04253f",
  "bg-ochre" => "#442c2f",
  "bg-lavender" => "#38325c",
  "bg-sage" => "#0f3d30"
}

COLOR_REGEXP = %r{\(mode-line-bg-color-name \. ([^\)]+)\)}

# When I have a "light" MacOS setting use the light colors.
table = `defaults read -g AppleInterfaceStyle 2>/dev/null`.strip.size.zero? ? COLOR_LOOKUP_LIGHT : COLOR_LOOKUP_DARK

# Set the default, which maps to my present default setting in Emacs.
color = table.fetch("bg-blue-subtle")

project_theme_colors_filename = File.join(Dir.home, "/git/dotemacs/emacs.d/jf-project-theme-colors.el")
if !File.exist?(project_theme_colors_filename)
  puts color
  exit! 0
end

# Recursively find the most dominant '.dir-locals.el' file in the ancestor directories
slugs = Dir.pwd.split("/")
(0...slugs.size).each do |i|
  filename = File.join(*slugs[0...(slugs.size - i)], ".git")
  next unless File.exist?(filename)
  project_name = filename.sub(Dir.home, "~").sub(/\.git$/, "")

  content = File.read(project_theme_colors_filename)
  match = %r{\("#{project_name}" \. (bg-[^\)]+)\)}.match(content)
  next unless match

  color = table.fetch(match[1], color)
  break
end
puts color

This bit of glue was the easiest to write. I chose to hard code the named color hex representation; as that decoupled me from needing an instance of Emacs running. Yes, sometimes I don’t have Emacs running on my machine.

Citing My Sources

Along the way, I used the following sources for reference to help me build this out:

• elisp - Is there an Emacs hook that runs after every buffer is created? - Stack Overflow
• macos - Different colour for new tabs in iTerm2 - Ask Different
• Modus Themes | Protesilaos Stavrou
• script-commands/hex-to-rgb.sh at master · raycast/script-commands
• Standard Hooks (GNU Emacs Lisp Reference Manual)
• tysonwolker/iterm-tab-colors: Tabs automatically change colors depending on the folder you are in.

Conclusion

I built this up from reading a variety of different sources and then experimenting in an incremental fashion. And now I have a little bit of color in my life to help me visually note when my Emacs buffer and iTerm2 path are pointing to the same project.

Spending a Bit of Time Sharing Some Perhaps Lesser-Known Packages

There are several packages that I consider vital to my Emacs 📖 experience:

• Eglot
• Helpful
• Magit
• Vertico + Consult + Orderless + Marginalia + Embark (these five are often referenced together)
• Org-Mode 📖

Yet those packages are often celebrated and explained. In this blog post I want to dive into some of my lesser sung packages.

The Lesser Sung Heroes of my Emacs Configuration

The following nine packages are just a few from my Emacs configuration:

• Deadgrep
• Denote
• Expand Region
• Fontaine
• Grab Mac Link
• Git Link
• Link Hint
• Titlecase
• TMR

Deadgrep

I only recently brought the Deadgrep package into my Emacs configuration. Previously I had used a combination of the Embark and Wgrep.

Deadgrep provides an interface Via a dedicated buffer to see and edit the results as well as refine the parameters of interaction; thus creating more of a “work space” for the task of finding, refining my criteria, and possibly updating the found entries.

Denote

The Denote 📖 is the back-bone of my Personal Knowledge Management (PKM 📖) system. I once used and still love Org-Roam 📖, but Denote came on the scene, I started exploring. I wrote about Exploring the Denote Emacs Package. By convention, it uses the filename of a note to encode the note’s metadata.

These concessions improves the collection of notes’s portability. And makes extensions on top of those conventions.

Org-Roam and the recent Emacs Knowledge Graph 📖 are interesting alternatives, but thus far, my emerging use-case appears well-served by the Denote. Each of my blog posts are a Denote Org-Mode formatted note that I export to Hugo 📖 via Ox-Hugo 📖. The benefit is that my public and private notes can reference each other, without needing to publish my private notes to the web.

Expand Region

I stumbled upon the expand-region package early in my Emacs adoption. I use it frequently to expand or contract my current selection to the next semantic selection.

In text files the semantic units are:

• word
• sentence
• paragraph
• document

In programming files those units are:

• word/symbol
• function definition
• class definition
• file

This helps me quickly select for cut and paste, formatting, extraction, or whatever.

Fontaine

The Fontaine package helps me manage my fonts for different modes of work; I’m often coding and blogging in my editor, so I don’t switch fonts much. But I find it useful when switching to an extended pairing session or presenting a “slide deck” in Emacs.

It’s a narrow focused package that gets things very right.

Git Link

I use the Git Link package all of the time: when I’m pairing, writing a blog post, writing up a commit message, reporting an issue, trying to help a team member orient to the code.

I like that git link creates a link that is for the current Secure Hash Algorithm (SHA 📖) of my local repository; which helps create a more durable context.

I use the Git Link package in my Custom Org-Mode Capture Function for Annotating Bad Code.

Grab Mac Link

I use the Grab Mac Link package to help me quickly grab the current “page” of a Macintosh Operating System (MacOS 📖) application and write it as a link in the current document.

I have further extended the Grab Mac Link package to quickly file the “page” away as a Denote note. Thus allowing me to:

• Annotate and comment on that page
• Find back-links to that resource

All in service of helping me grow my digital garden.

Link Hint

I use the Link Hint package to quickly open links found in my text editor. When I invoke the link-hint-open-link, it finds all visible links in my editor’s view port.

Where there’s only one link, it opens that link in the registered application (e.g. an http link will open in my Operating System (OS 📖)’s default web browser). Where there are multiple links in the view port, each link will be annotated with a letter. When I type one of the annotated letter’s, the link opens in the registered application.

Titlecase

Sometimes I’m surprised how much I use the titlecase package. Often when creating blog post titles and sub-titles, but also simply when working on local reports, estimates, and other documents that have headlines.

This package allows you to specify a title style guide from the following list:

• AMA Style
• AP Style
• APA Style
• Bluebook Style
• Chicago Style
• IMDB Style
• MLA Style
• New York Times Style
• Sentence style
• Wikipedia Style

I’ve chosen the Wikipedia Style for my default title case behavior. I invoke titlecase-dwim “Do What I Mean” function (DWIM 📖) functions in Emacs are typically a function that attempts to determine the current context and then dispatches to the function most related to the current context. There is titlecase-line, titlecase-region, and titlecase-sentence. The titlecase-dwim function determines which one to call. and presto, I have a conformant title.

TMR

I spend a lot of time in Emacs. And have grown to appreciate the TMR May Ring 📖 (Also Known As (aka 📖) “tmr”).

I can easily ignore my calendar application adding a notification. Yet when Emacs alerts me to something, I pay attention.

I use the TMR May Ring to set a timer in Emacs, providing an optional description. And when the timer stops, Emacs sends me a notification.

Conclusion

My Emacs configuration reflects my personal understanding and usage of Emacs. It’s a place where I maintain, grow, and groom the functions that help me better tell my computer what to do.

I also use this configuration as a chance to explore and practice a Lisp 📖 dialect.

Post Script

While writing this blog post, I made extensive use of my jf/menu–org-capture-safari function:

(defun jf/menu--org-capture-safari ()
  "Create a `denote' entry from Safari page."
  (interactive)
  (require 'grab-mac-link)
  (let* ((link-title-pair (grab-mac-link-safari-1))
         (url (car link-title-pair))
         (title (cadr link-title-pair)))
    (jf/denote-capture-reference :url url :title title)))

This helped me create a Denote note for each of the packages. Now, as I write more about those packages (or not), I’ll be able to build a list of references.

Further, if I were to move back towards a Literate Programming Emacs configuration, I could then use those notes to be the source of my configuration code.

An odd realization that I don’t plan on acting on, but I can begin to intuit how I might prefer that.

The above function, along with its siblings and cousins, helps ease my decision-making in regards to “Where do I put this thing that I want to later file away?”

Drawing Inspiration from Others

Today I read Enhance Emacs Ergonomics Under X: Happy Hands!; the post provides the following executive summary:

Describes customizations under X that makes my hands happier. Leverages features and (mis-features) of modern-day laptops to break old habits and reduce chording in common cases.

The author uses XCape to overload their Command, Control, and Alt keys.

I was curious about how I might do this in my MacOS setup. I use Karabiner-Elements 📖 for minor modifications. Prior to this post, the only mapping I had was for Left Shift + Right Shift to toggle Caps Lock.

Looking at the Karabiner repository and building from some examples I made the following complex modifications:

• Left Command pressed alone maps to Left Control + c
• Left Option pressed alone maps to Left Option + x
• Left Control pressed alone maps to Left Control + x
• Right Option pressed alone maps to Right Option + x
• Right Command pressed alone maps to Right Control + u

Those four modifier keys are the launching point for many of my Emacs 📖 commands. With a bit of practice, we’ll see how this works for me.

Something Borrowed, Something Old, and Something New

This post follows-up on my Thinking Through Capturing and Annotating “Bad” Code post by describing functionality. Then I dive into the implementation details.

Articulating Functions

I will write to a file that is in version control, it will likely be a Denote Org-Mode 📖 file.

I want to capture two layers. Starting from the inside, there is the Code level. The data (and metadata) for that region is:

Function name

The name of the function; my dream state would be to use Tree Sitter 📖 to get the correct scope.

File name

Where on my machine is this code.

Git link reference

What is the URL to the code in it’s current state.

Code-block

What’s the code

The outer layer is the Example level. The data (and metadata) for that region is:

State

For example TODO.

Title

The terse and yet unique name (default to timestamp).

Introduction

Explaining the situation that brought me there.

Code Block(s)

One or more Code levels (see above).

Discussion

What’s up with the code.

Resolution

After some time what did the refactor look like. What were the pull requests/issues filed.

I won’t directly create the Example level, instead I will rely on creating a Code level, which will be part of a Example.

Creating the Code has three options for:

• Assign to the current Example
• Prompt for a Example
• Create a new Example

The Example will require additional writing, but in the interest of expediency, I plan to quickly file it away.

I would like the ability to have a minor mode, that when active, calls attention to the fact that the current buffer has one or more Code levels in my catalog.

At this point, I want to revisit my capture tool chain. It’s already inserting the note at the right location. I assume that org-mode will provide lots of tooling.

Digging into Impelementation Details

Yesterday evening, I spent a few hours building out Org-Mode capture templates and functions.

There were four tasks:

• Prompt for the Example in which to write the Code.
• Position point (e.g. the cursor) in the target file according to the prompted value.
• Copy the content.
• Paste the content into the file.

I refined the structure of the Example to the following:

** TODO ${example} :${tag}:

*** TODO Context

*** Code :code:

*** TODO Discussion

*** COMMENT Refactoring

By establishing a structure, I could settle on how I would insert content. I also reviewed Org-Mode’s documentation and some of my existing code.

I then started writing jf/org-mode/capture/prompt-for-example. That function prompts for the target Example where I file away the Code. Below is a copy of that code:

(cl-defun jf/org-mode/capture/prompt-for-example
    (&optional given-mode &key (tag "example"))
  "Prompt for the GIVEN-MODE example."
  (let* ((mode (or given-mode (completing-read "Example:"
                                               '("Existing" "New" "Stored")))))
    (cond
     ((string= mode "New")
      (let ((example (read-string "New Example Name: "
                                   nil
                                   nil
                                   (format-time-string "%Y-%m-%d %H:%M:%S"))))
        (with-current-buffer (find-file-noselect
                              jf/org-mode/capture/filename)
          (end-of-buffer)
          (insert (s-format jf/org-mode/capture/example-template
                            'aget
                            (list (cons "example" example) (cons "tag" tag))))
          example)))
     ((string= mode "Existing")
      (with-current-buffer (find-file-noselect
                            jf/org-mode/capture/filename)
        (let ((examples (org-map-entries
                         (lambda ()
                           (org-element-property :title (org-element-at-point)))
                         (concat "+LEVEL=2+" tag) 'file)))
          (if (s-blank? examples)
              (jf/org-mode/capture/prompt-for-example "New" :tag tag)
            (completing-read "Example: " examples nil t)))))
     ((string= mode "Stored")
      (or jf/org-mode/capture/stored-context
          (jf/org-mode/capture/prompt-for-example "Existing" :tag tag))))))

I then wrote jf/org-mode/capture/set-position-file to find where to position point.

(cl-defun jf/org-mode/capture/set-position-file
    (&key
     (headline (jf/org-mode/capture/prompt-for-example))
     (tag "code")
     (parent_headline "Examples"))
  "Find and position the cursor at the end of HEADLINE.

The HEADLINE must have the given TAG and be a descendant of the
given PARENT_HEADLINE.  If the HEADLINE does not exist, write it
at the end of the file."
  ;; We need to be using the right agenda file.
  (with-current-buffer (find-file-noselect jf/org-mode/capture/filename)
    (setq jf/org-mode/capture/stored-context headline)
    (let* ((existing-position (org-element-map
                                  (org-element-parse-buffer)
                                  'headline
                                (lambda (hl)
                                  (and (=(org-element-property :level hl) 3)
                                       (member tag
                                               (org-element-property :tags hl))
                                       (string= headline
                                                (plist-get
                                                 (cadr
                                                  (car
                                                   (org-element-lineage hl)))
                                                 :raw-value))
                                       (org-element-property :end hl)))
                                nil t)))
      (goto-char existing-position))))

I then modified an existing function, renaming it to jf/org-mode/capture/get-content. That function grabs the content (and metadata) of the selected region. For this function, I drew inspiration from Capturing Content for Emacs.

(cl-defun jf/org-mode/capture/get-content (start end &key (include-header t))
  "Get the text between START and END returning an `org-mode' formatted string."
  (require 'magit)
  (require 'git-link)
  (let* ((file-name (buffer-file-name (current-buffer)))
         (org-src-mode (replace-regexp-in-string
                        "-\\(ts-\\)?mode"
                        ""
                        (format "%s" major-mode)))
         (func-name (which-function))
         (type (cond
                ((eq major-mode 'nov-mode) "QUOTE")
                ((derived-mode-p 'prog-mode) "SRC")
                (t "SRC" "EXAMPLE")))
         (code-snippet (buffer-substring-no-properties start end))
         (file-base (if file-name
                        (file-name-nondirectory file-name)
                      (format "%s" (current-buffer))))
         (line-number (line-number-at-pos (region-beginning)))
         (remote-link (when (magit-list-remotes)
                        (progn
                          (call-interactively 'git-link)
                          (car kill-ring)))))
    (concat
     (when include-header
       (format "\n**** %s" (or func-name
                               (format-time-string "%Y-%m-%d %H:%M:%S"))))
     "\n:PROPERTIES:"
     (format "\n:CAPTURED_AT: %s" (format-time-string "%Y-%m-%d %H:%M:%S"))
     (format "\n:REMOTE_URL: [[%s]]" remote-link)
     (format "\n:LOCAL_FILE: [[file:%s::%s]]" file-name line-number)
     (when func-name (format "\n:FUNCTION_NAME: %s" func-name))
     "\n:END:\n"
     (format "\n#+BEGIN_%s %s" type org-src-mode)
     (format "\n%s" code-snippet)
     (format "\n#+END_%s\n" type))))

I refined and refactored my jf/org-mode-capture/insert-command; it started as the function to write content to my current clock but now serves as either write to clock or write to my back log file.

(cl-defun jf/org-mode/capture/insert-content (start end prefix)
  "Capture the text between START and END.  When given PREFIX capture to clock."
  (interactive "r\np")
  (let* ((capture-template (if (= 1 prefix) "c" "C"))
         (include-header (if (= 1 prefix) t nil))
         ;; When we're capturing to clock, we don't want a header.
         (text (jf/org-mode/capture/get-content start end
                                                  :include-header
                                                  include-header)))
    (org-capture-string text capture-template)))

And last, here are the declarations of those two capture templates. I’d imagine there’s a way I could leverage the same capture template by using a different capture function. But for now, I’ll settle for duplication.

("c" "Content to Backlog"
 plain (file+function
        jf/org-mode/capture/filename
        jf/org-mode/capture/set-position-file)
 "%i%?"
 :empty-lines 1)
("C" "Content to Clock"
 plain (clock)
 "%i%?"
 :empty-lines 1)

Conclusion

I put this forward for others to stumble upon my approach. And as I was writing this, I realized “Wow, I’m copying a lot of code to this blog post and what I just wrote should be able to help with that.”

Another future project.

Now, it’s time to start capturing “bad” code and start writing about that.

First Steps in Following up on Morning Coffee Talk Inspiration

One of the absolute joys of working from home is my morning ritual with Jenny and our dogs. I’m the first one up, I tend to all the initial dog things. Taking them outside to relieve themselves. Feeding them breakfast.

I brew up coffee and sometimes do the dishes, maybe some exercise. An hour or so later, Jenny comes downstairs with the dogs so eager to see her.

Jenny grabs a coffee mug, heads to her seat across from me in our living room. She sits down and immediately Lacey, one of our border collies, jumps on her lap and begins the morning cuddles.

We’ll sip coffee, go for a walk, play with the dogs, all while chatting about anything, everything, and/or nothing.

Paying Attention to Jagged Edges

One of the topics that tends to come up is customer experience. Moments where we experienced good or bad customer experience. We celebrate the good and lament the bad while looking at both to see what lessons we can learn. Jenny owns Soapy Gnome 📖, so the customer service experience is always at the front of her mind.

This morning, I shared with her that I was looking at some code that would cause both a bad end-user experience and create an internal error message that was unhelpful in triaging the problem. Ultimately creating undo toil and frustration.

In that moment Jenny asked, “What if each time you encountered code such as this you noted it? You could then blog about it, or publish a book on these things, or share them with your team.”

Almost immediately my brain said, “Hey that’s a really good idea. You already wrote a function that does some of the heavy lifting. You could modify that code to help with the capture and organization.”

And I leapt into writing up what this might look like.

Penciling in an Idea

The aforementioned function is a different take on jf/capture-region-contents-with-metadata.

Below is the user story I wrote up. Using the Gherkin syntax.

Given an active region (or line) in the current buffer
  And a configured target file
When I execute the “capture” command
Then the capture will write to the target file
  a section that “notes” the active region

What does that section look like?

* Subject :org:repo:
:PROPERTIES:
:PROJECT: https://github.com/org/repo/
:END:

** Context

# What brought me to this neighborhood?  Introduce the part of the story that
# lead me to the code.

** Code
:PROPERTIES:
:LOCAL_PATH: [[file:~/git/repo/path/to/file]]
:END:

# BEGIN “repeatable” region;
#
# I want an optional parameter to append this region with a header so I can have
# the local file metadata for each of the regions.

#+begin_marginnote
See (remote) [[https://domain.com/org/repo/path/to/file]] for context.
#+end_marginnote

#+begin_src language
The code
#+end_src
# END “repeatable” region

** Discussion

# Why am I singling out this code?  What am I seeing as the problem?

** Refactoring

# This is for later revisitation.  But with the above clues, hopefully things are in good shape.

One thing I want to further explore, before I dive into implementation, is how I might see the note I just wrote as an annotation of the code. That is, while I’m looking at the code-base, when I’m in a file that has one of these sections, I would see some indicator that allowed me to jump to those annotations.

Why? So I can fill out the Refactoring section.

Conclusion

In years past, the above question and proposal would’ve felt daunting. In part because I hadn’t spent much time configuring and extending my text editors. But with Emacs 📖 the above functionality seems like it will be quick to make; perhaps as simple as a capture template for Org-Mode 📖.

This again speaks to the virtuous cycle I’ve experienced in both sharpening my tools (e.g. extending Emacs) and using Emacs for almost all of my writing: code, prose, poetry, blog posts, time sheet entries, and most anything else.

Now to prioritize the time to hack on this. And see what comes. After all, if I make it easy to draw attention to points of refactor, I’m making it easier for me to share the years of learning and knowledge I’ve accumulated in my career as a software developer.

Plus, I can once again hack on Emacs.

Accretion of Functionality Through Crafting and Extending

In Project Dispatch Menu with Org Mode Metadata, Denote, and Transient I wrote about using my Denote 📖 notes, written in Org-Mode 📖, to house metadata related to my project.

The tools been working quite well. I now have a means of grouping like paths, either Uniform Resource Locators (URLs 📖) or local file paths. And in that grouping file, I can add additional metadata. In essence, I have built up a rich bookmark system that overlays my note taking ecosystem.

The dynamism is useful because I also segment my notes. On my work machine I have one set of notes. On my personal machine I have another. But both machines use the same Emacs 📖 configuration.

This morning, I decided to build on this ecosystem to dynamically populate the magit-repository-directories variable.

The magit-repository-directories Variable

The magit-repository-directories variable provides the list of directories to show in the magit-list-repositories buffer. See the Repository List (Magit User Manual).

In brief, the magit-list-repositories shows an overview of the Git 📖 metadata for each of the repositories in the magit-repository-directories list. The metadata, which is customizable, includes the following:

• Version (e.g. Secure Hash Algorithm (SHA 📖) or tag)
• Summary of working directory’s status
• Current branch
• Number of commits behind the remote branch
• Number of commits ahead of the remote branch

From that overview, I can open the magit-status buffer to work on that repository.

What I like about the magit-list-repositories function is that it gives me an overview of what all I might be working on. And a convenient means of quickly moving through those projects.

Dynamically Populating magit-list-repositories

(defvar jf/git-project-paths
  '(("~/git/takeonrules.source/" . 1)
    ("~/git/burning_wheel_lifepaths/" . 1)
    ("~/git/dotzshrc/" .  1)
    ("~/git/dotemacs/" . 1)
    ("~/git/emacs-bookmarks/" . 1)
    ("~/git/org" . 1)
    ("~/git/takeonrules.source/themes/hugo-tufte" . 1))
  "A list of `con' cells where the `car' is the name of a directory
and the `cdr' is a ranking.  I have pre-populated this list with
repositories that are generally available on both machines.")

(defun jf/git-project-paths/dynamic ()
  "Return a list of code repository paths."
  (split-string-and-unquote
   (s-trim-right
    (shell-command-to-string
     (concat
      "rg \"^#\\+PROJECT_PATHS: +[^\\.]+\\. +\\\"(~/git/[^/]+/)\\\"\\)\" "
      "~/git/org --no-ignore-vcs --replace='$1' "
      "--only-matching --no-filename")))
   "\n"))

(dolist (path (jf/git-project-paths/dynamic))
  (add-to-list 'jf/git-project-paths (cons path 1)))

(setq magit-repository-directories jf/git-project-paths)

With the above Emacs Lisp (elisp 📖), I add the repositories that I consider part of my machine’s various project work-spaces. And from there, I can easily get a summary overview of my code.

From Deliberation to Liberation (in the Minutia of My Tools)

I use Emacs 📖 for my text editing. I came to Emacs having used several other text editors; It was a very deliberate decision to adopt Emacs. See Revisiting the Principles of My Text Editor.

Because of my later journey into Emacs relative to all of the other software I’ve used, I’ve grown accustomed to many design decisions. In particular the window/pane/frame management. Emacs is older than all of the other “applications” I use.

This blog post is me setting out to think through and commit to resolving what has remained a somewhat confounding experience.

In a Reddit thread on /r/emacs someone asked a question that was in the same constellation of what had been rattling in my brain. Namely “how do you get Emacs to open a file in the current window?”

FrostyX to create the current-window-only package. Curious, I reviewed the code, installed it, and enabled it. It was close to what I wanted, but had overcompensated. True to the label, everything opened in one window.

The primary workflow that changed was related to Magit.

Prior to current-window-only, when I would commit a change, I had a split window. The top window was where I would write my commit message. The bottom window was where I’d see a magit diff buffer; useful for reviewing changes that were part of the commit.

After enabling current-window-only, when I would commit, two windows would open. The focus started on the magit diff window. I’d close that and then get to the commit buffer. My mind began rationalizing this as acceptable. In part because of some of the reflections and realizations I identified in Using the Git Interactive Staging as a Moment to Facilitate Synthesis.

There were other small alterations, most notable in the help functions. Again, all as advertised.

At the bottom of current-window-only package’s README was the following “See Also” section:

• Demystifying Emacs’s Window Manager - Mastering Emacs
• Emacs: window rules and parameters (`display-buffer-alist’ and extras) - YouTube
• shackle: Enforce rules for popup windows
• emacsorphanage/popwin: Popup Window Manager for Emacs

And I realized up until this point, I hadn’t sought to more deeply understand the configuration, but more importantly I hadn’t articulated my desired state. I intuited what I think I wanted based on prior applications. But with Emacs, I can go beyond other constraints.

In other words, by introducing current-window-only package I realized that I needed to correct that over-simplification, but needed to understand how.

I did a read through of Demystifying Emac’s Window Manager, which was helpful but wasn’t quite sticky. I then watched Protesilaos Stavrou (Prot 📖)’s Emacs: window rules and parameters. In working on this blog post, I just discovered that Prot is now offering private coaching on Emacs, Linux, and Life.

Prot’s video walk through of “here’s the code and here’s what’s changing in Emacs” helped equip me with the language and perspective for better configuring my windows and frames. All of Prot’s packages and presentations deliver a quality knowledge-sharing/learning experience. He provides ample context, wonderful examples, and commentary to help create robust, welcoming, and accessible documentation.

Well, How Did I Get Here?

During the early years of building my Emacs configuration, I used the shackle package to tame some of the behavior. I accepted the foibles of Emacs windows and frames as I added packages to get to a somewhat stable present state.

Put another way, early in my Emacs adoption I was experimenting with things like packages and key bindings. Then I moved into writing functions to further improve my text editing experience.

My present state is that I’m more comfortable with Emacs Lisp (elisp 📖). I have built out functions to improve my workflow. I still have a list of functions I’m considering, but I am in an observation mode to determine if this is where I want to spend time. I frame writing functions, much like writing blog posts, as a time for practice, reflection, and learning.

Which leads me to turning my attention to Emacs windows and frames. To take inventory of “default” behavior, determine if that’s adequate, and then adjust accordingly. The benefit of this kind of ownership is that as new situations arise, I’m prepared to address them.

Conclusion, or To Wax Philosophical

This relates to a passage I recall in Eric Brinde’s book Better Off; an reflective account of joining an Amish community and adopting their lifestyle.

One through line I recall is as follows: This community looked at each bit of technology and determine through deliberation if it is something to bring into their daily lives.

Today, at this point of deliberation, I am again thankful that my text editor of choice, the primary tool for my occupation and vocation, is free, open, and almost unimaginably configurable.

I also marvel at the word “deliberate” itself; especially in relation to “liberate”. Normally the prefix of “de” indicates the “reversal of.” I only have the liberty to change because I had the opportunity to deliberate.

Now, instead of writing about these changes, it’s time to sip some coffee and hack on my text editor.

Yet Another Refinement to My Blogging Engine

Today, I made a slight modification to how I write my blog posts.

I use Org-Mode (Org-mode 📖) to write my blog posts. I use a a modified Tufte CSS theme, derived from Tufte CSS.

My theme has a concept of the side-note and the margin-note. A few years ago, I moved away from Tufte’s preferred fonts, instead letting folks’s browsers determine the font. The margin-note is for general “scribbling” in the margins. And the side-note is analogues to an inline footnote.

Up until today I leveraged Ox-Hugo’s shortcode customizations to handle both. This afternoon, I wrote the below function to replace Ox-Hugo 📖’s export function for footnotes.

(defun jf/org-hugo-sidenote (footnote-reference _contents info)
  "Transcode a FOOTNOTE-REFERENCE element from Org to Hugo sidenote shortcode.
CONTENTS is nil.  INFO is a plist holding contextual information."
  (let* ((element (car (org-export-get-footnote-definition footnote-reference info)))
         (beg (org-element-property :contents-begin element))
         (end (org-element-property :contents-end element)))
    (format "{{< sidenote >}}%s{{< /sidenote >}}"
            (s-trim (buffer-substring-no-properties beg end)))))

;; Over-write the custom blackfriday export for footnote links.
(advice-add #'org-blackfriday-footnote-reference
            :override #'jf/org-hugo-sidenote
            '((name . "wrapper")))

;; Don't render the section for export
(advice-add #'org-blackfriday-footnote-section
            :override (lambda (&rest rest) ())
            '((name . "wrapper")))

With the above function and advice all Org-mode exports, except to my blog, the footnotes retain their original export behavior. I definitely prefer to utilize as much of the native functionality as possible.

This guide gives you a starting point on writing a tree-sitter major mode. Remember, don’t panic and check your manuals!

Build Emacs with tree-sitter

You can either install tree-sitter by your package manager, or from
source:

git clone https://github.com/tree-sitter/tree-sitter.git
cd tree-sitter
make
make install

To build and run Emacs 29:

git clone https://git.savannah.gnu.org/git/emacs.git -b emacs-29
cd emacs
./autogen.sh
./configure
make
src/emacs

Require the tree-sitter package with (require 'treesit). Note that tree-sitter always appear as treesit in symbols. Now check if Emacs is successfully built with tree-sitter library by evaluating (treesit-available-p).

Tree-sitter stuff in Emacs can be categorized into two parts: the tree-sitter API itself, and integration with fontification, indentation, Imenu, etc. You can use shortdoc to glance over all the tree-sitter API functions by typing M-x shortdoc RET treesit RET. The integration are described in the rest of the post.

Install language definitions

Tree-sitter by itself doesn’t know how to parse any particular language. It needs the language grammar (a dynamic library) for a language to be able to parse it.

First, find the repository for the language grammar, eg, tree-sitter-python. Take note of the Git clone URL of it, eg, https://github.com/tree-sitter/tree-sitter-python.git. Now check where is the parser.c file in that repository, usually it’s in src.

Make sure you have Git, C and C++ compiler, and run the treesit-install-grammar command, it will prompt for the URL and the directory of parser.c, leave other prompts at default unless you know what you are doing.

You can also manually clone the repository and compile it, and put the dynamic library at a standard library location. Emacs will be able to find it. If you wish to put it somewhere else, set treesit-extra-load-path so Emacs can find it.

Tree-sitter major modes

Tree-sitter modes should be separate major modes, usually named xxx-ts-mode. I know I said tree-sitter always appear as treesit in symbols, this is the only exception.

If the tree-sitter mode and the “native” mode could share some setup code, you can create a “base mode”, which only contains the common setup. For example, there is python-base-mode (shared), and both python-mode (native), and python-ts-mode (tree-sitter) derives from it.

In the tree-sitter mode, check if we can use tree-sitter with treesit-ready-p, it will emit a warning if tree-sitter is not ready (tree-sitter not built with Emacs, can’t find the language grammar, buffer too large, etc).

Fontification

Tree-sitter works like this: It parses the buffer and produces a parse tree. You provide a query made of patterns and capture names, tree-sitter finds the nodes that match these patterns, tag the corresponding capture names onto the nodes and return them to you. The query function returns a list of (capture-name . node).

For fontification, we simply use face names as capture names. And the captured node will be fontified in their capture name (the face).

The capture name could also be a function, in which case (NODE OVERRIDE START END) is passed to the function for fontification. START and END are the start and end of the region to be fontified. The function should only fontify within that region. The function should also allow more optional arguments with &rest _, for future extensibility. For OVERRIDE check out the docstring of treesit-font-lock-rules.

Query syntax

There are two types of nodes: “named nodes”, like (identifier), (function_definition), and “anonymous nodes”, like "return", "def", "(", ";". Parent-child relationship is expressed as

(parent (child) (child) (child (grand_child)))

Eg, an argument list (1, "3", 1) would be:

(argument_list "(" (number) (string) (number) ")")

Children could have field names:

(function_definition name: (identifier) type: (identifier))

To match any one in the list:

["true" "false" "none"]

Capture names can come after any node in the pattern:

(parent (child) @child) @parent

The query above captures both the parent and the child.

The query below captures all the keywords with capture name
"keyword":

["return" "continue" "break"] @keyword

These are the common syntax, check out the full syntax in the manual: Pattern Matching.

Query references

But how do one come up with the queries? Take python for an example, open any python source file, type M-x treesit-explore-mode RET. You should see the parse tree in a separate window, automatically updated as you select text or edit the buffer. Besides this, you can consult the grammar of the language definition. For example, Python’s grammar file is at

https://github.com/tree-sitter/tree-sitter-python/blob/master/grammar.js

Neovim also has a bunch of queries to reference from.

The manual explains how to read grammar files in the bottom of Language Grammar.

Debugging queries

If your query has problems, use treesit-query-validate to debug the query. It will pop a buffer containing the query (in text format) and mark the offending part in red. Set treesit--font-lock-verbose to t if you want the font-lock function to report what it’s doing.

Set up font-lock

To enable tree-sitter font-lock, set treesit-font-lock-settings and treesit-font-lock-feature-list buffer-locally and call treesit-major-mode-setup. For example, see python--treesit-settings in python.el. Below is a snippet of it.

Note that like the current font-lock system, if the to-be-fontified region already has a face (ie, an earlier match fontified part/all of the region), the new face is discarded rather than applied. If you want later matches always override earlier matches, use the :override keyword.

Each rule should have a :feature, like function-name, string-interpolation, builtin, etc. This way users can enable/disable each feature individually.

Read the manual section Parser-based Font-Lock for more detail.

Example from python.el:

(defvar python--treesit-settings
  (treesit-font-lock-rules
   :feature 'comment
   :language 'python
   '((comment) @font-lock-comment-face)

   :feature 'string
   :language 'python
   '((string) @python--treesit-fontify-string)

   :feature 'string-interpolation
   :language 'python
   :override t
   '((interpolation (identifier) @font-lock-variable-name-face))

   ...))

In python-ts-mode:

(treesit-parser-create 'python)
(setq-local treesit-font-lock-settings python--treesit-settings)
(setq-local treesit-font-lock-feature-list
                '(( comment definition)
                  ( keyword string type)
                  ( assignment builtin constant decorator
                    escape-sequence number property string-interpolation )
                  ( bracket delimiter function operator variable)))
...
(treesit-major-mode-setup)

Concretely, something like this:

(define-derived-mode python-ts-mode python-base-mode "Python"
  "Major mode for editing Python files, using tree-sitter library.

\\{python-ts-mode-map}"
  :syntax-table python-mode-syntax-table
  (when (treesit-ready-p 'python)
    (treesit-parser-create 'python)
    (setq-local treesit-font-lock-feature-list
                '(( comment definition)
                  ( keyword string type)
                  ( assignment builtin constant decorator
                    escape-sequence number property string-interpolation )
                  ( bracket delimiter function operator variable)))
    (setq-local treesit-font-lock-settings python--treesit-settings)
    (setq-local imenu-create-index-function
                #'python-imenu-treesit-create-index)
    (setq-local treesit-defun-type-regexp (rx (or "function" "class")
                                              "_definition"))
    (setq-local treesit-defun-name-function
                #'python--treesit-defun-name)
    (treesit-major-mode-setup)

    (when python-indent-guess-indent-offset
      (python-indent-guess-indent-offset))))

Indentation

Indentation works like this: We have a bunch of rules that look like

(MATCHER ANCHOR OFFSET)

When the indenting a line, let NODE be the node at the beginning of the current line, we pass this node to the MATCHER of each rule, one of them will match the node (eg, “this node is a closing bracket!”). Then we pass the node to the ANCHOR, which returns a point (eg, the beginning of NODE’s parent). We find the column number of that point (eg, 4), add OFFSET to it (eg, 0), and that is the column we want to indent the current line to (4 + 0 = 4).

Matchers and anchors are functions that takes (NODE PARENT BOL &rest _). Matches return nil/non-nil for no match/match, and anchors return the anchor point. An Offset is usually a number or a variable, but it can also be a function. Below are some convenient builtin matchers and anchors.

For MATHCER we have

(parent-is TYPE) => matches if PARENT’s type matches TYPE as regexp
(node-is TYPE) => matches NODE’s type
(query QUERY) => matches if querying PARENT with QUERY
                 captures NODE.

(match NODE-TYPE PARENT-TYPE NODE-FIELD
       NODE-INDEX-MIN NODE-INDEX-MAX)

=> checks everything. If an argument is nil, don’t match that. Eg,
(match nil TYPE) is the same as (parent-is TYPE)

For ANCHOR we have

first-sibling => start of the first sibling
parent => start of parent
parent-bol => BOL of the line parent is on.
prev-sibling => start of previous sibling
no-indent => current position (don’t indent)
prev-line => start of previous line

There is also a manual section for indent: Parser-based Indentation.

When writing indent rules, you can use treesit-check-indent to
check if your indentation is correct. To debug what went wrong, set
treesit--indent-verbose to t. Then when you indent, Emacs
tells you which rule is applied in the echo area.

Here is an example:

(defvar typescript-mode-indent-rules
  (let ((offset 'typescript-indent-offset))
    `((typescript
       ;; This rule matches if node at point is ")", ANCHOR is the
       ;; parent node’s BOL, and offset is 0.
       ((node-is ")") parent-bol 0)
       ((node-is "]") parent-bol 0)
       ((node-is ">") parent-bol 0)
       ((node-is "\\.") parent-bol ,offset)
       ((parent-is "ternary_expression") parent-bol ,offset)
       ((parent-is "named_imports") parent-bol ,offset)
       ((parent-is "statement_block") parent-bol ,offset)
       ((parent-is "type_arguments") parent-bol ,offset)
       ((parent-is "variable_declarator") parent-bol ,offset)
       ((parent-is "arguments") parent-bol ,offset)
       ((parent-is "array") parent-bol ,offset)
       ((parent-is "formal_parameters") parent-bol ,offset)
       ((parent-is "template_substitution") parent-bol ,offset)
       ((parent-is "object_pattern") parent-bol ,offset)
       ((parent-is "object") parent-bol ,offset)
       ((parent-is "object_type") parent-bol ,offset)
       ((parent-is "enum_body") parent-bol ,offset)
       ((parent-is "arrow_function") parent-bol ,offset)
       ((parent-is "parenthesized_expression") parent-bol ,offset)
       ...))))

Then you set treesit-simple-indent-rules to your rules, and call treesit-major-mode-setup.

Imenu

Set treesit-simple-imenu-settings and call treesit-major-mode-setup.

Navigation

Set treesit-defun-type-regexp, treesit-defun-name-function, and call treesit-major-mode-setup.

C-like languages

[Update: Common functions described in this section have been moved from c-ts-mode.el to c-ts-common.el. I also made some changes to the functions and variables themselves.]

c-ts-common.el has some goodies for handling indenting and filling block comments.

These two rules should take care of indenting block comments.

((and (parent-is "comment") c-ts-common-looking-at-star)
 c-ts-common-comment-start-after-first-star -1)
((parent-is "comment") prev-adaptive-prefix 0)

Set c-ts-common-indent-offset,
c-ts-common-indent-type-regexp-alist, and and the following
rules should take care of indenting statements in {} blocks as
well as brackets themselves.

;; Statements in {} block.
((parent-is "compound_statement") point-min c-ts-mode--statement-offset)
;; Closing bracket.
((node-is "}") point-min c-ts-mode--close-bracket-offset)
;; Opening bracket.
((node-is "compound_statement") point-min c-ts-mode--statement-offset)

You’ll need additional rules for “brackless” if/for/while statements, eg

if (true)
  return 0;
else
  return 1;

You need rules like these:

((parent-is "if_statement") point-min c-ts-common-statement-offset)

Finally, c-ts-common-comment-setup will set up comment and filling for you.

Multi-language modes

Refer to the manual: Multiple Languages.

Common Tasks

M-x shortdoc RET treesit RET will give you a complete list.

How to...

Get the buffer text corresponding to a node?

(treesit-node-text node)

Don’t confuse this with treesit-node-string.

Scan the whole tree for stuff?

(treesit-search-subtree)
(treesit-search-forward)
(treesit-induce-sparse-tree)

Find/move to to next node that...?

(treesit-search-forward node ...)
(treesit-search-forward-goto node ...)

Get the root node?

(treesit-buffer-root-node)

Get the node at point?

(treesit-node-at (point))

Emacs’ release branch is now on complete feature freeze, meaning absolutely only bug fixes can happen on it. Now is a good time to talk about the state of tree-sitter in Emacs: what do you get in Emacs 29, what you don’t, and what would happen going forward.

What’s in Emacs 29

From a pure user’s perspective, Emacs 29 just adds some new built-in major modes which look more-or-less identical to the old ones. There aren’t any flashy cool features either. That sounds disappointing, but there are a lot of new stuff under the hood, a solid base upon which exciting things can emerge.

If Emacs 29 is built with the tree-sitter library, you have access to most of the functions in its C API, including creating parsers, parsing text, retrieving nodes from the parse tree, finding the parent/child/sibling node, pattern matching nodes with a DSL, etc. You also get a bunch of convenient functions built upon the primitive functions, like searching for a particular node in the parse tree, cherry picking nodes and building a sparse tree out of the parse tree, getting the node at point, etc. You can type M-x shortdoc RET treesit RET to view a list of tree-sitter functions. And because it’s Emacs, there is comprehensive manual coverage for everything you need to know. It’s in “Section 37, Parsing Program Source” of Emacs Lisp Reference Manual.

Emacs 29 has built-in tree-sitter major modes for C, C++, C#, Java, Rust, Go, Python, Javascript, Typescript, JSON, YAML, TOML, CSS, Bash, Dockerfile, CMake file. We tried to extend existing modes with tree-sitter at first but it didn’t work out too well, so now tree-sitter lives in separate major modes. The tree-sitter modes are usually called xxx-ts-mode, like c-ts-mode and python-ts-mode. The simplest way to enable them is to use major-mode-remap-alist. For example,

(add-to-list 'major-mode-remap-alist
             '(c-mode . c-ts-mode))

The built-in tree-sitter major modes have support for font-lock (syntax highlight), indentation, Imenu, which-func, and defun navigation.

For major mode developers, Emacs 29 includes integration for these features for tree-sitter, so major modes only need to supply language-specific information, and Emacs takes care of plugging tree-sitter into font-lock, indent, Imenu, etc.

Fontification

In tree-sitter major modes, fontification is categorized into “features”, like “builtin”, “function”, “variable”, “operator”, etc. You can choose what “features” to enable for a mode. If you are feeling adventurous, it is also possible to add your own fontification rules.

To add/remove features for a major mode, use treesit-font-lock-recompute-features in its mode hook. For example,

(defun c-ts-mode-setup ()
  (treesit-font-lock-recompute-features
   '(function variable) '(definition)))

(add-hook 'c-ts-mode-hook #'c-ts-mode-setup)

Features are grouped into decoration levels, right now there are 4 levels and the default level is 3. If you want to program in skittles, set treesit-font-lock-level to 4 ;-)

Language grammars

Tree-sitter major modes need corresponding langauge grammar to work. These grammars come in the form of dynamic libraries. Ideally the package manager will build them when building Emacs, like with any other dynamic libraries. But they can’t cover every language grammar out there, so you probably need to build them yourself from time to time. Emacs has a command for it: treesit-install-language-grammar. It asks you for the Git repository and other stuff and builds the dynamic library. Third-party major modes can instruct their users to add the recipe for building a language grammar like this:

(add-to-list
 'treesit-language-source-alist
 '(python "https://github.com/tree-sitter/tree-sitter-python.git"))

Then typing M-x treesit-install-language-grammar RET python builds the language grammar without user-input.

Other features

Things like indentation, Imenu, navigation, etc, should just work.

There is no code-folding, selection expansion, and structural navigation (except for defun) in Emacs 29. Folding and expansion should be trivial to implement in existing third-party packages. Structural navigation needs careful design and nontrivial changes to existing commands (ie, more work). So not in 29, unfortunately.

Future plans

Navigation

The tree-sitter integration is far from complete. As mentioned earlier, structural navigation is still in the works. Right now Emacs allows you to define a “thing” by a regexp that matches node types, plus optionally a filter function that filters out nodes that matches the regexp but isn’t really the “thing”. Given the definition of a “thing”, Emacs has functions for finding the “things” around point (treesit--things-around), finding the “thing” at point (treesit--thing-at-point), and navigating around “things” (treesit--navigate-thing). Besides moving around, these functions should be also useful for other things like folding blocks. Beware that, as the double dash suggests, these functions are experimental and could change.

I also have an idea for “abstract list elements”. Basically an abstract list element is anything repeatable in a grammar: defun, statement, arguments in argument list, etc. These things appear at every level of the grammar and seems like a very good unit for navigation.

Context extraction

There is also potential for language-agnostic “context extraction” (for the lack of a better term) with tree-sitter. Right now we can get the name and span of the defun at point, but it doesn’t have to stop there, we can also get the parameter list, the type of the return value, the class/trait of the function, etc. Because it’s language agnostic, any tool using this feature will work on many languages all at once.

In fact, you can already extract useful things, to some degree, with the fontification queries written by major modes: using the query intended for the variable query, I can get all the variable nodes in a given range.

There are some unanswered questions though: (1) What would be the best function interface and data structure for such a feature? Should it use a plist like (:name ... :params ...), or a cl-struct? (2) If a language is different enough from the “common pattern”, how useful does this feature remains? For example, there isn’t a clear parameter list in Haskell, and there could be several defun bodies that defines the same function. (3) Is this feature genuinely useful, or is it just something that looks cool? Only time and experiments can tell, I’m looking forward to see what people will do with tree-sitter in the wild :-)

Major mode fallback

Right now there is no automatic falling back from tree-sitter major modes to “native” major modes when the tree-sitter library or language grammar is missing. Doing it right requires some change to the auto-mode facility. Hopefully we’ll see a good solution for it in Emacs 30. Right now, if you need automatic fallback, try something like this:

(define-derived-mode python-auto-mode prog-mode "Python Auto"
  "Automatically decide which Python mode to use."
  (if (treesit-ready-p 'python t)
      (python-ts-mode)
    (python-mode)))

Other plans

Existing tree-sitter major modes are pretty basic and doesn’t have many bells and whistles, and I’m sure there are rough corners here and there. Of course, these things will improve over time.

Tree-sitter is very different and very new, and touches many parts of Emacs, so no one has experience with it and no one knows exactly how should it look like. Emacs 29 will give us valuable experience and feedback, and we can make it better and better in the future.

If you are interested, get involved! Read Contributing to Emacs for some tips in getting involved with the Emacs development. Read Tree-sitter Starter Guide if you want to write a major mode using tree-sitter. And of course, docstrings and the manual is always your friend. If you have questions, you can ask on Reddit, or comment in this post’s public inbox (see the footer).

Revisiting Past Writings, Further Describing Personal Workflow, and Issuing and Admonition

Long ago, I started writing an ebook for my “Take on Development”. It sits fallow.

One chapter that I continue to think about is the Modes of Development; in particular the section “Modes that I can think of”.

As I was preparing for the day’s work, I recognized a new mode of thinking. It’s the mode of work that comes before Writing a Commit Message. But before I dive into that, let’s first look at Writing a Commit Message.

Writing a Commit Message

Goal

To create a helpful message to future developers

Method

Follow a well established pattern for commit messages; 50 character first line, references to issue tracker incidences, meaningful message body if the concept is complicated. Unless self evident, the message should describe the state prior to the commit, then after the commit, and provide insight into why you made the change.

Caveat

If I am working on a feature that will require multiple commits, I will make "disposable commits" along the way. These commits are mental placeholders. I will revisit them as I prepare a branch for merging.

At this point in my workflow, I’m explaining what I’m sending forward into the future. Yet there’s an assumption and a mode I haven’t identified. And it maps to an extent Git 📖 commands: git add. The git add command is how you stage your changes, the changes that you’ll describe with your commit message.

In the years since writing the chapter, I’ve adopted a new approach. One that’s powered by Magit. But first let’s talk about the new Modes of Development:

Staging the Commit

Goal

To ensure that I’m sending forward the “right” stuff.

Method

Using the equivalent of git add --interactive (see Interactive Staging), I review the chunks of change (e.g. each diff section).

During this mode, I’m reviewing what has changed and explicitly accepting, reverting, or skipping the possible changes. I’m switching from writing’s author to it’s editor.

While writing the commit message, I’m moving from to publisher; writing the press release and headline.

As eager as I am to just publish this change, slowing down and following these steps provides opportunities for improved understanding.

I’ve caught bugs in the editing and publishing stages, because I’ve stopped coding. While staging the commit, I’m reflecting on the specific code I’ve written. And while writing the commit message, I’ve reflected not on the specific code, but the changing state of the code.

Put another way, this is about the process of synthesis; of reflecting on the written code and writing about what’s changed and why.

Using Tools for Interactive Staging

When I first started using Git, when I was preparing to write a commit, I’d run git status. This command would show me the files I was changing. I’d then run git add my/filename to add one file.

In interactive staging, you’re instead reviewing and operating on the chunks of changes grouped by file. As I mentioned before, I use Magit for this work.

First and foremost, I devote my full attention to this; it’s the only window I have open. And the only thing in that window is the list of untracked and unstaged changes (e.g. the default Magit dispatch buffer).

Why is this important? Focus. I’m no longer writing. I’m reviewing changes. I’m in a different frame of mind; one that deserves focus.

I don’t want the rest of my editing ecosystem—such as minimaps, terminal sessions, project navigation trees, and other tabs with code in them—cluttering my current task.

Next, I begin reviewing the diff of the changed files. Within Magit I can stage, skip, or even revert each chunk. I can also highlight lines within a diff to just stage or revert those lines. Surgical precision! As I complete my actions on one region, Magit advances me to the next.

Sometimes, I’ll do all of this rather quickly; if I haven’t had any disruptions from when I first started writing the code. Responding to Slack, jumping to another project, stepping away for the evening all can count as distractions. When that is done, I begin shifting modes. Now I’m ready to begin Writing a Commit Message. For this I also use Emacs 📖; if I’m going to write something, I want to do it in my text editor!

Commit Hygiene Admonition

Use a proper editor for writing your commit message. The git commit -m "Fixing broken template" is thumbing your nose at future maintainers. If this is an important change, take time to describe and summarize it. The summary is the commit’s title (e.g. the first line). The body is the lines that come after.

I’ve wrote more about this in Commit Messages Do Matter and Leveraging Commit Messages as Pull Request Text. Too Long; Didn't Read (TL;DR 📖) All of the major remote Git services will use your commit’s title as the title of the Pull Request (PR 📖) and the body of the PR as the text of the pull request.

And to follow the principles of Publish (on your) Own Site, Syndicate Elsewhere (POSSE 📖) and Don't Repeat Yourself (DRY 📖), write those messages to the Git repository. Let them travel with you and future code maintainers.

I cannot emphasize the above enough.

But to add some additional context consider the following: Have you ever wanted to get better at using your text editor or Integrated Development Environment (IDE 📖)?

If the answer is yes, then one way to do that is to use it. And if you’re not using it to write commit messages, this is a golden opportunity to add another place of practice.

Take the time to do this. Think of your text editor as a mental muscle that you exercise and strengthen and keep flexible.

For myself, I’m always hungry to move new processes and approaches into Emacs; because I have seen the dividends in the output and I assume quality of my writing. I mention this in passing in Wrapping up Another Year for Take on Rules. Here are some posts that touch on my thoughts regarding my text editor:

• Principles of My Text Editor
• Revisiting the Principles of My Text Editor
• On Sharpening Your Tools

Conclusion

Modernity fills life with distractions; ever hungry for our attention. Consider the state of mind your work requires and establish procedures rituals to get you there.

Want to see how I do the interatcive staging? Contact me and I could do a screen share and walk through this. I’d probably consider recording that session so we could share it out further.

And while I favor Emacs, I know that this approach is most certainly available in your editor of choice. Go forth and practice.

John McCarthy, the progenitor of LISP is quoted to have said

An excessive knowledge of Marxism is a sign of a misspent youth.

I can’t find any good source for this quote, but I can think of a contemporary variation

An excessive knowledge of Lisp is a sign of a misspent youth.

My first encounter with “Lisp” was exactly 8 years ago, when a college of my father lent me his copy of The Structure and Interpretation of Computer Programes. I did not understand it immediately, and I can now reflect on the manifestation of my misunderstandings by reading through a failed implementation I wrote as a teenager (it also demonstrates my lack of understanding C). Eventually I grokked Scheme, Common Lisp and Emacs Lisp (and C). The last of these three has become the one I am the most familiar and the most comfortable using. For the fun of it, I recently wrote a 16-page article on a simple Lisp macro, just to comment on various aspects of the family of languages.

Over time I began contributing to emacs-devel, the development mailing list and besides maintaining my own packages, I tried to help with the popularisation of NonGNU ELPA by collecting packages and reviewing submissions. I contributed a few features to core Emacs such as IRCv3 support for rcirc and recently support for installing packages directly from source using package.el.

While I do think that this has been a valuable experience, I do feel a certain embarrassment when reflecting upon the time I have spent on Emacs and related technologies. Combine this with the fact that I am coming close to the end of my studies, one has to acknowledge that (Emacs) Lisp is not an active research topic. While my interests lie in theoretical computer science, a professor was right in noting to me after a sub-optimal exam: “I know you are a good hacker, but you also need to know theory”.

I look at the number of articles I have written on this site on the topic of Emacs and it saddens me that my repertoire is so one sided: My life as a “programmer” up until now has roughly been divided into two stages. The first (2012-2017) was as an autodidact, where I attempted to teach myself various topics through books, blogs and whatever I could understand. My main mistake during this phase was that I intentionally tried to avoid certain topics, as to not “taint” my understanding of theoretical topics I ultimately never engaged with, in some naive hope to perhaps be able and contribute a novel perspective. The second (2017-now) is as a student where I was taught the curriculum of a CS bachelor and master.

While the latter has been important in forcing me to engage with topics I didn’t get at first or was too lazy to acquire on my own (the formal notion of computation, proof and logic, the functioning operating systems, etc.) I am also under the impression that the quality of my learning has been a lot worse. If I had to guess, this is probably due to the structuring effect of the necessity to grade, hence to write exams, hence to teach what is “examable” – that is to say what can reasonably be inspected to see if students appear to have understood some topic. On more than one occasion I have been too stubborn to engage with this fact. I refuse to memorise what interests me, and instead try to acquire a fundamental understanding of some topic; which is not the best strategy when you only have a few days to prepare for an exam. Another time I just had no interest to study because I found myself already knowing everything on a topic that interested me. My grade suffered accordingly. It should be said that none of this is surprising from an institution that is in some sense a thousand years old, caught between the needs of modern industry and it’s own bureaucracy.

My hope is that the next five years, and beyond that I will manage to grow beyond the shortcomings of the past. I want to both get around to learning and engaging with various topics (both theoretical and the opportunity to engage with programming languages I had been wanting to properly learn for a while, among others: Ada/SPARK, Erlang, Smalltalk, Julia, APL, Forth), and make use of a clearer and improve understanding of the fundamentals. As with all new years resolutions, there is a certain naivete in this desire, but I intend to make it slightly more realistic by taking a break from my “work” on and around Emacs.

Sadly this is not as easy, as I have taken up some responsibilities, that out of respect towards other users and developers I don’t just wish to drop. The most important one of these is the development of a backwards compatibility package “Compat”, that will have to be updated for the upcoming release of Emacs 29.1 (the majority of the work has been done, what is missing is to check if any major functions have been added and to ensure that upgrading will work as expected).

If anyone is interested in temporarily maintaining, or perhaps even adopting any of the following packages, I would be most grateful:

• setup, a configuration macro and a alternative to use-package
• shell-command+, a extended version of shell-command with Eshell-like abilities.
• vc-backup, a VC backend that uses backup files.
• compat, an Elisp compatibility library (back until 24.3) used by a number of prominent packages such as Magit, ERC and Consult.
• autocrypt, a implementation of the autocrypt protocol.
• clang-capf, a completion at point backend that uses Clang.
• face-shift, a package that allows shifting the hue of buffers depending on their major mode
• insert-kaomoji, a categorised collection of eastern emoticons
• package-vc, a VC based backend for package.el (in developed in core)

Most of these are “finalised” and don’t require much work, let alone have many users to worry about. Note that due to being part of Emacs, most of these require a copyright assignment to the FSF. If necessary, I am willing to relocate development to any code forge, including GitHub.

I should note that this decision has not only been made out of boredom but also because of other personal issues I’d rather not go into here. While I know that my role in the world around Emacs is nothing close to significant people like Eli Zaretskii, Stefan Monnier or Jonas Bernoulli, I nevertheless apologise in case this turns out to be an inconvenience.

I thank anyone who has read this far, and in case anyone is interested in helping out, please contact me via email.

Moving Quickly to Project Work Spaces

On Saturday in the middle of November I wrote Project Dispatch Menu with Org Mode Metadata Denote and Transient; since then I’ve further extended and refined and generalized the functionality. You can find the underlying code in my dotemacs jf-project.el package.

In this post, I want to walk through the functionality and generalizations.

Explaining the Terms

There are two major terms to discuss: project and work space.

Project

In my work flows, I create a single note for a project. I use the Denote 📖 package for managing that note and all other notes. The format I’ve chosen for my notes is Org-Mode 📖.

I indicate that a note is a project in two ways:

• Adding the :projects: to the note’s #+FILETAGS: keyword’s list.
• Adding the top level #+PROJECT_NAME: to the note.

The purpose of the #+FILETAGS: is for searching via tags. The purpose of the #+PROJECT_NAME: is to be the short-name of how I reference the project. For example “Newman Numesmatic Portal” is the title of note, but NNP is the #PROJECT_NAME:.

This #+PROJECT_NAME: is also what I use when I’m tracking my work time.

Project Work Spaces

In this context a work space is a place where I read and write material related to a project.

Examples are:

• A project board for managing the team activities of work.
• A local code repository, for code, tests, and inline documentation.
• A local project notes file, for indexing high level project information.
• A remote repository, for working on issues and pull requests.
• A timesheet, for tracking time and writing notes regarding task resolution.

This list is arbirtrary, but a common property is that each work space has two pieces of data:

Title

The name of the work space.

Path

The path to that work space; either a local file name or one on a remote server (e.g. http://takeonrules.com/path/to/project).

While working on a project I often move between those work spaces. I also bounce between projects, helping unstick folks or looking for previous reference implementations.

Implementation Example

Here is the project metadata for the The Travels of Duinhir Tailwind series:

#+TITLE:      The Travels of Duinhir Tailwind
#+DATE:       [2022-11-29 Tue 09:18]
#+FILETAGS:   :rpgs:the-one-ring:projects:
#+IDENTIFIER: 20221129T091857

#+PROJECT_NAME: The Travels of Duinhir Tailwind
#+PROJECT_PATHS: ("Notes" . "~/git/org/denote/indices/20221129T091857--the-travels-of-duinhir-tailwind__rpgs_the-one-ring.org")
#+PROJECT_PATHS: ("WWW Series" . "https://takeonrules.com/series/the-travels-of-duinhir-tailwind/")

TITLE

The name of the project.

DATE

The date I created this note.

FILETAGS

The tags associated with this note.

IDENTIFIER

The unique identifier assigned by Denote.

PROJECT_NAME

The name I say to myself when “working on” this project.

PROJECT_PATHS

Each line is a Cons cell with the `car` (e.g. first element) as the work space name and the `cdr` (e.g. last element) as the path.

Org-Mode takes these multiple PROJECT_PATHS and creates a list. Something that is super helpful for Emacs 📖’s completing-read function.

Selecting a Project and Work Space

I have bound Cmd + 2 (or s-2 in Emacs syntax) to jf/project/jump-to/project-work-space. The function does the following:

1. Prompts for the project
2. Then for the given project, prompts for the work space
3. And then opens that work space.

The prompt for project applies some “Do What I Mean” function (DWIM 📖) logic; currently the rules are as follows:

• If I am tracking time to a project use that instead of prompting.
• If the current buffer is an Org-Mode agenda file and the cursor (e.g. point) is a descendant of headline is a project, use that instead of prompting.
• If the current buffer is a file that is part of a version control project that is registered as one of the PROJECT_PATHS, use that instead of prompting.

To force the prompt and not auto-select, I can provide a prefix argument when I call jf/project/jump-to/project-work-space.

Conclusion

My hope in sharing this is to reinforce that one of the pillars of Emacs is quick navigation. In this case, I built up functionality to navigate between related concepts; leveraging metadata and my existing plain text note taking ecosystem to power the functionality.

And in sharing these details, my hope is that others can come along, glean approaches from my code, and continue to extend their tools to do what they want and need.

A month ago, I was talking with a team mate. They were sharing their note taking process. I listened and made one suggestion: to find a tool that doesn’t lock your notes within the note taking application.

I pointed them to Logseq 📖; in part because they articulated a strong need for note taking across multiple devices and I wasn’t trying to sell them on Emacs.

I hope this post demonstrates the utility of having notes that can be accessed by a “scripting” tool; thus extending the utility of those notes. Which in turn increases the “value” of the notes; creating what I hope to be a virtuous cycle.

Over a year ago, I wrote a text titled “What I’d like to see done in Emacs”. There I mentioned a few ideas and projects related Emacs that I think are worth perusing.

Some time later, one has materialised (Compat), one has been worked on in a variation (package-vc) and one more has been worked on, but I lost the patch…

Nevertheless, I’d like to propose a few more ideas along the same lines. An early “five-year-plan” so to speak. Of course I don’t (and can’t reserve) any exclusivity on tackling these projects – anyone interested pick these up gladly, with our without my cooperation.

Eww Unmangler

It is no secret that the web is a mess. The last 30 years have demonstrated that HTML hasn’t been expressive enough to satisfy the needs of a world-wide-web. CSS and Javascript didn’t suffice as a thin layer, but became the material on which the desired path was built that led us to where we are. Combine this with the economic interests of platforms centred around advertisement and the trajectory, especially the deviation from the initial intentions doesn’t appear that surprising – in retrospect.

It doesn’t come to a surprise to many that the built-in browser EWW struggles with handling anything beyond the simplest websites. For a website to work with these kinds of text-oriented web browser (links, w3m, …) a web-developer has to consciously restrict their technologies and attempt to design a site that can be used with simpler tools. This is not a given.

To make the web work with EWW, or at least some segment of the web usable, it appears that manual intervention is necessary. What I have been thinking about it a tool that hooks into EWW, and depending on the site applies transformations to make the site readable: Reformatting headers, removing unnecessary elements, making the page more readable.

This requires a database of popular websites (For programmers the focus would initially lie in pages like StackOverflow, Reddit, Quora, GitHub, GitLab, …), and ideally a DOM-manipulation language to make it more maintainable. This could make use of eww-after-render-hook, but I fear that this would be too late. Instead it might be necessary to advise eww-display-html (or in the future extend EWW to make these kinds of manipulations easier).

I have been working on an initial sketch for a package like this, but am not satisfied with what I have written up until now. As mentioned above, the main problem is not technical but in finding an elegant way to express the problem.

ELPA Submission Helper

Sharing and publishing packages should be easier than it is now. Of course you can just upload somewhere, but then any interested user would have to fetch, install check for dependencies and look out for updates it manually. There is a reason why package managers like package.el are popular.

Sadly it is not as convenient on the other end. Sure, contributing a package to GNU or NonGNU ELPA just requires sending an email to emacs-devel@gnu.org, but there remain a number of implicit conventions that new contributors may be confused by. In my own experience, these include but are not restricted to:

• Adding a package to GNU ELPA requires signing the FSF copyright assignment.
• It is best to provide a URL to a Git repository.
• Running the byte-compiler and checkdoc can detect common mistakes.
• Adding an .elpaignore file can be used to remove files that don’t have to be distributed to users
• Packages shouldn’t have hard dependencies on non-free software

I believe it should be possible to provide a little package that checks and interactively/mechanically prompts the user most of these questions, resulting in a message that can be simply sent out.

This could include preparing a Git repository and suggesting a Git forge like Codeberg or Sourcehut. After explaining the difference between NonGNU and GNU ELPA, a request to sign the CA could be prepared as well, if the maintainer chooses to distribute their package as part of GNU ELPA, which is part of Emacs. It might even make sense to clone emacs/elpa.git or emacs/nongnu.git and directly prepare a patch.

If package-lint is added to NonGNU ELPA, then that could also be integrated into the process.

Another advantage of this approach is that the message could be generated with some special header information that would allow a CI-like process to detect the message and run a few automatised tests on some recent versions of Emacs, to be shared on the mailing list.

I don’t have any code for this idea yet, but preparing a preliminary version shouldn’t be that difficult, if there is interest in a little elpa-helper.

Distributed Content-Addressable Elisp Pastebin

I have a lot of small utility functions that I wrote-up once, and never changed since. One of the most frequent commands I use is this:

(defun site/kill-region-or-word ()
  "Call `kill-region' if there is an active region.
Otherwise kill the last word, just like in Unix."
  (interactive)
  (call-interactively (if (region-active-p)
                          #'kill-region
                        #'backward-kill-word)))

The chance is slim that this will ever require changing. This is finished code, and will work for as long as all the function it uses work.

If I wanted to share this snippet with someone else, I don’t think that creating a package and submitting it to an archive would be the right approach. I am not fond the idea of packages that just collect unrelated functionality like crux or consult. Sending them my code directly works just well enough for 1:1 situations.

What I really want is some pastebin service dedicated to (Emacs-)Lisp code. Ideally content-addressable and distributed. Perhaps this could be based on Torrents, perhaps IPFS, or something else entirely. Maybe this could also use CRDT as a basis?

Perhaps an example demonstrating can clarify my idea. Assuming some name, say dcap (distributed content-addressed pastebin), I could define a little function in my own configuration as follows

(dcap-defun kill-region-or-word ()
  "Call `kill-region' if there is an active region.
Otherwise kill the last word, just like in Unix."
  (interactive)
  (call-interactively (if (region-active-p)
                          #'kill-region
                        #'backward-kill-word)))

Basically, the same as above, with the minor difference that I used a macro called dcap-defun instead of defun. This would define a function for my own use, and declare a public snippet. The snippet would then be addressed using a hashsum, say 6465c9e5c3426b66a9fa45663946884faebc80db3260c55192d1cd4322472450.

On the other end, someone might decide to use this command and include it on their end. They might write something like

(defalias 'kill-word-or-region
  (dcap-fetch-func "6465c9e5c3426b66a9fa45663946884faebc80db3260c55192d1cd4322472450"))

Note that the name used here is not the same as the one I used. What dcap-fetch-func does is retrieve a definition from the network (say using a more generic primitive like dcap-fetch) or use a cached copy, and ensure the return value is a function.

As a hashsum like this can be inconvenient, having alias lists could be useful. Each such list could be designated by a URI that contains an association of human-readable names to hashsums.

$ curl https://some-website.net/path/to/my-alias.file
((kill-region-or-word "6465c9e5c3426b66a9fa45663946884faebc80db3260c55192d1cd4322472450)
 ;; ...
)

If configured, you could then do the same as above using a more sensible name (or a more convenient macro):

(defalias 'kill-word-or-region (dcap-fetch-func 'kill-region-or-word))

Implementing this is primarily a technical issue. Points that have to be considered are:

• How to ensure a definition is shared among the network and doesn’t get lost,
• How to prevent the network from being flooded with spam or illegal content. Even if the content is restricted to s-expressions only, you could host anything using strings,
• Figure out if and how dependencies between snippets can be established. Is any other meta-data necessary (Date, Author, etc.)?

It might be that a distributed system introduces too much complexity? It is probable that the network couldn’t just rely on idle Emacs instances, and a stand-alone node implementation would have to be implemented.

The important thing is: Reducing the overhead in sharing small improvements is one of the things I admired about the Emacs Commune. Packages usually imply there is a long-term project, that might grow over time. Copying code verbatim can be a nuisance and is not always reliable. I believe there is a niche between the two that can be satisfied.

Esoterical Text Manupulation Language

Another far-fetched idea is a little programming language for text manipulation. The twist is that I’d like to combine various features from different paradigms and programming languages.

My motivation stems from an appreciation but persistent scepticism regarding modal editing. I do believe that an expressive language for manipulating text is of use, but I don’t think that vi’s approach – throw you into “normal mode” at first, and have “insert mode” be a something you request – is ideal. I know that evil-mode, an emulator mode for people who have previously been using “Vim” (vi’s little brother), has the option of inverting this by default, but I remain unsatisfied. Other modal editing systems like Objed might be interesting if developed further, but I have been thinking if an entirely different approach could be viable instead.

So how about this: A programming language that has buffer ranges as a primitive data type, and treats these as mutable strings. We borrow the stack paradigm from Forth, implicit mapping of function over lists from APL, and intuitive regular expressions from AWK. This would allow us to express an intention like

Take all empty substrings at the beginning of each line and append a constant string.

There are many ways we could write this. Say we want to be verbose, and type out every intention word for word:

/^/ match-all "foo" append

we can imagine the stack being manipulated by each command as follows:

TOS                 ;; We start with an empty stack (TOS: Top of Stack)

TOS
/^/                 ;; A regular expression matching the beginning of a line

TOS
[(0;0) (41;41) ...] ;; A list of buffer intervals that match /^/

TOS
"foo"               ;; A constant string
[(0;0) (41;41) ...] 

TOS
[(0;3) (41;44) ...] ;; The intervals have been modified

This is fairly trivial, but how about an idea like

Match all lines that include of “bar”, “baar”, “baaar”, … and reverse their order of occurrence.

This time let us assume a terse syntax,

/ba+r/ ml lr

Again, we begin with a regular expression, request all intervals of the lines that match it and then reverse the list – which has an effect on the buffer. We press enter and the program is executed. This could involve a special interpreter or it could be compiled into Emacs Lisp.

One more feature I would like to see is strong typing – specifically interactive and immediate strong typing. While we are at it why not throw dependent typing into the mix? Let us consider an example to illustrate my point. Imagine the following

Replace each instance of “foo” with a number in increasing order of their occurrence.

This time we use a single unicode character for each command and assume an appropriate input-method is provided.

foo×↑ρι%s→

This time regular expressions (foo) and strings (%s) aren’t quoted at all. They are distinguished by being regular ASCII characters, so adding quotation is optional. Next we…

• ×: Select all regions that match a regular expression, just like match-all in the first example,
• ↑: Duplicate the top-of-stack (type ∀ α . α → α),
• ρ: Return the length of the list (type ∀ α . [α] → ℕ),
• ι: Generate a list of integers in rising order (type ℕ → [ℕ]),
• →: Replace the region (or in our case list of regions) with the result of applying the format string %s using the in–between argument (type (fmt : 𝕊) → (formatted fmt) → ⅈ, where ⅈ is a region and formatted is a function that returns a type for a format string).

Ideally this should fail and ding right after typing →, because the values on the stack are a list of integers, and not strings, before anything is even executed. Replace the %s with a %d and the program types. It can now be executed. While this is going on, and since the typing is interactive, the active buffer intervals and their replacements can be visualised on the active portion of the window.

(A different question is if you actually want this degree of strictness in a convenience language…)

The main issue here will be figuring out a good vocabulary (which will probably have to be user-extensible) and a flexible syntax to accommodate its needs.

A User Compat Library

Last year I shared the idea of working on a Forwards-Compatibility library for Emacs Lisp, and it has since not only been implemented but also published. It allows versions of Emacs going back to 24.3 (released 2013) to make use of a number of newer functions and macros. I am currently working on preparing support for Emacs 29.1, and hope to release it soon.

One restriction I drew when starting the project was that it won’t include any user-facing code. Any function that is also a command would only be usable as a function. The development branch for Emacs 29 intentionally leaves out the setopt macro. This is because Compat is a package that is rarely installed manually. Instead it is added as a dependency. And as dependencies are, they might appear or disappear, depending on what packages are installed and how clean you keep your package list.

(This argument is actually not that solid, because Emacs intentionally doesn’t draw a line between developers and users. If Compat is installed on Emacs 24.3, you could be using and-let* in your personal code in init.el and suddenly be confused if the dependency is removed.)

The idea here is simple: Provide a package with these missing definitions (commands and user-facing macros), that is supposed to be explicitly installed by the user.

There is not that much more to this idea, just a nice thing I think some people would appreciate. Being a package people would consciously installed, it could risk being more invasive and opinionated, e.g. by (pseudo-)depending on other packages in ELPA such as project, xref, etc. to ensure the newest versions are installed.

I am curious to hear if anyone things if these ideas have any merit. It would be great if someone were interested in collaborating on developing or even implementing these projects. Right now, I am under the impression that I am reaching a limit as to how much time I want to invest into a hobby like Emacs development. I am a full-time student (and part-time TA) after all. This means I’ll be thinking twice before starting any new Elisp project, as I always have other ideas I would like to work on as well.

Creating a Means of Quickly Navigating Amongst a Projects Important “Pages”

At Software Services by Scientist.com I work on several different projects. Ideally, one project at a time, but within a week I might move between two or three. Note taking procedures help me re-orient to a new project.

On Wednesday I spent some time reflecting on the common jumping off points for a project:

Local Source Code

the local repository of code for the project; thus far each project has one repository.

Remote Source Code

the remote repository where I interact with issues and pull requests.

Remote Project Board

the page that has the current project tasks and their swimlanes.

Agenda/Timesheet

the place where I track my local time and write notes.

Local Project Note

the place where I track important links or information regarding the project.

When I’m working on the project, I’m often navigating between those five points. Since I work in Emacs I figured I’d write up some code.

First, I thought about the data. Where should I store this information? Looking at the above list, the best candidate was the Local Project Note; a note written in Org-Mode and I use Denote to help me manage this kind of note.

For each project document I added the following keywords (e.g. those that can be found by the org-collect-keywords function):

#+PROJECT_NAME:

By convention, this is the short-name that I use for my timesheet and task management. (See Org Mode Capture Templates and Time Tracking for more details.)

#+PROJECT_PATH_TO_CODE:

The file path to the code on my machine.

#+PROJECT_PATH_TO_REMOTE:

The URL of the remote repository.

#+PROJECT_PATH_TO_BOARD:

The URL of the remote project board.

The Helper Functions

I wanted a common mechanism for selecting the project. I wrote the following function:

(cl-defun jf/project/list-projects (&key (project ".+")
					 (directory org-directory))
  "Return a list of `cons' that match the given PROJECT.

The `car' of the `cons' is the project (e.g. \"Take on Rules\").
The `cdr' is the fully qualified path to that projects notes file.

The DIRECTORY defaults to `org-directory' but you can specify otherwise."
  (mapcar (lambda (line)
	    (let* ((slugs (s-split ":" line))
'		   (proj (s-trim (car (cdr slugs))))
		   (filename (file-truename (s-trim (car slugs)))))
	      (cons proj filename)))
	  (split-string-and-unquote
	   (shell-command-to-string
	    (concat
	     "rg \"^#\\+PROJECT_NAME: +(" project ") *$\" " directory
	     " --only-matching --no-ignore-vcs --with-filename -r '$1' "
	     "| tr '\n' '@'"))
	   "@")))

It searches through my org-directory for the given project; by default that project is a fragment of a regular expression. That regular expression is “any and all characters.” I can use the above function as a parameter for completing-read.

I also want to set my default project. For this, I used Transient’s transient-define-suffix function. Below is jf/project/transient-current-project, a function I use to manage and display the jf/project/current-project variable.

(defvar jf/project/current-project
  nil
  "The current contextual project.")

(transient-define-suffix jf/project/transient-current-project (project)
  "Select FILES to use as source for work desk."
  :description '(lambda ()
		  (concat
		   "Current Project:"
		   (propertize
		    (format "%s" jf/project/current-project)
		    'face 'transient-argument)))
  (interactive (list (completing-read "Project: "
				      (jf/project/lis't-projects))))
  (setq jf/project/current-project project))

I also recognized that I might want to auto-magically select a project. So I wrote up the basic jf/project/find-dwim:

(defun jf/project/find-dwim ()
  "Find the current project."
  (completing-read "Project: " (jf/project/list-projects)))

The above function could look at the current clock in Org Mode and determine the associated project. Or, if I’m in a repository look to see what project it is associated with. Or whatever other mechanisms. For now, it prompts for me to pick a project.

The Interactive Functions

With the above “plumbing” I wrote five functions:

• jf/project/jump-to-agenda
• jf/project/jump-to-board
• jf/project/jump-to-code
• jf/project/jump-to-notes
• jf/project/jump-to-remote

The jf/project/jump-to-agenda function is a bit different, it tries to jump to today’s agenda item for the project.

(cl-defun jf/project/jump-to-agenda (&optional project
				     &key
				     (tag "project")
				     (within_headline
				      (format-time-string "%Y-%m-%d %A")))
  "Jump to the agenda for the given PROJECT."
  (interactive)
  (let ((the-project (or project (jf/project/find-dwim))))
    (with-current-buffer (find-file jf/pri
mary-agenda-filename-for-machine)
      (let ((start (org-element-map (org-element-parse-buffer)
		       'headline
		     ;; Finds the begin position of:
		     ;; - a level 4 headline
		     ;; - that is tagged as a :project:
		     ;; - is titled as the given project
		     ;; - and is within the given headline
		     (lambda (hl)
		       (and (=(org-element-property :level hl) 4)
			    ;; I can't use the :title attribute as it
			    ;; is a more complicated structure; this
			    ;; gets me the raw string.
			    (string= the-project
				     (plist-get (cadr hl) :raw-value))
			    (member tag
				    (org-element-property :tags hl))
			    ;; The element must have an ancestor with
			    ;; a headline of today
			    (string= within_headline
				     (plist-get
				      ;; I want the raw title, no
				      ;; styling nor tags
				      (cadr
				       (car
					(org-element-lineage hl)))
				      :raw-value))
			    (org-element-property :begin hl)))
		     nil t)))
	(goto-char start)
	(pulsar-pulse-line)))))

The jf/project/jump-to-board function assumes a remote URL.

(cl-defun jf/project/jump-to-board (&optional
				    project
				    &key
				    (keyword "PROJECT_PATH_TO_BOARD"))
  "Jump to the given PROJECT's project board."
  (interactive)
  (let* ((the-project (or project (jf/project/find-dwim)))
	 (filename (cdar (jf/project/list-projects :project the-project))))
    (with-current-buffer (find-file-noselect filename)

      (let ((url (cadar (org-collect-keywords (list keyword)))))
	(eww-browse-with-external-browser url)))))

The jf/project/jump-to-board function assumes a directory on my local machine. The code is similar to the jf/project/jump-to-board.

(cl-defun jf/project/jump-to-code (&optional
				   project
				   &key
				   (keyword "PROJECT_PATH_TO_CODE"))
    "Jump to the given PROJECT's source code."
    (interactive)
    (let* ((the-project (or project (jf/project/find-dwim)))
           (filename (cdar (jf/project/list-projects :project the-project))))
      (with-current-buffer (find-file-noselect filename)
        (let ((filename (file-truename (cadar
					(org-collect-keywords
					 (list keyword))))))
          (if (f-dir-p filename)
              (dired filename)
            (find-file filename))))))

The jf/project/jump-to-notes prompts for the project and then finds the filename.

(cl-defun jf/project/jump-to-notes (&optional project)
  "Jump to the given PROJECT's notes file.

Determine the PROJECT by querying `jf/project/list-projects'."
  (interactive)
  (let* ((the-project (or project (jf/project/find-dwim)))
	 (filename (cdar (jf/project/list-projects :project the-project))))
    (find-file filename)))

Akin to the jf/project/jump-to-board, the jf/project/jump-to-remote opens a remote URL.

(cl-defun jf/project/jump-to-remote (&optional
				     project
				     &key
				     (keyword "PROJECT_PATH_TO_REMOTE"))
  "Jump to the given PROJECT's remote."
  (interactive)
  (let* ((the-project (or project (jf/project/find-dwim)))
	 (filename (cdar (jf/project/list-projects :project the-project))))
    (with-current-buffer (find-file-noselect filename)
      (let ((url (cadar (org-collect-keywords (list keyword)))))
	(eww-browse-with-external-browser url)))))

The Menu

Using Transient I define a menu for my projects. Lower case is for dispatching to the current project. Upper case prompts for the project then dispatches.

(transient-define-prefix jf/project/menu ()
  "My Project menu."
  ["Projects"
   ["Current project"
    ("a" "Agenda…" (lambda () (interactive)
		     (jf/project/jump-to-agenda jf/project/current-project)))
    ("b" "Board…" (lambda () (interactive)
		    (jf/project/jump-to-board jf/project/current-project)))
    ("c" "Code…" (lambda () (interactive)
		   (jf/project/jump-to-code jf/project/current-project)))
    ("n" "Notes…" (lambda () (interactive)
		    (jf/project/jump-to-notes jf/project/current-project)))
    ("r" "Remote…" (lambda () (interactive)
		     (jf/project/jump-to-remote jf/project/current-project)))
    ("." jf/project/transient-current-project :transient t)]
 '  ["Other projects"
    ("A" "Agenda…" jf/project/jump-to-agenda)
    ("B" "Board…" jf/project/jump-to-board)
    ("C" "Code…" jf/project/jump-to-code)
    ("N" "Notes…" jf/project/jump-to-notes)
    ("R" "Notes…" jf/project/jump-to-remote)]
   ])

Conclusion

During my day I spend a lot of time writing and reading; and for large chunks of time those are all related to a single project. Each project has predictable “places” where I will read and write.

The above functions help me both document the relationship of those predictable “places” and automate my navigation to those different tools. In all things Emacs remains my homebase; it is where I can go to re-orient.

My jf-project.el document has the above code and any further updates, bug fixes, etc to the above now static code.

Over on /r/emacs the community has been answering What are your favorite packages for improving vanilla emacs text editing?

Such a great thread; folks sharing their favorite packages.

Personally I learned about symbol-overlay and am giving it a spin.