Since I had no interesting books to read today, nor interesting films
to watch, I decided to scavenge for the most intriguing content one
can find online. I ended up reading the Linux kernel mailing lists,
but those discussions seemed to be 18+, so I settled for the
comparatively civil emacs-devel.
For those unfamiliar, emacs-devel is the primary development
discussion list for GNU Emacs – where design decisions get made,
patches get reviewed, and occasionally where people spend 200 messages
arguing about version control software. This is the story of that
last one.
2008: “This question is over and decided”
In March 2008, Emacs was migrating from CVS (yes, CVS) to something
more “modern”. The two contenders were Git and Bazaar.
- Git, created by Linus Torvalds for the Linux kernel.
- Bazaar was a GNU project, maintained by Canonical
A 236-message thread erupted on emacs-devel. People benchmarked both
tools. The results were not subtle. Andreas Schwab, one of the core
developers, reported his first impression:
My first impression is that bzr is slow, so slow that it is completely
unusable. How can it come that a simple bzr log takes more than a
minute to even start? Even cvs log is instantaneous in comparison,
although it has to request the log from the server.
David Kastrup found it equally puzzling:
I find this surprising: “git log” is pretty much instantaneous, and
git recalculates a code piece’s history in the process. In contrast,
one has to tell Bazaar when one copies or moves or renames files, so
it should have the information available right away.
The actual numbers:
git log | head -1 0.012 seconds. The same command with Bazaar took 21.5 seconds.
- Committing a single-file change: 0.08 seconds with Git, 17 seconds with Bazaar.
The benchmarks kept coming. Stefan Monnier, the head maintainer, set
the bar low:
I don’t care if Bzr is slower or faster than Git, but in
order to switch to Bzr, we need it to be ‘fast enough’. Currently it
is not. At the very least the ‘bzr diff’ should not take more than a
couple seconds.
Meanwhile, Jonathan Lange, an actual Bazaar developer from Canonical,
was in the thread doing tech support.
His recommended workflow for the initial checkout:
An even better way to do the initial download is this:
$ wget http://bzr.notengoamigos.org/emacs.tar.gz
$ tar xzf emacs.tar.gz
$ bzr init-repo emacs-bzr
$ cd emacs-bzr
$ bzr branch ../emacs trunk
$ cd trunk
$ bzr pull –remember http://bzr.notengoamigos.org/emacs/trunk/
Compare that to git clone!
Someone in the thread finally asked the obvious question:
To the emacs maintainers and decision makers: What more information is
required to convince bzr is not the right tool at the present moment?
Richard Stallman’s reply:
This decision is not a decision for the present moment. It is a long
term decision. So it would be better to wait a few months while Bzr
developers improve it, than to make some other “temporary” decision
that would probably be hard to reverse.
And in case anyone missed the point, in a separate message:
This question is over and decided.
We will use GNU Bzr, because it is a GNU package.
When someone pointed out that this political decision was “wiping away
all technical arguments,” RMS replied:
The rule that GNU packages should support each other helps make the
GNU system as a whole work better.
Someone asked the obvious follow-up: “Why can’t we just make Git part
of the GNU system?”
RMS:
We could include it in the GNU system, but
its developers are not likely to want to make it a GNU package.
To be fair to RMS, there’s a real principle here. If the GNU project
doesn’t use its own tools, it sends a message that those tools aren’t
good enough, which undermines the whole idea of a self-sufficient free
software ecosystem. He’d been making this argument for decades, and
it had served the project well in many other cases. The problem
wasn’t the principle. The problem was that Bazaar couldn’t live up to
it.
The 236-message thread, the benchmarks, the Canonical employee’s
workarounds; none of it changed the outcome. The decision was
political, and the politics were settled.
2008-2012: The long tail
The rest of the world moved to Git. GitHub launched in 2008 and
exploded. Emacs contributors, meanwhile, had to learn Bazaar, a
tool they used nowhere else, just to submit patches.
Threads like “Help me unstick my bzr, please” and “Can NOT bzr the
emacs repos (may be bzr has a memory leak)” became regular
occurrences.
Then in 2012, Canonical laid off the Bazaar development team.
2013
In March 2013, a year after Bazaar’s development ceased, John Wiegley
posted what everyone had been wanting to say:
We have often debated the merits of Git vs. Bazaar, and which one the
GNU project should use for Emacs development. I think now is an
appropriate time to revisit this decision.
My main reason for bringing this up again is that Bazaar development
has effectively stalled. There are major bugs which have been in their
bug-tracker for years now – bugs affecting Emacs development, such as
the ELPA repository – which have been ignored all this time.
So, to Richard as the undisputed Czar of all things Emacs: can we now,
pretty please, switch to Git?
200 messages followed.
RMS’s first response:
The maintainer says he is fixing some bugs, and
I asked him just yesterday to fix the ELPA branch bug. I’d like to
give him a reasonable time to do this.
The bug was 1.5 years old. Dmitry Gutov called it out:
Isn’t this a bit late? The bug is 1.5 years old. Was he not aware of it before?
Then RMS revealed his hand:
I am trying to determine whether Bzr is effectively maintained or
not. I’d rather get a Yes answer than a No answer.
That’s a remarkably honest thing to say publicly. He wasn’t hiding
his preference. He genuinely believed in the principle of GNU
projects supporting each other, and he was hoping reality would
cooperate.
Joakim Verona, a longtime Bazaar user and Emacs contributor, described
the reality on the ground:
I have done my best to be a constructive user of the tool, and I have
had many technical difficulties. When I try to find solutions to the
issues I notice the following:
- The bzr community is very helpful. This is good.
- There are many well known bugs. There are also many well known
patches for these, some of them provided by Emacs developers. They
never enter upstream. By “never” I mean years. This is bad.
RMS replied:
I don’t have time to read the Bzr mailing list. Or any development
mailing list. The only such list I am on is this one. You might as
well tell me to fly to the moon as tell me to read something on the
Bzr list.
And when asked whether the users of Bazaar should have a say in
whether Bazaar is sufficiently maintained:
When I have to decide whether a maintainer is doing an adequate job or
needs to be replaced, I pay attention to whatever relevant information
I get. However, to give users “a say” in the decision seems improper,
so I don’t do that.
Karl Fogel, a veteran open source developer (author of Producing Open
Source Software and one of the original Subversion developers),
delivered the sharpest critique in the thread:
Well, really, you don’t have time to pay close enough attention to Bzr
development to competently decide whether it’s still a good choice for
Emacs. That’s fine – no one has time to do every important thing,
and you do many other important things.
But then why do you think you still have the time & mental bandwidth
to make this decision well? Why not delegate it to the Emacs
maintainers on the grounds that you no longer have time to do a good
job of this evaluation?
He pointed out that asking one person about one bug is not a proxy for
project health, and that others in the thread had already done more
thorough research than RMS could, given his time constraints.
RMS’s reply:
Because more than Emacs is at stake here.
One line, but it’s the core of his worldview. If the flagship GNU
project abandons a GNU tool, what signal does that send to every other
GNU package? He wasn’t wrong about the stakes. He was wrong about
the tool.
Karl pushed once more:
You should either devote enough time to evaluating Bzr’s maintenance
state to get a reliable answer, or delegate to someone who can do
so. Instead, you’re asking the maintainers to rely on your
investigation… yet you clearly don’t have time to do a good
job. This is a poor use of everyone’s time.
RMS:
I already have a plan for how to proceed on this, and I am doing it.
No details. No timeline. No delegation. Trust me.
Meanwhile, Stefan Monnier, posted exactly one substantive message in
the entire 200-message thread:
Just like I didn’t fight Richard’s choice of Bazaar, I don’t care very
much whether we keep using Bazaar or we change to Git, Monotone,
Darcs, Mercurial, OpenCM, Fossil, younameit.
The only thing I care for now is to move away from Bazaar for the
’elpa’ branch because Bazaar can’t handle it properly.
Leo Liu summarized what everyone was thinking:
Most GNU projects aren’t using BZR as you might be aware.
While helping BZR fixing bugs might be a gain for BZR, it is a loss as
a whole for GNU. Volunteers spend their spare time on GNU projects
and if 20% of that time is taken up by wrestling with BZR, it becomes
costly to the point discouraging people from joining.
For the greater good of GNU, move off BZR seems like the only sound
choice.
The thread ended without a clear resolution. RMS had a plan. He was
working on it. The 200 messages didn’t produce a decision, but they
did make the community’s position unmistakable.
2013: The ELPA branch breaks
Later that year, Stefan made the practical move that set the stage for
everything that followed. The ELPA branch was broken on Bazaar, a bug
that crashed on checkout, with no one left to fix it. Stefan moved it
to Git, and his announcement showed exactly the kind of careful
leadership that had kept Emacs development running through all of
this:
I’m not terribly happy about this change, since it means we’ll be
using two different tools (Git for ’elpa’ and Bzr for ’trunk’), but I
really see no other way out.
I don’t want this to be a discussion about the merits/pitfalls of Git
vs Bzr, and this is not an occasion to discuss the use of Git for the
’trunk’ either.
He knew exactly what everyone was thinking “if Git is good enough for
ELPA, why not for trunk?” and he headed it off. One problem at a
time.
2014: ESR pushes the button
By August 2014, Eric S. Raymond had the conversion scripts ready.
He’d been working on it quietly:
You haven’t heard much about it because the hard work is all done. I
have the scripts ready to go and need only about eight hours’ notice
before pushing the button.
The actual migration happened in November 2014. On November 13th, ESR
posted a six-word message:
Commits are open. Have at it.
Which some even described as heroic.
Six years of debate, 236 messages in the 2008 thread, 200 messages in
the 2013 thread, Stefan’s years of quiet maintenance, countless “help
me unstick my bzr” pleas, one dead version control system, and it
ended with six words.
The aftermath
The days following the migration were educational. Half the core
contributors had never used Git:
- “This Is The Git Help Mailing List”
- “git pull fails with merge conflicts. How can this possibly happen?”
- “A simple git workflow for the rest of us”
- “need help adjusting workflow to git”
- “Good book on Git”
- “Obscure error/warning/information message from git pull” (124 messages)
These were people who’d been developing one of the most important text
editors in the world for years, asking basic Git questions, because
they’d been stuck on Bazaar while the rest of the world moved on.
What a bzr saga! Anyway, better than any film I could have watched
tonight.