📰 newsreader

In January 1992 I still didn't know what a kernel was. I found the discussion much later, around 2009, on some late afternoon when I should have been doing something else entirely, and I read the whole thread in one go, on the small screen of a laptop whose fan made a noise that annoyed me. What stayed with me was less the technical content and more the tone. Two guys arguing about operating system architecture as if they were arguing about someone's honor.

The title already said it all: "LINUX is obsolete". Andrew Tanenbaum posted it on January 29, 1992, in comp.os.minix, which was where the Minix enthusiasts gathered to trade patches and complaints. Linux was about five months into its public life. Version 0.01 had come out in September 1991 with something like ten thousand lines of code, and Torvalds was a twenty-two-year-old Finnish student who had started the thing, in his own words, as a hobby, nothing big and professional like GNU. It's worth remembering that scale, because when the story is told today it sounds like two giants squared off. They didn't. A full professor, author of the textbook half the world used, wrote a post saying a kid's project was technically dead on arrival. And the kid answered.

I'm going to try not to repeat the postcard version, the one where Torvalds won because Linux is everywhere and therefore the microkernel lost. It's a lazy version and, I think, wrong in more places than people admit.

Let's start with what Tanenbaum actually said, which is different from what gets attributed to him. The thesis was that a monolithic kernel, in that year, was a "giant step back into the 1970s". He said the future lay in small kernels with only the essentials inside, basic scheduling, minimal memory management, interprocess communication, and everything else (file system, drivers, network stack) running as ordinary processes in user space, talking by messages. The second thesis, less quoted, was about portability. Linux was glued to the 386, written with its eyes on the 386, with constructs that only made sense there, and Tanenbaum believed, with considerable conviction, that x86 was a dead end and that RISC chips would take over everything within five years. Here he was wrong, and wrong in style. Not only did x86 not die, thirty-odd years later you still need a very good excuse not to run x86 on a server. But mind you, the bet on RISC wasn't exactly ridiculous. There was SPARC, MIPS, Alpha coming later, and nobody had any idea Intel was going to pour so much money into converting CISC instructions into micro-ops internally. Anyone who had read the computer architecture papers of 1990 found it plausible.

And there's a third thing, small, which always struck me as the most revealing of his state of mind: he wrote that Torvalds would have gotten a bad grade if he were his student. "Be thankful you are not my student. You would not get a high grade for such a design." That's a sentence a professor writes without thinking and which then stays glued to his reputation forever. It has nothing to do with microkernels anymore. It's academic authority talking down to someone it thinks should be listening.

Torvalds replied that same night, or in the early hours of the next, I can't pin it down. He started with something that still makes me smile: he said Minix had its share of "brain damage" and that Tanenbaum had an excuse, because a professor's job is to research and teach, not exactly to deliver a system people use for real. It's cruel, it's partly unfair, and it has a core of truth that Tanenbaum himself never denied, because Minix was deliberately a teaching system. Minix's license, for instance, didn't allow much redistribution, and Tanenbaum wanted the code readable and small, he didn't want it to grow. That was his decision, defensible, but it left a gigantic vacuum that someone was going to fill.

Then comes Torvalds's technical reply, which is more honest than its reputation. He admitted, and you can read this right there in black and white, that from a theoretical and aesthetic point of view Linux lost. He agreed that microkernels were prettier on paper. His argument was a different and very practical one: Linux existed, it worked, it was available now, and its monolithic design solved concrete performance problems that the microkernels of the day paid dearly for. He also said that Minix, although it presented itself as a microkernel, wasn't all that clean, with the file system and the memory manager living as separate tasks but without much real isolation, and that its task model, instead of helping, complicated simple things. It was one of the points where, honestly, I think Torvalds was right and Tanenbaum defended himself badly.

But this is where the normal story falls apart, and it's why I'm bothered by the version in which Linux won on points. What was being discussed was never just "which is faster". It was where you put the complexity, and who pays for it.

In a monolithic kernel, the complexity all sits inside the same address space. A network card driver has access to the same world as the scheduler's code. This is fast because a function call is a function call, a few clock cycles, and not a context switch with message copying. And it's dangerous for exactly the same reason. If the driver writes where it shouldn't, it writes into the whole kernel. I've seen this firsthand in a fairly ridiculous way. In 2014 I had a home server, an old box with a ten-euro USB wifi dongle bought at a flea market, and an out-of-tree driver that every so often would spit out "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008" followed by "Kernel panic - not syncing: Fatal exception in interrupt". Whole machine down. Because of a dongle. With a real microkernel, the driver would crash on its own and be restarted by a reincarnation server, which, incidentally, is exactly what Minix 3 does. Tanenbaum was right, and I was, that night, in front of a black screen with white text, agreeing with him without meaning to.

Except, and this "except" is everything, his being right cost something that Linux never wanted to pay.

Back to the cost. The problem with first-generation microkernels, and Tanenbaum defended it as if Mach were the final solution when he himself must already have been suspicious, was interprocess communication. Every service that used to be a function call becomes a message: copy, switch context, copy back. Mach's numbers were ugly. In measurements from the nineties, BSD on top of Mach 3 lost a serious slice of performance compared to monolithic BSD, and the Chen and Bershad paper from 1993 showed that part of the problem wasn't even the switching itself but cache behavior, memory footprint, the accesses that a fat kernel in the Mach style ended up making. In other words, the idea could be right and the implementation wrong. And that's more or less what Jochen Liedtke demonstrated with L4, in the mid-nineties. He rewrote IPC with an almost obsessive discipline, working in registers instead of memory, and got something on the order of twenty times faster than Mach. Twenty times. This is the point at which the 1992 discussion should have been reopened, and it wasn't, because by the time L4 existed, Linux already had a mass of users and a queue of contributors that nobody was going to stop.

Here I hesitate. Because the "Linux won through the ecosystem" story may be just another way of saying "won through luck", and I don't entirely believe that. There were decisions by Torvalds that were good and that an academic would hardly have made. Accepting patches from everyone, for example. Choosing the GPL in February 1992, shortly after the discussion, when before he'd had a more restrictive license of his own. Being rude when it was needed, and pragmatic when it was needed. The development model is an incentive system, not just code. And the monolithic architecture fit badly with Tanenbaum and very well with that incentive system, because in a single code repository inside the same memory space, anyone with a problem on the table could make a change and see the result. In a pure microkernel, the boundary of each service forces you to negotiate interfaces, and negotiating interfaces among volunteers scattered around the world is slow, political and tends to come to nothing.

This is my most committed opinion and the one I like least to see contested: the victory of the monolithic kernel was, to a large extent, organizational and not technical. Torvalds said later, and it's a theme he came back to several times, that splitting everything into isolated components turns simple problems into distributed systems problems. The phrase is his, the idea is his, and I think there's an uncomfortable truth in it. Anyone who has tried to debug two processes talking by message, waiting on each other, with a deadlock that only shows up when the machine is loaded, knows what he's talking about. In a monolithic kernel there's one deadlock, and one stack trace. In a microkernel there's a conversation that went badly and nobody knows who spoke first.

But at the same time, and this is the contradiction that leaves me in a bad mood, Linux has been eating microkernel ideas without admitting it. FUSE lets you put file systems in user space. DPDK and similar things pull the network stack out of the kernel when real performance demands it, which is a curious inversion: the microkernel took things out of the kernel for safety, DPDK takes them out for speed, and ends up in the same place. And eBPF, which has become fashionable in recent years and which people throw themselves at with an enthusiasm that tires me a little, is basically the admission that you want to load logic into the kernel but verified, bounded, isolated, with a static verifier deciding whether the program is safe. What is that if not an attempt to recover the safety of isolation without paying the price of switching address spaces? Tanenbaum could, without lying, say that Linux is reinventing the microkernel under the table, in pieces and badly. And Torvalds could answer, without lying, that it only does so where it pays off.

The numbers help convey the scale of the problem. The Linux kernel today runs to more than thirty million lines, and an enormous slice, somewhere in the sixty-odd percent range, is driver code and architecture support. When someone says Linux is "a kernel" they're using a generous word. It's a federation of tens of thousands of small territories all running with full privilege. The bug rate per thousand lines may even be reasonable, the reviewers are good, there are fuzzers like syzkaller hammering everything day and night, but the absolute size of the attack surface is what Tanenbaum called, in 1992, a fundamental error. And I think he's still partly right about that, even if he was wrong about nearly everything else in the prediction.

Another thing that rarely appears in the simplified versions: he never lost faith, and he did something few academics do, which was to spend decades building the alternative. Minix 3 is a microkernel of a few thousand lines, with drivers in user space, a reincarnation server that kills and relaunches failing components, and it was presented with the explicit idea of reliability, in a 2006 paper in which Tanenbaum asked whether it was possible to build reliable and secure operating systems. And then, in 2017, something almost comical happened. Researchers at Positive Technologies discovered that the Intel Management Engine, that hidden subsystem in almost every modern Intel processor, ran Minix 3. From what was understood, Minix was probably the most widely deployed operating system on the planet, on hundreds of millions of machines, running in secret, without Intel even having told the author. Tanenbaum wrote an open letter saying he knew nothing about it and that, of course, he would have liked to be informed. This doesn't prove the microkernel won, mind you. It only proves that a company that needed a small, isolated, trustworthy operating system inside the chip itself went and got a microkernel and not a Linux. When reliability really matters and raw performance doesn't, the choice changes.

For that matter, QNX has been around for decades in cars and medical equipment, and seL4, to which the Sydney group and later Data61 gave a formal proof of correctness in 2009, is probably the closest thing to "the microkernel as it should be" that exists: about ten thousand lines of C and a mathematical demonstration that the implementation respects the specification. You can't do that with thirty million lines. Nor with thirty thousand, as far as I know, of driver code.

The systems we all use every day, meanwhile, are the hybrid, the compromise, the thing nobody wants to admit is a compromise. Windows NT was born with microkernel ideas and kept pulling everything into the kernel for performance, graphics, part of the video stack, and then pushed some things back out when video drivers started producing blue screens by the dozen. XNU, which is in macOS and iOS, has Mach inside but with BSD glued alongside, in the same address space, which makes it monolithic for almost everything that matters. Calling it a microkernel is a marketing lie that has lasted twenty years. And Fuchsia, from Google, with its Zircon, is the most recent and best-funded attempt to redo the game with a modern microkernel design. It spent years in silence, and when it shows up it's on a smart home display. I don't say this with disdain, I say it with the caution of someone who has seen many grand rewrites end in a product smaller than what was promised.

I have a blind spot on this subject and I'd rather say so. I've never written production kernel code. I've written toy modules, read a lot, compiled kernels by hand on machines that didn't deserve it, and spent too many nights with menuconfig open turning things off that I later discovered were necessary. But I'm not a kernel hacker. The ones who are, and whom I know from forums and the odd conference, tend to have a relationship with Torvalds that doesn't look like the general public's. They respect him in a dry, almost professional way. They don't see him as a genius, they see him as a very good manager of chaos, and I think that's exactly what he was on January 29, 1992, when he answered that professor. He didn't write a refutation, he wrote a position: I have code that works, your theory is pretty, let's see who survives.

What bothers me most in the popular narrative isn't even the conclusion. It's the idea that the debate "was settled". Some people made that reading, saying Tanenbaum lost, that the microkernel was an academic illusion, and that time handed victory to pragmatism. But if we look with some care, there are two questions mixed together. One is "which of the two systems conquered the world?", and the answer is obvious and has nothing technical about it, it has to do with licenses, timing, community and the luck of showing up when 386 boxes were getting cheap. The other question is "which of the two architectures is better for running code we don't trust?", and that one remains open. Moreover, the second question is increasingly the one that matters, because more and more code runs in places where trust is low. Containers, virtual machines, enclaves, browser extensions, all of it is people raising walls between components that share hardware. If Tanenbaum came back today to write the post, maybe he wouldn't need to say Linux is obsolete. It would be enough to say that we've spent thirty years putting walls around a kernel that has no walls inside.

There's another layer, and here I admit I'm speculating. The monolithic kernel also won because the hardware helped. For two decades, per-core performance grew so fast that the extra cost of a microkernel always seemed surmountable with the next processor generation. And Torvalds's argument, the one about it having to be fast now, weakens when the CPU no longer gains what it used to. Since the mid-2000s, when frequency increases stopped, performance has come to depend on parallelism and cache, and that's precisely where communication costs between components weigh most. I don't know whether this plays for or against the microkernel. It might play for it, because a system made of independent services distributes better across many cores. It might play against it, because every message between cores costs cache line invalidations. I've read solid arguments in both directions and left both with the same doubt.

Back to the beginning, to that late afternoon in 2009. What stayed with me was something small and almost stupid, which is the way Tanenbaum wrote. He was convinced. He wrote like someone certain of being vindicated by history, and the most curious thing is that, in part, he was. Only not for the reason he imagined, nor in time to do him much good. There's a cruelty in that. Being technically right and being set aside, watching your system, Minix, be remembered mainly as the thing that inspired Linux, and spending decades hearing the same question in interviews, over and over. He reacted with an elegance I wouldn't have, mind you. He has said, more than once, that Linux is a good system and that the people who build it are competent, and that what bothered him was the design decision, not the result. He conceded what there was to concede. Torvalds, for his part, never apologized for the 1992 tone, as far as I know, and frankly I don't see why he would.

If you ask me which side I'm on, I'll answer quickly that it's Torvalds's, because I'm a web programmer and I spend my life making things that work today at the expense of things that may not work tomorrow. My ethic is "ship first, tidy up later", and anyone dealing with a real-time chat and SEO and a server that can't go down at three in the morning understands that. But then I think of the ten-euro USB dongle and the kernel panic, I think of the CVEs that come out every week in drivers nobody reads, I think of Minix running inside every Intel chip without anyone knowing, and the certainty dissolves a little.

I think what was really lost in 1992 wasn't an argument, it was an opportunity. It would have been possible to have both things, the open community and the clean architecture, if Minix had been free from the start and if Tanenbaum had wanted a system that grew instead of a system that taught. It didn't happen. And now here we are, patching the walls by hand. I don't quite know how to end this.