📰 newsreader

Virtual Oxide Labs: Emulating a Full Oxide Rack

The trouble with testing and developing for rack-scale hardware is that rack-scale is not all that easy to come by. I mean, nevermind the whole DDR5-as-a-currency thing; even if all the parts were available in abundance, any completed racks go swiftly to the front lines (AKA our customers' data centers).

As a member of the Product Assurance Team at Oxide, I find that totally unfair yet completely reasonable. And so we have to get a bit creative. We have to be able to:

Replicate diverse customer networking topologies

Run our actual control plane

Test up to date versions of our software

Represent the hardware components (we make racks, after all) including our switches and microcontrollers

And the cherry on top: we need to do that without cutting special "test version" releases. Because we want to keep our CI sleek, but also because that wouldn’t be playing fair.

Now, don’t get me wrong — we’re not without our tricks. When I joined Oxide last year, one of the first things I was introduced to was our racklettes; that is, smaller than average racks of four sleds and two switches each. These racklettes can be initialized and run just like a real rack, precisely because they are a real rack. Just not in any form factor it would be economical to sell. But for testing they work like a charm. We have four of them today, soon to be eight, where we can test any combination of hardware/software requirement.

We also have our various simulated methods of running Omicron. You can read more about those at How to Run Simulated and the full(er) How to Run Guide. I won’t go into a ton of detail here, but suffice it to say that most parts of our control plane (and more) can be run independently, in tandem, and in various degrees of simulation.

Do you remember the game Q*bert? It was one of the first wildly popular isometric games, and looked a little like this:

Why am I showing you a ridiculously small game screenshot? Well, each cube in that image represents a 'voxel,' or a point in 3D space. It’s actually the 3D version of a pixel. Volume Element + Pixel = Voxel. Neat, right?

Anyway, for Oxide, Virtual OXidE Labs (or Voxel) means an incredible amount of fidelity packed into a single piece of hardware. It is the spiritual successor to our prior virtual labs (known as a4x2) which allowed us to hack out decent racks running in VMs.

Pay attention to that word "fidelity" by the way. For it is, as Bryan Cantrill puts it, the Prime Directive: labs should use stock Oxide software without hack or modification.

This is what Voxel gives us. A complete software emulation stack, built on illumos in Rust, where we can build and test a variety of realistic configurations:

Sled virtualization provided by Propolis

BGP (numbered and unnumbered) or static networking with Debian and FRR

P4-powered switches via Oxide’s own SoftNPU, a software P4 switch

Microcontroller simulation and emulation for our Service Processor (SP) and Root of Trust (RoT)

And what’s more, it gives us all these features in a fairly small footprint. At home, I run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and 96GB RAM. On that computer, I can spin up 6 sleds with ease, either in the form of one rack or in a multirack configuration (two separate racks, one management plane). All it requires is Oxide Helios and a decently fast NVMe drive. Launch, and in about 10 minutes you have a full Oxide rack with each "sled" in its own Propolis VM — alongside switches (also in Propolis) and other infrastructure.

Remember how I said we have to use stock Oxide software? That actually makes things easier. The first thing we do (following a common practice in hypervisor tooling) is to create a reusable snapshot.

Oxide sleds are only ever one of two things: a normal sled (a "gimlet" for the first generation, "cosmo" for the current) or a "scrimlet," which is a sled wired to the sidecar switches that runs a switch zone. It’s all amazingly elegant in its complexity, you should read more about it in our networking docs.

To create a snapshot for Voxel, all we must do is create a "sled" (virtual machine), install our control plane (Omicron) on it, snapshot it before the point of configuration, et voilà: a ready-to-launch sled. The control plane can be installed one of two ways:

- From source

Voxel clones the Omicron repo on GitHub (or uses a locally defined source for rapid development) and uses

cargo

to compile- From TUF repo

We use Tufaceous to package Omicron releases using The Update Framework (TUF) and our CI tool Buildomat

Using the second option as an example, here’s the beginning of the process where voxel image create

pulls out the key pieces that make up the sled config:

$ pfexec ./target/debug/voxel image create --from-tuf ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip 5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c

[voxel] TUF source ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip: system version 23.0.0-0.ci+git5d176e0be7e, omicron 5d176e0be7e

[voxel] extracted 12 zones from ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip

[voxel] extracted 3 measurement corpus artifacts

[voxel] extracting host artifacts -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-host

[voxel] host boot image 1258295296 bytes, phase 1 rom 33554432 bytes

[voxel] extracted SP/RoT firmware -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-fw/ca9184058bd7

After a decent amount of unpacking and staging, it will launch a VM that everything gets written/installed to. And finally:

[voxel] ready: voxel-cp version=unknown unpacked_artifacts=18 built=2026-09-06T21:37:58

[voxel] clearing device-instance map, halting

[voxel] flushing, up to 25s

[voxel] stopping hypervisor

[voxel] writing voxel image voxel-cp-5d176e0be7e-tuf

That image is a zfs snapshot which can be used and reused to create our rack.

Launching the rack uses a similar process, except the control plane is already installed. All we need at this point is:

An "frr" image (the Debian VMs for networking)

A control plane image (like the one we just created)

The

voxel-init

agent that gives each virtual sled its assignment

The log output is pretty long, but you can scroll through it if you’d like. I’ll recap after.

$ pfexec ./target/debug/voxel launch --emu

[voxel] sled-agent config schema (image properties): data_links=Tagged disks=Hardcoded

Writing key pair to "./cargo-bay/sprockets/test-root-a.key.pem"

Writing key pair to "./cargo-bay/sprockets/test-signer-a1.key.pem"

...

Writing certificate to "./cargo-bay/sprockets/test-root-a.cert.pem"

Writing certificate to "./cargo-bay/sprockets/test-signer-a1.cert.pem"

...

Writing pki path to "./cargo-bay/sprockets/test-sprockets-auth-0.certlist.pem"

Writing pki path to "./cargo-bay/sprockets/test-alias-0.certlist.pem"

...

add net fdb0:a840:2500:1::/64: gateway fdb0:a840:25ff:0::2

add net fdb0:a840:2500:5::/64: gateway fdb0:a840:25ff:0::4

[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300

[identity] seed = 0x2aac7ae3f8cfbb1e (3074967765137734430)

[identity] SP UID 13120858064df80da1642f2b, RoT UUID 00a94a8e206f3f138a790fba1b12db36

[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/sidecar.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/archives/sp.zip

flashed 786688 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/sp-flash.bin

[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310

[identity] seed = 0xa65abed0e20d08d1 (11987103162554976465)

[identity] SP UID 72a3c3327682e0df38de478a, RoT UUID fcb600f7322f3393899f72036aa2b4fd

[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/archives/sp.zip

flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/sp-flash.bin

[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320

[identity] seed = 0xbab29f9c81fda354 (13452990531453690708)

[identity] SP UID 0883f59f877ab40da88e8072, RoT UUID 15468fb5bffd30d3b381e8759621fb4f

[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/archives/sp.zip

flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/sp-flash.bin

[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330

[identity] seed = 0xea954465eef85936 (16903491980996663606)

[identity] SP UID 49c0e331f4877c7d0afc71eb, RoT UUID 03ec945840463823a95d38dda9e261d1

[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/archives/sp.zip

flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/sp-flash.bin

[voxel] rack 0 SP fleet up on fdb0:a840:25ff:0::1 (4 SP(s): [33300, 33310, 33320, 33330])

Sep 06 22:15:47.126 INFO launching runner: deployment(voxel) persistent(true) custom_propolis_binary(Some("/export/home/helios/propolis-server")) dataset(testbed/falcon) falcon_dir(.falcon)

Sep 06 22:15:47.126 INFO starting preflight for deployment voxel

Sep 06 22:15:48.162 INFO creating links

Sep 06 22:15:48.162 INFO creating simnet link 'voxel_ce_vn_sim0'

Sep 06 22:15:48.164 INFO creating vnic link 'voxel_ce_vn_vnic0'

...

Sep 06 22:15:48.249 INFO creating external links

Sep 06 22:15:48.249 INFO creating external link voxel_ce_vn_vnic2

Sep 06 22:15:48.251 INFO creating external link voxel_g0_vn_vnic3

...

Sep 06 22:15:48.255 INFO creating nodes

Sep 06 22:15:48.255 INFO g0: launching node

Sep 06 22:15:48.257 INFO g1: launching node

...

Sep 06 22:15:48.290 INFO launched instance g0 with pid 13041 on port 43736

Sep 06 22:15:48.291 INFO g0: instance ensure

Sep 06 22:15:48.295 INFO launched instance g1 with pid 13042 on port 33187

Sep 06 22:15:48.295 INFO g1: instance ensure

Sep 06 22:15:48.297 INFO launched instance g2 with pid 13043 on port 51110

...

Sep 06 22:15:52.423 INFO g1: instance ensure completed after 0 retries

Sep 06 22:15:52.423 INFO g1: instance run

Sep 06 22:15:52.479 INFO g2: instance ensure completed after 0 retries

Sep 06 22:15:52.479 INFO g2: instance run

Sep 06 22:15:52.534 INFO g0: instance ensure completed after 0 retries

Sep 06 22:15:52.534 INFO g0: instance run

Sep 06 22:16:03.961 INFO cr1: mounting /opt/cargo-bay

Sep 06 22:16:03.969 INFO cr2: mounting /opt/cargo-bay

Sep 06 22:16:03.994 INFO cr1: finished mounting /opt/cargo-bay

Sep 06 22:16:03.995 INFO ce: mounting /opt/cargo-bay

Sep 06 22:16:04.002 INFO cr2: finished mounting /opt/cargo-bay

...

Sep 06 22:16:09.236 INFO ce: launch start

Sep 06 22:16:09.237 INFO cr1: launch start

Sep 06 22:16:09.237 INFO cr2: launch start

Sep 06 22:16:12.893 INFO ce: [voxel-init] pinned enp0s10 DHCP client-id to MAC; re-DHCPing host-LAN uplink

Sep 06 22:16:12.893 INFO ce: [voxel-init] NAT rack egress via enp0s10

Sep 06 22:16:12.894 INFO ce: [voxel-init] static edge IP 192.168.68.170/22 added on enp0s10

Sep 06 22:16:12.894 INFO ce: [voxel-init] router bring-up complete

Sep 06 22:16:12.894 INFO ce: launch ok

Sep 06 22:16:12.956 INFO cr1: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink

Sep 06 22:16:12.956 INFO cr1: [voxel-init] NAT rack egress via enp0s11

Sep 06 22:16:12.956 INFO cr1: [voxel-init] router bring-up complete

Sep 06 22:16:12.956 INFO cr1: launch ok

Sep 06 22:16:12.985 INFO cr2: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink

Sep 06 22:16:12.985 INFO cr2: [voxel-init] NAT rack egress via enp0s11

Sep 06 22:16:12.985 INFO cr2: [voxel-init] router bring-up complete

Sep 06 22:16:12.985 INFO cr2: launch ok

Sep 06 22:16:12.985 INFO g0: launch start

Sep 06 22:16:12.986 INFO g1: launch start

Sep 06 22:16:12.987 INFO g2: launch start

Sep 06 22:16:28.581 INFO g1: [voxel-init] underlay(jumbo)=["vioif0", "vioif1"] ext-candidates=["vioif2"]

Sep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0 (2FAKE001-M20): M.2 layout written

Sep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0: host phase 2 seeded

Sep 06 22:16:28.581 INFO g1: [voxel-init] c3t1d0 (2FAKE001-M21): M.2 layout written

Sep 06 22:16:28.582 INFO g1: [voxel-init] c3t1d0: host phase 2 seeded

Sep 06 22:16:28.582 INFO g1: [voxel-init] c4t1d0 (2FAKE001-U20): labeled

Sep 06 22:16:28.582 INFO g1: [voxel-init] c5t1d0 (2FAKE001-U21): labeled

Sep 06 22:16:28.582 INFO g1: [voxel-init] c6t1d0 (2FAKE001-U22): labeled

Sep 06 22:16:28.582 INFO g1: [voxel-init] c7t1d0 (2FAKE001-U23): labeled

Sep 06 22:16:28.582 INFO g1: [voxel-init] c8t1d0 (2FAKE001-U24): labeled

Sep 06 22:16:28.582 INFO g1: [voxel-init] 7 sled disks (2 M.2, 5 U.2)

Sep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_626cc492-5a93-4b5c-833b-c45a504d8300)

Sep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_711d472d-edbc-4426-9e33-4b000914f4da)

Sep 06 22:16:28.582 INFO g1: [voxel-init] sled-agent activated

Sep 06 22:16:28.582 INFO g1: [voxel-init] gimlet bring-up complete

Sep 06 22:16:28.582 INFO g1: launch ok

Sep 06 22:16:33.159 INFO g0: [voxel-init] underlay(jumbo)=["vioif1", "vioif2"] ext-candidates=["vioif0", "vioif3"]

Sep 06 22:16:33.159 INFO g0: [voxel-init] SP network fdb0:a840:25ff:0::2/64 on vioif3 (deprecated as a source)

Sep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0 (2FAKE000-M20): M.2 layout written

Sep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0: host phase 2 seeded

Sep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0 (2FAKE000-M21): M.2 layout written

Sep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0: host phase 2 seeded

Sep 06 22:16:33.159 INFO g0: [voxel-init] c4t1d0 (2FAKE000-U20): labeled

Sep 06 22:16:33.159 INFO g0: [voxel-init] c5t1d0 (2FAKE000-U21): labeled

Sep 06 22:16:33.159 INFO g0: [voxel-init] c6t1d0 (2FAKE000-U22): labeled

Sep 06 22:16:33.159 INFO g0: [voxel-init] c7t1d0 (2FAKE000-U23): labeled

Sep 06 22:16:33.160 INFO g0: [voxel-init] c8t1d0 (2FAKE000-U24): labeled

Sep 06 22:16:33.160 INFO g0: [voxel-init] 7 sled disks (2 M.2, 5 U.2)

Sep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_ab4a1425-a7f2-47c7-9ef8-c70180ef02c9)

Sep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_410f70f9-1eea-49fe-8aae-8331cebc4ea1)

Sep 06 22:16:33.160 INFO g0: [voxel-init] switch0 enforcer started (pid 769), log /tmp/switch-enforcer.log

Sep 06 22:16:33.160 INFO g0: [voxel-init] sled-agent activated

Sep 06 22:16:33.160 INFO g0: [voxel-init] gimlet bring-up complete

Sep 06 22:16:33.160 INFO g0: launch ok

Sep 06 22:16:38.179 INFO g2: [voxel-init] underlay(jumbo)=["vioif1", "vioif2"] ext-candidates=["vioif0", "vioif3"]

Sep 06 22:16:38.179 INFO g2: [voxel-init] SP network fdb0:a840:25ff:0::4/64 on vioif3 (deprecated as a source)

Sep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0 (2FAKE002-M20): M.2 layout written

Sep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0: host phase 2 seeded

Sep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0 (2FAKE002-M21): M.2 layout written

Sep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0: host phase 2 seeded

Sep 06 22:16:38.179 INFO g2: [voxel-init] c4t1d0 (2FAKE002-U20): labeled

Sep 06 22:16:38.179 INFO g2: [voxel-init] c5t1d0 (2FAKE002-U21): labeled

Sep 06 22:16:38.179 INFO g2: [voxel-init] c6t1d0 (2FAKE002-U22): labeled

Sep 06 22:16:38.179 INFO g2: [voxel-init] c7t1d0 (2FAKE002-U23): labeled

Sep 06 22:16:38.179 INFO g2: [voxel-init] c8t1d0 (2FAKE002-U24): labeled

Sep 06 22:16:38.179 INFO g2: [voxel-init] 7 sled disks (2 M.2, 5 U.2)

Sep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_270f44df-c4df-43fd-b02f-e5440857b547)

Sep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_1c986103-d332-4270-b7e1-9341310a621d)

Sep 06 22:16:38.179 INFO g2: [voxel-init] switch1 enforcer started (pid 779), log /tmp/switch-enforcer.log

Sep 06 22:16:38.179 INFO g2: [voxel-init] sled-agent activated

Sep 06 22:16:38.179 INFO g2: [voxel-init] gimlet bring-up complete

Sep 06 22:16:38.179 INFO g2: launch ok

Sep 06 22:16:38.652 INFO rack-init: waiting for the commission API: switch-zone bootstrap address: no bootstrap address on oxz_switch

Sep 06 22:17:54.860 INFO rack-init: waiting for the commission API: tunnel opened

Sep 06 22:17:59.876 INFO rack-init: commission API reachable on g0

Sep 06 22:18:23.892 INFO rack-init: 3 bootstrap sleds ready

..+.......+.....+.+..+.......+..+...............+....+.....+.+.....+......+....+...+..+....+.....+++++++++++++++++++++++++++++++++++++++*.....+.+.....+......+.+++++++++++

............+..+.+..+...+.........+......+...+..........+++++++++++++++++++++++++++++++++++++++*...+...+..+++++++++++++++++++++++++++++++++++++++*..................+.....

-----

Sep 06 22:18:24.023 INFO rack-init: commission-driven RSS started

Sep 06 22:18:24.027 INFO rack-init: watching RSS progress on the RSS node ...

Sep 06 22:18:24.228 INFO rack-init: polling RSS status via ssh root@192.168.68.167

Sep 06 22:18:32.410 INFO rack-init [9/16]: initializing internal DNS

Sep 06 22:18:49.186 INFO rack-init [11/16]: initializing NTP

Sep 06 22:19:13.745 INFO rack-init [12/16]: waiting for time sync

Sep 06 22:20:51.850 INFO rack-init [12/16]: waiting for time sync ... still working (1m in this step)

Sep 06 22:22:22.134 INFO rack-init [13/16]: waiting for database

Sep 06 22:22:47.334 INFO rack-init [14/16]: initializing cluster

Sep 06 22:22:55.805 INFO rack-init [15/16]: initializing zones

Sep 06 22:24:33.717 INFO rack-init [16/16]: handing off to Nexus

Sep 06 22:26:11.824 INFO rack-init [16/16]: handing off to Nexus ... still working (1m in this step)

Sep 06 22:27:49.894 INFO rack-init [16/16]: handing off to Nexus ... still working (3m in this step)

Sep 06 22:28:06.231 INFO rack-init: complete, rack initialized (RSS run b20a2f8d-e511-4a67-a666-7b74b2ce68d7)

Sep 06 22:28:06.231 INFO launch complete

Sep 06 22:28:06.250 INFO external route set: 198.51.100.0/24 -> 192.168.68.170 (ce)

Sep 06 22:28:06.292 INFO rack: external network reachable (dns 198.51.100.20)

Like I said, a lot of steps. Here’s what happened:

Trust material is generated in the form of sprockets PKI

Emulated STM32H753 and LPC55 microcontrollers are flashed with Hubris firmware

Our automated network topology suite (known as falcon) wires the topology with simnet and VNIC pairs for every link, and stages the sleds (g0, g1, and g2 in this case)

Propolis VMs launch; 6 total in this example, with 3 driving the network and 3 as Oxide sleds

voxel-init

runs: a lightweight Rust agent on each node thatSets up NAT and uplink addressing

Configures underlay setup on each sled

Writes the M.2 partition layout on emulated boot disks and labels five U.2 data disks

Seeds the host phase 2 image and install dataset to the M.2s

Assigns the scrimlets and runs a switch enforcer so they "remember" their identity across restarts

Activates

sled-agent

, the per-sled configuration layer for the control plane

Connects to the commission API for

wicketd

, which brings the rack through its initialization phases

It’s a complex dance of steps, and I really buried the lede on the whole "emulated

microcontrollers" front. But suffice it to say these are the complete and right steps for

any Oxide rack, except the hardware is represented as distinct pieces of code. For example,

need a new interconnect between two virtual racks? Add a couple lines of code in rack.rs

.

There are a few things that don’t follow the Prime Directive (yet!). For example, those certs that make up the sprockets PKI up top aren’t set at manufacturing and coming from the RoT like our production racks. But that’s one of those things that we can live with for now, as it doesn’t affect actual rack runtime.

This is the bit I mentioned a moment ago. A big part of what makes our racks unique is

the embedded RoT and SP

in every sled. We’ve long had an SpSim

construct that served as a stub for the service

processor and would return the right info at the right times. But that’s simulation, not

emulation. It’s always going to be happy path, and that’s just not enough fidelity for

real testing cases.

As such, we built actual emulated Cortex-M7 and Cortex-M33 devices for illumos and Linux (x86 and ARM). The repo for this tool (sp-emu) is open source. Each of the two devices runs in a single thread, and (mostly) faithfully emulates the three main requirements: instructions, time, and interrupts.

From there, you can flash our Hubris firmware directly to these emulated devices, and then communicate by way of an ordinary socket. The Voxel lab can speak with these virtual devices via the Management Gateway Service (MGS) and Inter-Processor Communication Channel (IPCC). You can learn everything you want and more about these concepts in the Management Gateway Service repo and Hubris docs.

There really is no "run real microcontrollers as virtual devices with this one neat trick!"

I can think of. But the real challenge of sp-emu’s job is time. Remember all of the code

we’re running there is meant for silicon. Hubris schedules off a 1ms SysTick and the emulator doesn’t

use wall time; as such, we had to invent a clock. That said, emulated hardware is also

slower than the real thing. So there’s… well, less time per time. So instead of trying

to get all temporal about it, time became instructions. For example, I give you maybe_tick

:

if self.systick == 0 {

self.systick = self.syst_rvr;

}

self.systick -= 1;

if self.systick == 0 {

self.systick = self.syst_rvr;

self.tick_events = self.tick_events.wrapping_add(1);

if self.syst_csr & 2 != 0

&& self.mode == Mode::Thread

&& !self.primask

{

self.exception_entry(15, bus); // SysTick exception

}

}

In this case, we count instructions against the kernel’s own reload value and fire exceptions anywhere the single-priority design makes it safe to take it — in thread mode, interrupts unmasked. By the same token, peripheral access costs one tick of our clock (an instruction), whereas on silicon it might stall for dozens of cycles on the bus.

You may have noticed self.exception_entry

which is defined

elsewhere in the code:

fn exception_entry(&mut self, vecnum: u32, bus: &mut Bus)

In short, this function models everything the hardware does on a Cortex-M core taking

an exception, all the way between "exception becomes active" to the first handler

instruction. vecnum

here is an ARMv7-M exception number (also the vector table index

for that exception’s handler). bus

is the memory system which is used to actually

push the frame and read the table.

Hubris has a non-preemptible kernel, though it schedules tasks preemptively. SVCall, PendSV, SysTick and all IRQs run at the same priority level, while faults keep a higher priority. As such, the emulator doesn’t have to nest handlers. It processes pending IRQs based on the PRIMASK, FAULTMASK, and BASEPRI settings. If one of them pended PendSV, that will perform the deferred context switch for the kernel. For more details on how it works internally, you can check this lovely bit of documentation.

if let Some(irq) = bus.next_irq() {

if self.basepri == 0 || (bus.irq_prio(irq) as u32) < self.basepri {

bus.clear_pending(irq);

self.exception_entry(16 + irq as u32, bus);

return;

}

}

if bus.take_pendsv() {

self.exception_entry(14, bus);

}

There’s a lot more we can talk about here, but to sum it all up you can look at this pseudo-ish loop:

loop {

cpu.step(&mut bus, host.as_mut())?; // one Thumb-2 instruction

cpu.maybe_tick(&mut bus); // SysTick: time is instructions

cpu.maybe_interrupt(&mut bus); // NVIC and PendSV delivery

if cpu.cycles % 4096 == 0 {

bus.pump_eth(host.as_mut()); // MGS packets in and out

bus.pump_uart(host.as_mut());

}

}

Note: yes I’m aware I said there’s no cycles then have a field named cycles. That’s the "fake it till you make it" part of emulation.

That’s the overall heartbeat of sp-emu, and it runs ad infinitum. Everything else exists to make those calls seem real.

Past the actual working of the emulated sleds and microcontrollers, it’s all good ol'

fashion user experience. When you’re running that many components on a single system, it

can get really confusing trying to figure out what pieces are where, how to connect

to one sled on one rack versus another, etc. So the voxel

command that we use to

launch and destroy virtual racks also serves as a connectivity tool. For example:

$ pfexec ./target/debug/voxel host login g0

[voxel] ssh root@192.168.68.167 (g0 global zone)

The illumos Project helios-3.0.23976 May 2026

root@g0:~# zoneadm list

global

oxz_switch

oxz_internal_dns_5b07f3e7-444c-462c-9633-6967e52d6a3b

oxz_ntp_b308987c-3408-4077-b9a0-d803ed1e420a

oxz_cockroachdb_daa29fcd-fff0-4b3d-a722-b969e03398d9

oxz_cockroachdb_117986ee-8b2d-4567-8c6f-e01f3e63447e

oxz_crucible_310a91b5-1b67-46cb-b08d-ad6534702a3d

oxz_crucible_627c130b-97d6-4e8f-b13e-eb03b671c2a3

oxz_crucible_pantry_65e1b527-26fc-46fd-8584-d9792771926c

oxz_crucible_2692a74a-3a22-420c-857a-a3c786e4b600

oxz_external_dns_baa049eb-7bdd-4d8e-ace3-c0ef054845bd

oxz_crucible_2ffc1456-b0e1-4f8e-8b6d-6ab01377bd25

oxz_crucible_c5f7d004-9d9b-4741-9246-c4793b9c7976

oxz_nexus_c1a31871-bdb5-422c-b5b1-2de62aa31e30

That command logs the user straight into the g0 sled in the global zone. Alternatively:

$ pfexec ./target/debug/voxel tp login switch0

[voxel] ssh root@192.168.68.167 -> zlogin oxz_switch (g0 switch0)

[Connected to zone 'oxz_switch' pts/3]

Last login: Sun Sep 6 22:17:52 from fdb0:a840:2500:

The illumos Project helios-3.0.23976 May 2026

root@oxz_switch:~#

tp

in this case acts like our technician ports,

and allows the user to connect straight to a switch zone for rack-wide commands and networking.

There are also numerous commands to manage things like images (export/import, delete), config

(network topology, sled sizing, multi-rack, etc.), and more features coming as we build.