Simba 🐱 – Raft with deterministic simulation testing

Overview

Raft: A consensus algorithm. A group of nodes agrees on one shared log, even if some crash or messages are delayed.
Deterministic simulation testing: The clock, the network and the failures are all controlled by the engine. The same seed always gives the same run.
Matrix mode: The simulated Simba 🐱: time is a counter and the network is a queue of events.
Real mode: The same Raft with real timeouts, disk and sockets. Only the interfaces change.

What is Raft?

Leader election

Every node is a follower, a candidate or the leader. A follower that stops hearing from the leader becomes a candidate and asks for votes. With a majority, it becomes the leader.

The leader sends heartbeats and replicates its log. An entry is committed once a majority has stored it.

An election, in a loop.

How Simba 🐱 runs Raft

One Raft, two modes

Raft never touches time, storage or the network directly. It uses interfaces. Matrix mode plugs in simulated versions and Real mode plugs in the real ones.

Every node is just a struct with its own state. The engine moves them forward.

type Node struct { ID string Term int Role Role // follower, candidate, leader Log []Entry CommitIndex int }

type Time interface{ Tick() int } type Storage interface{ Save(...) } type Network interface{ Send(msg) }

Deterministic simulation testing

What is it?: Run the whole system inside a simulation where everything that could be random is driven by one seed.
Why?: Distributed bugs are rare and hard to reproduce. With a seed, a failing run can be replayed exactly, as many times as needed.
Fast: Time is not real, so hours of cluster life run in seconds. The limit is your CPU.

A time machine for your cluster

There is no real clock and no real network. Simba 🐱 never waits for a timeout or a packet: it just moves the counter forward.

So the only limit is how fast your CPU can run the loop. Days of cluster life fit in a few seconds, and every run can be replayed with its seed.

0.0 sreal time, your CPU

0 htime inside the cluster

10 s of CPU = 10 days of cluster

Illustrative numbers. No real time, no real network.

MatrixChecker

Thousands of seeds, looking for one bug

An internal tool. It takes many seeds, runs the simulation with each one at different fuzz levels, and watches Raft the whole time.

Raft is full of assertions: rules that must always be true. The moment one is broken, the run panics. That seed is a bug, and it can be replayed exactly.

one leader per term
committed entries never change
logs match on the same index and term

A rule Raft did not expect stops the run right away.

Time

Time is an integer

There is no system clock. The engine is a loop that adds one to a counter. Timeouts are just “wake me at tick N”.

for tick := 0; tick < maxTicks; tick++ { engine.Step(tick) }

tick0

Each tick is one step of the loop.

Network

The network is a queue of events

Nodes never call each other. A message is an event with a sender, a receiver and a delivery tick. The engine delivers it when time gets there.

The latency is simulated: each message gets a random delay drawn from probabilities set by the fuzz level.

low fuzzmedium fuzzhigh fuzz

event queuetick 0

Soonest delivery on top. Gone when its tick arrives.

Fault injection

The fuzz level decides how often things go wrong. Same seed, same faults.

Latency

A message takes some ticks to arrive. Sometimes few, sometimes many.

Duplication

The network delivers the same message twice. Raft must handle it.

Message loss

The message disappears in the network and never arrives.

Node crash

A node goes down, drops messages, and later comes back to life.

Contact

Questions, ideas, suggestions or business inquiries? Write to me at this email, or open an issue on GitHub.

GitHub LinkedIn [email protected]

Simba 🐱 is a custom implementation of Raft, tested with deterministic simulation.