README
¶
Fuzzy Raft
Inspired by http://colin-scott.github.io/blog/2015/10/07/fuzzing-raft-for-fun-and-profit/ this package is a framework and set of test scenarios for testing the behavior and correctness of the raft library under various conditions.
Framework
The framework allows you to construct multiple node raft clusters, connected by an instrumented transport that allows a test to inject various transport level behaviors to simulate various scenarios (e.g. you can have your hook fail all transport calls to a particular node to simulate it being partitioned off the network). There are helper classes to create and Apply well know sequences of test data, and to examine the final state of the cluster, the nodes FSMs and the raft log.
Running
The tests run with the standard go test framework, run with go test . [from this dir] or use make fuzz from the parent directory. As these tests are looking for timing and other edge cases, a pass from a single run isn't enough, the tests needs running repeatedly to build up confidence.
Test Scenarios
The follow test scenario's are currently implemented. Each test concludes with a standard set of validations
- Each node raft log contains the same set of entries (term/index/data).
- The raft log contains data matching the client request for each call to raft.Apply() that reported success.
- Each node's FSM saw the same sequence of Apply(*raft.Log) calls.
- A verifier at the transport level verifies a number of transport level invariants.
Most tests run with a background workload that is constantly apply()ing new entries to the log. [when there's a leader]
TestRaft_LeaderPartitions
This creates a 5 node cluster and then repeated partitions multiple nodes off (including the current leader), then heals the partition and repeats. At the end all partitions are removed. [clearly inspired by Jepson]
TestRaft_NoIssueSanity
Is a basic 5 node cluster test, it starts a 5 node cluster applies some data, then does the verifications
TestRaft_SlowSendVote
Tests what happens when RequestVote requests are delaying being sent to other nodes
TestRaft_SlowRecvVote
Tests what happens when RequestVote responses are delaying being received by the sender.
TestRaft_AddMembership
Starts a 3 node cluster, and then adds 2 new members to the cluster.
TestRaft_AddRemoveNodesNotLeader
Starts a 5 node cluster, and then then removes 2 follower nodes from the cluster.
TestRaft_RemoveLeader
Starts a 5 node cluster, and then removes the node that is the leader.
TestRaft_RemovePartitionedNode
Starts a 5 node cluster, partitions one of the follower nodes off the network, and then tells the leader to remove that node, then heals the partition.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Logger ¶
type Logger interface {
Log(v ...interface{})
Logf(s string, v ...interface{})
}
Logger is abstract type for debug log messages
type LoggerAdapter ¶
type LoggerAdapter struct {
// contains filtered or unexported fields
}
LoggerAdapter allows a log.Logger to be used with the local Logger interface
func (*LoggerAdapter) Log ¶
func (a *LoggerAdapter) Log(v ...interface{})
Log a message to the contained debug log
func (*LoggerAdapter) Logf ¶
func (a *LoggerAdapter) Logf(s string, v ...interface{})
Logf will record a formatted message to the contained debug log
type TransportHooks ¶
type TransportHooks interface {
// PreRPC is called before every single RPC call from the transport
PreRPC(src, target string, r *raft.RPC) error
// PostRPC is called after the RPC call has been processed by the target, but before the source see's the response
PostRPC(src, target string, r *raft.RPC, result *raft.RPCResponse) error
// PreREquestVote is called before sending a RequestVote RPC request.
PreRequestVote(src, target string, r *raft.RequestVoteRequest) (*raft.RequestVoteResponse, error)
// PreAppendEntries is called before sending an AppendEntries RPC request.
PreAppendEntries(src, target string, r *raft.AppendEntriesRequest) (*raft.AppendEntriesResponse, error)
}
TransportHooks allow a test to customize the behavior of the transport. [if you return an error from a PreXXX call, then the error is returned to the caller, and the RPC never made]
Source Files