Unity testing framework with physics interaction

Most developers understand the importance of automated testing, but in practice very few actually do something about it. It is probably perceived as an overhead, but with the right tools and a little practice one can become very efficient and write bullet proof code. The goal of this article is to give some practical example of testing high level interaction that involves 3D physics (rigid bodies) simulation.

Prerequisites knowledge:

• Test Driven Development (TDD) concept
• Unity basics
• Coroutines
• Rigid body interaction
• Unity Test Runner and UTF basics

Topics that will be covered:

• Determinism
• Setup and tear down
• load and unload additional scenes
• Manual physics simulation

Unity Testing Framework (UTF) is quite extensive and well integrated with the editor. There are two types of tests: Edit mode and play mode tests. Edit mode tests are used for 'classic' unit testing of functions, but they are quite limited in what they can do (in Unity). With play mode test you can check more complicated interactions that may need to run over a few frames. The upside of high level tests is getting high coverage quickly and the downside is less trivial tracking of root cause when a test fails. Ideally we should use both types, but if I have limited time and need to choose between the two, I would go for high level play mode tests.

When we write unit test, we would expect the same output given the same input (AKA determinism). It sound like a very strict requirement but fortunately unity and its default physics system are deterministic, under some conditions that we will demonstrate.

The following example drops a few rigid bodies on top of each other and compare the state of the objects after running the same simulation a few times. We start by creating a new test file and class by the same name: PhysicsTest. All the subsequent code will be added to this class (unless explictly stated otherwise).

public class PhysicsTest
{
}        

Let's create 10 dynamic objects on top of each other with a small lateral offset. After that, we add a static plane collider below the dynamic objects. We save references to the objects we created, since we want to track their state after they fall and collide.

List<GameObject> objects = new List<GameObject>();


void CreateObjects()
{
	objects.Clear();
	// create dynamic objects
	for (int n=0; n<10;++n)
	{
		var go = GameObject.CreatePrimitive(n % 2 == 0 ? PrimitiveType.Cube : PrimitiveType.Sphere);
		go.AddComponent<Rigidbody>();
		go.transform.position = new Vector3(n * 0.5f, n * 1.5f, 0.0f);
		objects.Add(go);
	}
	// create floor collider
	var plane = GameObject.CreatePrimitive(PrimitiveType.Plane);
	plane.transform.position = new Vector3(0, -2, 0);
	objects.Add(plane);
}        

The initial state will look like this

We let the game run for 100 update frames and save the state of the objects in the end.

List<PhysicsState> states = null;



struct PhysicsState
{
	public Vector3 position;
	public Quaternion rotation;
};



IEnumerator Run()
{
	CreateObjects();
	for (int n = 0; n < 100; ++n)
		yield return null;
	states = new List<PhysicsState>();
	foreach (var go in objects)
	{
		var state = new PhysicsState
		{
			position = go.transform.position,
			rotation = go.transform.rotation
		};
		GameObject.Destroy(go);
		states.Add(state);
	}
	Debug.Log($"First object position {states[0].position.ToString("G17")}");
}        

The final state of the objects will look like this

We are going to run the simulation once, save the state of the objects, and then run it additional 4 times, comparing the final state of each run to the first run. Since the first run isn't really testing anything, lets use the [UnitySetup] for that:

List<PhysicsState> initialStates
[UnitySetUp]
public IEnumerator Setup()
{
	yield return Run();
	initialStates = states;
};        

For the other runs, we finally get to write the test function.

[UnityTest]
public IEnumerator PhysicsTestDeterminism()
{        
	for (int numRuns=0; numRuns<4; ++numRuns)
	{
		yield return Run();
		for (int obj=0; obj< states.Count; ++obj)
		{
			Assert.AreEqual(initialStates[obj].position, states[obj].position, $"Compare position failed on run {numRuns}, object {obj}");
			Assert.AreEqual(initialStates[obj].rotation, states[obj].rotation, $"Compare rotation failed on run {numRuns}, object {obj}");
		}            
	}                
}        

The test fails and we get the following output

PhysicsTestDeterminism (0.236s)
---
Compare position failed on run 0, object 0
  Expected: (0.0, -0.1, 0.0)
  But was:  (0.0, -0.1, 0.0)
---
First object position (0, -0.0824039951, 0)
First object position (0, -0.0588599965, 0)        

The precision in the assert message is not high enough but our debug log shows there is indeed a difference. Let's run the test one more time:

PhysicsTestDeterminism (0.191s)
---
Compare position failed on run 0, object 0
  Expected: (0.0, -0.1, 0.0)
  But was:  (0.0, 0.0, 0.0)
---
First object position (0, -0.0588599965, 0)
First object position (0, -0.0235439986, 0)        

We get different result between tests and between runs in the same test, a glorious failure! The prime suspect is related to the duration of the test. 100 frames may take different time to run and therefore the number of fixed updates may vary. We can fix this by replacing the yield return class from null to new WaitForFixedUpdate(). It will work in the current example, but in other cases where the game logic is split between Update and FixedUpdate it may fail. The robust solution it to ensure the number of fixed updates between each update. This can be achieved with captureDeltaTime as follows:

[OneTimeSetUp]
public void OneTimeSetUp()
{
	Time.captureDeltaTime = 2 * Time.fixedDeltaTime;
}


[OneTimeTearDown]
public void OneTimeTearDown()
{
	Time.captureDeltaTime = 0;
}        

Let's run the test twice and check the output:

PhysicsTestDeterminism (0.262s)
---
Compare position failed on run 0, object 0
  Expected: (0.0, -1.5, 0.0)
  But was:  (0.0, -1.5, 0.0)
---
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00311325886, -1.49999988, 0.00205041328)
==================================================================
PhysicsTestDeterminism (0.269s)
---
Compare position failed on run 0, object 0
  Expected: (0.0, -1.5, 0.0)
  But was:  (0.0, -1.5, 0.0)
---
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00311325886, -1.49999988, 0.00205041328)        

Both tests show the same results, but the results between two runs of the same test are still different. Why would we get any difference if we destroy and recreate all the objects with the same initial state? This issue is more elusive. Apparently physics scene keeps some internal state that affects the simulation. To make the simulation deterministic on the same machine you have to reset or recreate the physics scene. Since currently unity doesn't expose a direct interface to recreate a physics scene, the only alternative is to create a new scene with a local physics scene. We add these lines at the beginning of the Run method:

var newScene = SceneManager.CreateScene("PhysicsScene", new CreateSceneParameters(LocalPhysicsMode.Physics3D));
SceneManager.SetActiveScene(newScene);        

And these at the end of the Run method to unload the scene:

var op = SceneManager.UnloadSceneAsync(newScene);
while (!op.isDone)
	yield return null;        

Since we unload the entire scene (including the objects in it), we no longer need to manually delete the game objects we created during the test.

The next thing we should note is that local physics scenes are not simulated automatically. We need to decide when to advance the simulation and what is the size of the step. For simplicity, we will do exactly what the automatic simulation does using this custom PhysicsStepper component (in a different file):

public class PhysicsStepper : MonoBehaviour
{        
    void FixedUpdate()
    {
        gameObject.scene.GetPhysicsScene().Simulate(Time.fixedDeltaTime);
    }
}        

And we need to attach this component to one of the game object in our new scene:

plane.AddComponent<PhysicsStepper>();        

Putting all this together, the test pass and we get the following output:

PhysicsTestDeterminism (0.795s)
---
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00259990082, -1.49999964, 0.00184755772)
First object position (-0.00259990082, -1.49999964, 0.00184755772)        

Determinism achieved!

One last change before we wrap up. Since the test may throw some errors, a good practice will be to unload the scene during UnityTearDown. This way we make sure there are no leftover objects from one test that affect the following tests (if we are running multiple tests). We refactor the unload lines from the Run method and add the following:

IEnumerator UnloadScene()
{
	if (localScene.isLoaded)
	{
		var op = SceneManager.UnloadSceneAsync(localScene);
		while (!op.isDone)
			yield return null;
	}
}


[UnityTearDown]
public IEnumerator TearDown()
{
	// in case of error during the test, the scene may remain loaded
	yield return UnloadScene();
}        

Summary

• Ensure the number of fixed updates per update is consistent
• Create a new scene with local physics mode
• Add manual simulation for the new physics scene
• Use setup and tear down to save original state and to restore it after the test

Final project code is available in GitHub.

Useful resources:

• Unity Test Framework Manual
• Everything you need to know about Testing In Unity3D (even if you've never written a unit test)
• NVidia PhysX determinism