At their core, golden tests compare the current output of your code against a pre-saved "golden" output. If the current output matches the golden output, the test passes. If they differ, the test fails, signaling a potential issue.
To illustrate the usefulness of golden tests, let's take the example of bla - a small Python library for performing code analysis. Its repository has a folder examples containing sample scripts showcasing its functionality.
One way to ensure that example scripts are not broken by code changes is to run them manually, the better way would be to automate it by adding "golden tests".
For golden tests to run, we need two things: something producing output and a reference output to compare it with.
The tested output is produced by invoking python examples/<example>.py
.
The corresponding reference is stored in tests/golden/expectations/<example>.out
.
See folder structure bellow:
.
├── .pre-commit-config.yaml
├── examples
│  ├── dekker.py
│  ├── halting.py
│  └── ...
├── tests
│ ├── golden
│ │ ├── expectations
│ │ │  ├── dekker.out
│ │ │  ├── halting.out
│ │ │  └── ...
│ │ └── run_tests.py
... # rest of the repo
The testing is done by tests/golden/run_tests.py. It's fairly simple:
Iterates over all examples;
Executes them and grabs the output;
Compares the output with the reference and reports a failure if a mismatch is found;
Why automatically update reference outputs?
If the code change intentionally affects the output, manually updating multiple reference outputs can be tedious. That's why I've added the --update
flag to the golden test runner.
There is a risk of updating the reference output to an undesired one by accident, but if your workflow includes a review of the changes before merging (e.g., PR reviews), this risk is mitigated.
To automatically execute golden tests before committing, I've added them as a pre-commit hook:
# .pre-commit-config.yaml
repos:
- repo: local
hooks:
- id: golden
name: golden
entry: python tests/golden/run_tests.py --update
language: system
types: [python]
pass_filenames: false
Here is an example of how it looks:
$ git commit -a -m "Add halting example; minor output refactoring"
Check Yaml...........................................(no files to check)Skipped
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
black....................................................................Passed
golden...................................................................Failed
- hook id: golden
- exit code: 1
- files were modified by this hook
Running golden tests:
halting - FAIL: tests/golden/expectations/halting.out not found ... updating
inconsistency - PASS
dekker - FAIL: mismatch ... updated
# pre-commit failed due to golden test failure
# I've had to review changes in reference output, and
$ git add tests/golden/expectations/halting.out
$ git commit -a -m "Add halting example; minor output refactoring"
# OK