What I learned from Snapshot Testing for the Masses

Most of the time you have to perform tests to verify that a system is working correctly. There are several methods for testing, unit tests being one of them. But what if the data changes during the development phase of the application. Which means more tests make the code base less changeable and development significantly slower. This is where I initially saw the perfect use case for snapshot testing. Snapshot testing enables the developer to save the results of one specific test in ASCII representation. The ASCII string is saved in a file and can be compared with the output of the given function to verify the functionality. This ASCII file can be updated easily if the API and thus the data changes.

TigerBeetle is a financial transaction database written in Zig. In this domain testing is vital to succeed. The blog post Snapshot Testing for the Masses by matklad was an inspiration for me to go deeper into the topic snapshot testing.

The Definition of Snapshot Testing

ChatGPT 4o says:

Snapshot testing is a type of testing primarily used in software development, especially with user interfaces (UIs). It involves capturing a snapshot (or serialized representation) of a component’s output at a specific point in time and then comparing future outputs to this saved snapshot. The goal is to ensure that the component’s behavior or appearance doesn’t change unexpectedly.

Furthermore, it says that snapshot testing is used primarily in UI testing. This point is interesting for me because I know snapshot testing particularly from CLI applications. In this blog article I will discuss how TigerBeetle uses snapshot testing in their application built in Zig. In addition to that we will look at snapshot testing at Jane Street, a trading firm which primarily uses Ocaml as their programming language of choice. Then I will show some examples of snapshot testing in the wild with Jest (JavaScript/TypeScript), Py-Test (Python), Expect-Test (Ocaml), OhSnap (Zig), and Insta (Rust).

Snapshot Testing at TigerBeetle

TigerBeatle wrote their own snapshot testing library in the course of developing their product. They use it for example in the argument parser for the command line. This parser checks whether the arguments to the command line are parsed correctly. If the arguments are wrong it checks whether the right error message is displayed. The result of the test argument parser is displayed in text form for the standard output and the standard error. If the output is not as simple as comparing the strings you can adjust the comparison function for the tests. This enables you to test cases where there is, for example, a current time involved which changes whenever the test is executed. This time can be exclusively checked for a specific string pattern instead of one to one matching (e.g. \d\dh:\d\dm:\d\ds for 10h:20m:30s). The conclusion of the article is clear: Snapshots are great, text is a powerful narrative to display even complex data structures, in this case self aware strings are used which remember their position in source code, and keep snapshot testing in mind for tests that would make code less changeable.

Snapshot Testing at Jane Street

JaneStreet, in the same way as TigerBeatle, built their own snapshot testing library or as they call it expect tests. The reason matches those of TigerBeatle. Expect tests are for operations that could slow you down while developing. Imagine having to think hard to come up with the right result for a complex function just to test it. Yes you should think hard what you are testing and what the result should be but what if you are suddenly writing out a full web page in HTML or a TUI output or make the 100th calculation with our calculator of choice to get the expected result from a complex math function. Here expect tests come in clutch with the ability to update your source code tests on the fly. The only thing you need is a text representation of the thing you want to test. This might be a data structure or something else. You would be surprised what things can be expressed in text. One very good example is the visualization of sound for testing a music player that Andrew Kelley once spoke about in a talk. He built a transcription for the amplitude and pitch of a song in ASCII to debug a tricky bug. He also built a tool for that waveform. This output could be used in an expect test and test sound output.

Snapshot Testing in the Wild

This section shows several examples for snapshot testing in different programming languages. Furthermore, I will assess the ease of implementation for these snapshot testing libraries.

Expect Test Ocaml

An example is shown in the following lines (copied from the GitHub repo of expect-test):

open! Core

let%expect_test "addition" =
  printf "%d" (1 + 2);
  [%expect {| 4 |}]
;;

These tests update by creating a corrected file which holds the corrected value of the test in this case | 3 |. This file diff can be accepted and overwrites the source code in the right position similar to the snapshot testing library of TigerBeatle. The JaneStreet expect test library uses the Ocaml preprocessor for the snapshot tests. I will not assess whether the expect test is hard or easy to implement because I have not tried it out myself (had no fitting project).

OhSnap

OhSnap is a snapshot library for Zig. It was extracted out of the TigerBeetle code base which has a snapshot testing library. With macros Zig is able to modify the source code of the test itself to mirror the snapshot. This brings new challenges but is also an improvement over the other snapshot testing libraries in the domain of locality. A snapshot test looks like this:

fn check_base(input: []const u8, want: OhSnap.Snap) !void {
    const ta = std.testing.allocator;
    const output = try base64Encode(input, ta);
    try want.expectEqual(output);
}

test "BASE64_1_snap" {
    try check_base("", oh.snap(@src(),
        \\[]u8
        \\  ""
    ));
}

The check_base function is the setup function to test several different input strings for the my own Base64 implementation. In the test block you can see the snapshot test. With the string \\<update> you can update a snapshot and it will be displayed in the code. The display of the result is done with a pretty printing library. OhSnap is a easy to use library for Zig and the test updating in the source file is just a great feature compared to the other snapshot testing libraries for simple snapshots.

PyTest-Snapshot

PyTest-Snapshot is an extension to the well known Python testing library pytest. It integrates seamlessly with pytest.

def test_limit(snapshot):
    snapshot.snapshot_dir = SNAPDIR
    url = (
        EarthquakeAPIURLBuilder()
        .time(starttime=TEST_START, endtime=TEST_END)
        .limit(limit=10)
        .finalize()
    )
    snapshot.assert_match(url, "test_url_builder_limit.txt")

As an argument to the test function provide a name canonically snapshot. If you want define the directory in which snapshots are stored. To test against a snapshot call the assert_match method on the provided snapshot argument with the value to test which has to be a string or a bytes object and optionally the name of the snapshot. You can update snapshots with the --update-snapshots flag after the pytest call. This snapshot testing library is really easy to integrate and provides all the necessary features you wish to have. That being said PyTest-Snapshot is still in beta phase (17.09.2024).

Insta

Insta is a Rust snapshot testing library. I used it to verify the output of a CLI application I wrote to generate QMK keyboard layout arrays out of normal characters in a text file. Insta is very comfortable to use. You add a test like every other test in Rust:

#[cfg(test)]
mod tests {
    use super::*;
    use insta::assert_snapshot;

    #[test]
    fn some_snap_test() {
        ...
        assert_snapshot!("snapshot id", "some expression to test")
    }
}

As one can see in the code snippet, it is stupidly easy to create a new snapshot test. You just have to import the assert_snapshot macro, optionally give the snapshot an ID, write an expression and voila it works! The tests fails first then you are able to update the snap with cargo insta review. All new snapshots are stored in the src/snapshot directory by default and can easily be pushed to version control. Bottom line Insta is a great snapshot library and I will use it whenever I have a Rust project that requires snapshot testing.

Jest

Snapshot testing is a built in feature of Jest. You mainly use it to fix UI that should not change any more or have a specific function. With Jest you can render React components for example and save the result in a file.

it("snapshot test test", () => {
  const { container } = render(<Home />);
  expect(container).toMatchSnapshot();
});

Use render to render a component and save the result in a constant and subsequently you can match against a snapshot. If you know JavaScript and Jest this framework for testing is great.

Conclusion

I love snapshot testing! It is a powerful tool and it enables you to develop good software comfortably and fast. I will use it more and more in the next time to assess the usefulness of snapshot testing even more. One point I will improve at is the feeling for when to use snapshot testing over normal assert tests.

Links

• Snapshot Testing for the Masses
• TigerBeetle microlibrary
• OhSnap
• Using ASCII waveforms to test hardware designs
• What if writing tests was a joyful experience?
• Insta
• PyTest-Snapshot
• Expect Test
• Jest