How Bevy Catches Rendering Regressions Across Platforms

How do you test that a game engine still renders correctly? You can unit test the ECS, the math, the asset loader. But "the frame looks right" is not something a unit test can assert: the render world is too complex, and hand-written assertions against draw commands are brittle and unmaintainable.

Bevy, the popular open-source Rust game engine, answers this with snapshot testing at scale: render every example on every supported platform, screenshot a deterministic frame, and compare it against the previous run with PixelEagle. This is the production deployment that PixelEagle was originally built for, and it shows what frame-level visual regression testing involves on genuinely difficult content.

The full story was presented by Bevy maintainer François Mockers at Bevy Meetup #4:

Step 1: Make the Frames Deterministic

You can't compare two screenshots of a game unless both runs rendered the same thing. Bevy solves this with a bevy_ci_testing feature that turns any example into a reproducible test:

• A fixed timestep, so frame N is always the same simulation moment
• Seeded randomness, so particles and procedural content are identical between runs
• Frame-keyed events: "take a screenshot at frame 200, exit at frame 900"

This is the game-engine version of freezing animations and mocking the clock in a web test, and it captures the central idea of the whole system: visual regression testing for rendering is mostly a determinism problem. The examples already exist for documentation purposes; the determinism harness turns 240+ of them into test fixtures for free.

The harness ships as a Bevy feature rather than internal CI tooling, so game developers building on Bevy can use the same mechanism for their own visual tests.

Step 2: Render on Every Platform (However You Can)

Cross-platform GPU rendering in CI is held together with pragmatism:

• Linux: llvmpipe (software rendering) with xvfb for a virtual display
• macOS: Apple Silicon GitHub runners
• WebGL2 / WebGPU: real browsers driven by Playwright, which also takes the screenshots
• iOS and Android: real physical devices on BrowserStack, driven by Appium

Each platform captures screenshots with whatever mechanism works there: Bevy's own screenshot facility natively, Playwright in browsers, Appium on phones. This heterogeneity is exactly why PixelEagle is capture-agnostic: the platform doesn't matter once the PNGs are uploaded as a run tagged with its metadata.

The device dimension matters more than you'd expect: GPU feature support varies per phone (MSAA, shadows), and at least one specific device revealed a GPU preprocessing bug that no emulator would have caught.

Step 3: Compare Cheaply, Review Only What Changed

A full run produces hundreds of screenshots per platform, and the overwhelming majority are identical to the previous run. PixelEagle's comparison strategy is built around that distribution:

1. Every screenshot is content-hashed first, and identical images match for free.
2. Only screenshots whose hashes differ get an actual image comparison.
3. The review UI surfaces only changed screenshots; with 240+ examples times N platforms, showing everything would bury the signal.

This last point is easy to underestimate. The talk's implicit thesis is that reviewer throughput, not comparison precision, is what caps how much visual coverage a team can sustain. The report exists so that a human looks at ten diffs instead of a thousand screenshots.

Exact pixel comparison has a known limit here: Bevy's stochastic examples (auto-exposure, SSAO, transmission) differ at the pixel level on every run even when nothing regressed. That noise is precisely what perceptual comparison with NVIDIA ꟻLIP is for, and why PixelEagle offers it alongside exact matching.

What It Catches

Real regressions found by this system include an emissive material exposure regression (lights rendering too bright), broken UI rendering, and shader breakage: the kind of bug that doesn't crash anything, survives code review, and would otherwise be discovered by a player weeks later. And because most diffs turn out to be intended changes, the maintenance load stays low: review is a quick confirm, not an investigation.

The Economics Are Part of the Design

The talk is refreshingly honest about cost. A full cross-platform run takes hours of CI time; running on every PR of a large open-source project would cost thousands of dollars a month. So the system runs on a schedule against main, within what GitHub's OSS tier and BrowserStack's open-source sponsorship make free.

This constraint shaped PixelEagle's pricing model directly: hash comparisons are free, deeper comparisons spend tokens only on screenshots that actually differ, and diff results are cached by content so nothing is computed twice. When most of your screenshots don't change between runs (the normal case), cost tracks real visual change rather than test frequency.

What This Means for Your Project

You probably don't need four platforms and physical phones. But the pattern scales down cleanly:

1. Make frames deterministic: fixed timestep and seeded RNG in an engine; frozen animations and mocked clocks on the web
2. Capture wherever your product runs, with whatever tool works there
3. Tag runs with metadata (platform, branch) so baselines match like with like
4. Let hashing eat the unchanged majority, and review only real diffs
5. Use perceptual comparison when your renderer has legitimate pixel-level variance

If a game engine can keep hundreds of rendered scenes under regression control across five platforms, a chart-heavy dashboard or a canvas editor is well within reach. Getting started takes a project, a token, and a CI step.

Key Takeaways

• Bevy snapshot-tests 240+ rendered examples across Linux, macOS, browsers, and real mobile devices with PixelEagle
• Determinism (fixed timestep, seeded RNG, frame-keyed screenshots) is the prerequisite that makes frame comparison possible
• Hash-first comparison makes the unchanged majority free; only real differences cost anything
• Reviewer throughput is the real bottleneck; surfacing only changed screenshots is what keeps the system usable
• The same pattern applies to any graphically rich application, at a fraction of the scale