Proving a refactor changed nothing: geometry diffing a 20-tile collage
When a refactor's requirement is "look exactly the same," screenshot diffs waste time on noise. For a twenty-tile homepage collage moved from hardcoded JSX into CMS documents, we dumped each tile's classes, bounding box, link, and sibling index before and after — then treated an empty JSON diff as proof. The snapshot is scaffolding: use it, delete it.
You cannot eyeball twenty tiles
Screenshot comparison is the obvious answer and a bad fit here. Pixel diffing catches a four-pixel shift, and it also catches font rendering differences between two runs, an image that loaded at a different moment, and an animation captured mid-frame. You spend more time triaging false positives than you would have spent just looking.
It also tells you that something is different without telling you what. A red smear in the corner of a diff image is the beginning of an investigation, not the end of one.
What we actually cared about was structural. Does each tile have the same classes, occupy the same box, and sit in the same position in DOM order? That is a much narrower question than are these two images identical, and it has a much cleaner answer.
Carry the CSS as data
The collage's layout lived in class names — a set of hand-written positioning and sizing classes applied per tile, tuned by hand over a long time. There was no rule generating them. Tile seven looked the way it did because somebody looked at it and made a decision.
The temptation in a migration like this is to rationalize: derive the classes from a size field, invent a small layout DSL, replace the bespoke geometry with something systematic. That is a redesign wearing a refactor's clothes, and it is how you end up shipping a homepage nobody approved.
So we treated the classes as opaque data. Each project document carries its legacy class string. The component applies it. The CMS does not know what the string means and does not need to. It is not the layout system we would design from scratch, and it is exactly what the requirement — render identically — implies.
- 01Dump beforeClass, geometry and child order for every tile
- 02RefactorHardcoded markup becomes data-driven
- 03Dump afterThe same script, same widths
- 04DiffAny delta is a regression, named
- 05Delete the snapshotIt proved one change; it isn't a spec
Keeping the snapshot as a CI test would fail on the next intentional design change, and train everyone to approve the diff unread.
Dump, refactor, dump, diff
The verification is a snippet you run in the console. For every tile: the full class list, the bounding box from getBoundingClientRect rounded to whole pixels, the tag name, the link target, and the index among its siblings. Serialize to JSON and save it.
Run it before the refactor. Do the work. Run it again against the new implementation at the same viewport width with the same content. Diff the two files. An empty diff is the proof.
Two practical notes. Round the geometry, because subpixel values differ between runs for reasons that do not matter and an unrounded diff is all noise. And capture at more than one width, because a layout that matches at 1440 and diverges at 768 is a real regression a single snapshot will happily miss.
What it actually caught
The diff was not empty on the first run, which is the entire point. Ordering was the first problem. The hardcoded version had an implicit order from the source file, and the CMS query returned documents in a different one. That is invisible in a screenshot of a collage where every tile is absolutely positioned — and very visible in a diff of sibling indices. It mattered for tab order and for screen readers.
The second was a tile whose class string had been transcribed with a modifier missing. One tile, slightly different height, in a collage where tiles are deliberately different heights. Nobody would have caught that by looking at it.
Both took minutes to fix once the diff pointed straight at them. Both would have shipped otherwise, and the second is the kind of thing that resurfaces three months later as a vague comment that the homepage looks a bit off.
When it is worth the hour
This technique earns its keep when the requirement is genuinely no visible change and the surface is large enough that manual checking is unreliable. Twenty tiles qualifies. A three-card row does not. Just look at it.
It is also the right tool when the thing you are changing is the data source rather than the design. Swapping where markup comes from while keeping the markup identical is a common and underrated kind of refactor, and it is one of the few where you can state the success criterion precisely enough to test it mechanically.
The broader habit: before a refactor whose goal is no change, write down what no change means in terms a script can check. Sometimes that is a snapshot of geometry. Sometimes it is a list of routes and their status codes, or a dump of every generated URL, or a hash of the rendered text content. The specific form matters less than having one, because refactors that were supposed to change nothing are exactly the refactors nobody reviews carefully.
Throw the snapshot away
One decision people get wrong at the end: what to do with the before-and-after files. The instinct is to commit them and turn the geometry dump into a regression test that runs in CI forever. Resist it.
The snapshot is a proof about one change, not a specification of the design. Commit it and the very next intentional layout change fails the test — correctly, in the sense that the geometry did change, and uselessly, because that was the entire point of the work. What follows is a few months of updating the snapshot as a reflex, which trains everyone to accept the diff without reading it. A test people update without looking at is worse than no test, because it carries authority it has stopped earning.
So treat the snapshot as scaffolding. Use it, confirm the diff is empty, delete it. What is worth keeping is the script that generates it, which costs nothing to hold on to and will be useful the next time somebody has to prove a change was invisible. The proof is disposable. The instrument is not.
Questions
- Why not use visual screenshot comparison for a layout migration?
- Pixel diffs catch font rendering, image load timing, and mid-animation frames as false positives, and they do not name what changed. Structural dumps — classes, rounded getBoundingClientRect boxes, DOM order — answer the narrower question a CMS migration actually asks.
- What regressions did geometry dumping catch?
- Document query order differed from the hardcoded sibling order (invisible in absolute positioning, bad for tab order and screen readers), and one tile was missing a class modifier that changed its height. Both would have been easy to miss by eye.
- Should the before/after geometry snapshot live in CI forever?
- No. It proves one migration; the next intentional layout change fails it usefully once, then trains people to update snapshots without reading. Keep the generator script; throw away the files after the empty diff.