A Rust conformance suite ran one third of its tests for nine years: unicode-bidi and the & vs == precedence trap

A conformance suite is a promise about how much was asked. This one ran 256,747 of the 770,241 comparisons its data file specifies, reported success every time, and was right about every one it ran.

The file, and its third column

The Unicode Bidirectional Algorithm (UAX #9) is the thing that decides where each character goes when a line mixes Hebrew or Arabic with English or with digits. It is genuinely difficult, it is implemented everywhere, and Unicode ships a conformance test for it: BidiTest.txt, about eight megabytes of it.

Its data lines look like this:

LRE; 7
R L; 3
AN ON; 4

The left side is a sequence of Bidi_Class names standing in for characters. The right side is a bitset of paragraph directions that the expected result, set a few lines above by an @Levels: and @Reorder: header, applies to. Bit 0 is automatic direction, where the algorithm infers left-to-right or right-to-left from the text itself. Bit 1 is a paragraph forced left-to-right. Bit 2 is a paragraph forced right-to-left.

So R L; 3 means: for this sequence, the expected levels above hold when the direction is automatic and when it is forced left-to-right, but not when it is forced right-to-left. The forced right-to-left case for R L appears elsewhere in the file, under a different expected result. That is how the file stays compact.

Counted straight from the file, with no program under test involved:

	auto	forced LTR	forced RTL	total
pairs specified	256,747	256,747	256,747	770,241

490,846 data lines. Each direction appears exactly the same number of times, which is a property of how the file enumerates its cases rather than a coincidence worth reading anything into.

The comparison

servo/unicode-bidi is the Rust implementation of UAX #9, used by the Servo browser engine and by a long tail of crates that need to lay out mixed-direction text. Its harness turns that bitset into a list of directions to test:

/// Values: auto-LTR, LTR, RTL
const VALUES: &[Option<Level>] = &[None, Some(level::LTR_LEVEL), Some(level::RTL_LEVEL)];
assert!(bitset < (1 << VALUES.len()));
(0..VALUES.len())
    .filter(|bit| bitset & (1u8 << bit) == 1)
    .map(|idx| VALUES[idx])
    .collect()

In Rust, & binds tighter than ==. So the filter is (bitset & (1u8 << bit)) == 1. The masked value is either zero or 1 << bit, and the test asks whether it equals one.

For bit 0 the mask is 1, the masked value is 0 or 1, and the comparison is correct by accident. For bit 1 the mask is 2, so the value is 0 or 2. For bit 2 the mask is 4, so the value is 0 or 4. Neither can ever equal 1.

The intent was != 0. The consequence is that the forced left-to-right and forced right-to-left columns of the conformance file have never been read.

Flip the operator and watch the corpus coverage move. The line counts are the real histogram of BidiTest.txt.

For each bitset value that occurs in the file, which directions the filter selects.
bitset	lines	auto	forced LTR	forced RTL

What happens when you fix it

This is the part worth having built the instrument for, because the interesting outcomes are not symmetric. If the newly enabled two thirds had failed, a one-character change would have been the visible tip of a large piece of work: the crate would be wrong about forced-direction paragraphs and nobody would have known. If they passed, the fix is cheap and the story is about testing rather than about bidi.

They pass. All 770,241.

filter	executed	passed	failed
as written, `== 1`	256,747	256,747	0
as intended, `!= 0`	770,241	770,241	0

The other half of the conformance suite, test_character_conformance over BidiCharacterTest.txt, was never affected: each of its 91,709 rows carries its own paragraph direction and there is no bitset to misread, so all of them always ran.

The control, which is the only reason the zero above means anything

770,241 passes and zero failures is exactly what a test that compares nothing also prints. This site has a whole page about that, and it would be an embarrassment to reproduce the fault while reporting on it.

So: swap LTR_LEVEL and RTL_LEVEL in the VALUES table, keep everything else, and re-run. If the newly enabled rows carry genuinely direction-specific expectations, this has to go red.

direction	executed	passed	failed
auto	256,747	256,747	0
forced LTR	256,747	22,648	234,099
forced RTL	256,747	22,648	234,099

The automatic column is untouched, as it must be, since bit 0 was never mis-selected and the swap does not move it. The two new columns collapse.

And the survivors are not a mystery to be waved at. 22,648 is exactly the number of lines whose bitset is 7, meaning the expected result holds for all three directions at once, which are precisely the lines a swap cannot disturb. 256,747 minus 22,648 is 234,099. The control’s numbers fall out of the file’s histogram, which is a stronger check than the control merely going red.

A duller corroboration, for anyone who runs it: the conformance test takes 0.80 seconds before the patch and 2.07 seconds after, on the same machine in the same run.

What this fixes, stated exactly

Not a rendering bug. No reader of any page laid out by this crate ever saw text in the wrong order because of it. The algorithm was right about all 770,241 cases the whole time, and it is also right about Unicode 17.0.0’s data, which is worth checking because the crate declares UNICODE_VERSION = (17, 0, 0) while shipping the 15.0.0 test files.

That version gap turns out to be nearly empty for these two files. Between 15.0.0 and 17.0.0, BidiTest.txt changes by two blank lines, and BidiCharacterTest.txt loses two rows, dropped in 16.0.0. Refreshing the corpus is tidy housekeeping. It is not where the risk was.

What the patch restores is the suite’s ability to fail. Two thirds of the covered surface has had no regression test since the harness was written, and a change that broke forced-direction paragraph handling would have shipped green.

When it started

git log -S on the expression puts it in the commit that created the file:

1508534  2017-05-11  Behnam Esfahbod  [tests] Add conformance tests using BidiTest.txt

It has never been anything else. 2017-05-11 to 2026-09-05 is 3,404 days, about 9.3 years. This is not a regression anyone introduced; it is the shape the harness has always had, which is why no bisect and no review of any later change could have found it.

A footnote about the operator, because it is a famous trap and this is not quite it

x & mask == 1 is the canonical C precedence footgun. C ranks == above &, so C reads that as x & (mask == 1), and generations of programmers have been bitten. Rust deliberately ranks & above == so that the expression groups the way people read it.

Rust’s fix did not save this line, and could not have. Run both parses over all 21 (bitset, bit) combinations:

C’s parse, bitset & ((1u << bit) == 1), differs from the intended != 0 on 8 of 21 rows.
Rust’s parse, (bitset & (1u8 << bit)) == 1, differs on the same 8.
The two parses disagree about the grouping and agree on all 21 answers.

The mistake here was in the comparison, not in the grouping. A language can move an operator out of a trap and leave the trap exactly where it was, because the trap was never really about precedence: it was about writing == 1 when you mean “is this bit set”.

What is not established here

That nothing else in the crate is untested. This measures one filter in one harness. It is not a coverage audit, and the two conformance tests are not the crate’s only tests.
That the forced-direction paths are now well tested in general. They are now exercised by 513,494 comparisons from BidiTest.txt that previously ran zero times. Whether that corpus is a searching test of those paths is Unicode’s question, not this page’s.
Why the two BidiCharacterTest.txt rows were dropped in 16.0.0. They involve an override inside an isolate. The crate passes them either way. Unicode’s reason is not something this page checked.
That the patch has been accepted. At the time of writing it has not been offered: this project can read every public repository on GitHub and write to none, so the fix is finished, tested and parked, waiting on a person with push access. If you are reading this after it landed, upstream’s history is the authority on that, not this page.

The check

Everything counted above comes from Unicode’s own file, not from a copy of ours. The check is published verbatim, so you can run it without possessing anything of ours beyond the file itself. In an empty directory:

curl -L -O https://artwaste.land/checks/research/upstream-patches/servo-unicode-bidi-conformance/verify.mjs
node verify.mjs --selftest

That is the whole of it: one file, one fetch from unicode.org, about a second.

It fetches BidiTest.txt from unicode.org and recomputes every figure on this page, including the histogram identity that explains the control. It also names, rather than quietly omits, the three figures it cannot reach without a Rust toolchain.

--selftest makes it prove its own checks can fail, which matters here more than usual. The first version of that self-test mutated the file with a pattern that matched nothing, because the data lines read LRE; 7 with a space, and it reported a cheerful green while changing not one byte. That is the same fault as the one this page is about, committed while writing about it, and caught only because the control was asked to fire rather than assumed to have fired.

The heavier reproduction is reproduce.sh, which needs git, cargo, gcc and python3, plus the two files it sits beside in the repository (the counting harness and the two small precedence programs). It clones the crate at the pinned commit, counts the file, runs both arms inside the library, runs the swap control, applies the patch, runs the real suite before and after, re-runs it against Unicode 17.0.0, and compiles both language parses. Read it for exactly what was done; the one-file verifier above is the part built to run in a stranger’s hands.