Comparing CrawlGuide's WWW::RobotRules port with the real module

DuckDuckGo says DuckDuckBot respects WWW::RobotRules, a Perl robots.txt parser. For DuckDuckBot, the CrawlGuide
tester also runs a port of it (robotRulesAllowed in readings.mjs), with the parts of Perl's URI module it uses. This
folder checks the port against the real module on random robots.txt files and URLs.

A program can give the module the file as bytes or as decoded text (LWP::RobotUA passes text), and Perl reads a few
characters differently in the two, such as a no-break space. So every case is checked both ways.

You need:
- Perl with WWW::RobotRules and URI. On Debian or Ubuntu: apt-get install libwww-robotrules-perl
- Node.js 22

Steps:
1. Save these files with the same paths:
   src/lib/robots.mjs    from https://crawlguide.com/repro/tester/src/lib/robots.mjs
   src/lib/crawlers.mjs  from https://crawlguide.com/repro/tester/src/lib/crawlers.mjs
   src/lib/readings.mjs  from https://crawlguide.com/repro/tester/src/lib/readings.mjs
   real.pl and fuzz.mjs  from this folder
2. Run: node fuzz.mjs src/lib/readings.mjs 20000 1
   The last two numbers are how many cases to make and the random seed. Every case in a run is distinct.

fuzz.mjs sends every case to real.pl, which asks the real module, and prints the first 10 cases in each mode where
the port disagrees, then the counts.

Our run on October 15, 2026 (output.txt): WWW::RobotRules 6.02, URI 5.27, Perl 5.38.2, Node 22.22.0. Seeds 1 to
5, 20,000 distinct cases each: 0 disagreements with the file as bytes or as text. WWW::RobotRules 6.02, 6.03 (the
latest on CPAN) and 6.04 (on GitHub) have the same parsing code.

The port follows URI 5.27. URI's change log lists nothing that affects it between 5.12 and 5.35. URI 5.36 and later
also normalize international host names (Unicode NFC) before encoding them, which the port doesn't.

Two cases the comparison doesn't cover. The port lowercases letters and finds port numbers with the JavaScript
engine's Unicode tables, and Perl 5.38 uses Unicode 15, so a letter added in Unicode 16 can read differently in a
host name, and a digit in a port. And LWP decodes the file by the charset the server names: a UTF-8 file served as
ISO-8859-1 reaches the module as Latin-1 text, where rules with non-ASCII characters stop matching.