Protego parses directives set in robots.txt, including non-standard ones such as Request-Rate and Host. It also handles some typos in the directive names.
Installed Size: 43.0 kB Architectures: all