qwarc

A framework for quick web archiving

История

JustAnotherArchivist 820384fe1e Stop deduping small responses For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.		преди 4 години
..
__init__.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	преди 4 години
aiohttp.py	Flush big responses to a temporary file instead of trying to keep everything in-memory	преди 4 години
cli.py	Support deep/multiple inheritance from Item	преди 4 години
const.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	преди 4 години
utils.py	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	преди 4 години
version.py	Use setuptools_scm for versioning	преди 4 години
warc.py	Stop deduping small responses	преди 4 години