qwarc

A framework for quick web archiving

文件历史

JustAnotherArchivist 820384fe1e Stop deduping small responses For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.		4 年前
..
__init__.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	4 年前
aiohttp.py	Flush big responses to a temporary file instead of trying to keep everything in-memory	4 年前
cli.py	Support deep/multiple inheritance from Item	4 年前
const.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	4 年前
utils.py	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	4 年前
version.py	Use setuptools_scm for versioning	4 年前
warc.py	Stop deduping small responses	4 年前