qwarc

A framework for quick web archiving

Vēsture

JustAnotherArchivist 820384fe1e Stop deduping small responses For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.		pirms 4 gadiem
..
__init__.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	pirms 4 gadiem
aiohttp.py	Flush big responses to a temporary file instead of trying to keep everything in-memory	pirms 4 gadiem
cli.py	Support deep/multiple inheritance from Item	pirms 4 gadiem
const.py	Catch exceptions in Item.process and mark the items as errors instead of crashing	pirms 4 gadiem
utils.py	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	pirms 4 gadiem
version.py	Use setuptools_scm for versioning	pirms 4 gadiem
warc.py	Stop deduping small responses	pirms 4 gadiem