qwarc

Commit Graph

Аутор	SHA1	Порука	Датум
JustAnotherArchivist	b30ccf8bf8	Move response/exception history to ClientResponse.qhistory It is rarely necessary to access the history, and the tuple return value clutters the spec file code. As a consequence, it's no longer possible to return None if an error occurred without losing the history. To replace that, this also introduces a DummyClientResponse, which is kind of ClientResponse-like, has the same qhistory attribute, and evaluates to False when cast to bool (such that the intuitive `if response` works as expected).	пре 3 година
JustAnotherArchivist	e69527c715	Add defaultResponseHandler on the Item level	пре 3 година
JustAnotherArchivist	03336e4988	Add item to response handler arguments (e.g. for logging)	пре 3 година
JustAnotherArchivist	6bdcfe71f0	Refactor database creation and item generation: call `Item.generate()` on every qwarc run and dedupe its output, allowing the addition of further items by modifying the spec file	пре 3 година
JustAnotherArchivist	c878241f24	Switch from concurrent.futures.CancelledError to asyncio.CancelledError Since Python 3.8, the latter does not inherit from the former anymore.	пре 3 година
JustAnotherArchivist	749158b97a	Use the Future's result directly rather than awaiting again The asyncio documentation does not specify whether awaiting a Future multiple times is supported or not: https://bugs.python.org/issue41275	пре 3 година
JustAnotherArchivist	a85e80ffa2	Configurable request timeout	пре 3 година
JustAnotherArchivist	429ac94689	Make it possible to override and remove headers	пре 3 година
JustAnotherArchivist	e40be54578	Document verify_ssl parameter	пре 3 година
JustAnotherArchivist	d3437bde19	Move default headers to qwarc.const	пре 3 година
JustAnotherArchivist	1678075a89	Log traceback on exceptions raised from an item	пре 4 година
JustAnotherArchivist	b1a1c03f7e	Handle STOP file and high memory usage before full disk to allow stopping while the disk is above the limit	пре 4 година
JustAnotherArchivist	dd44d9b174	Adjust logging levels: log individual request failures only at WARNING and cancelled tasks at ERROR level	пре 4 година
JustAnotherArchivist	91035d769c	Catch exceptions in Item.process and mark the items as errors instead of crashing	пре 4 година
JustAnotherArchivist	69984765b3	Fix taskType typo silencing cancellation warnings	пре 4 година
JustAnotherArchivist	c263ad0b03	Return ClientResponse object from fetch only if the retrieval was successful If an exception was raised and caught, the object is still present in the history.	пре 4 година
JustAnotherArchivist	cb0d11284e	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	пре 4 година
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	пре 4 година
JustAnotherArchivist	08d96b37c5	Support deep/multiple inheritance from Item	пре 4 година
JustAnotherArchivist	9d8de13775	Add Item.flush_subitems to flush the new subitems to the database while the item is still being processed This also renames add_item to add_subitem for clarity.	пре 4 година
JustAnotherArchivist	50b936b18c	Refactor QWARC class to keep relevant variables in instance attributes instead of local variables	пре 4 година
JustAnotherArchivist	c5d8d93166	Remove stray whitespace	пре 4 година
JustAnotherArchivist	7d53577522	Add parameter for disabling SSL/TLS certificate validation	пре 4 година
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	пре 4 година
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	пре 4 година
JustAnotherArchivist	d751844626	Fix starting another item before stopping on STOP file or memory limit exceedance	пре 4 година
JustAnotherArchivist	2b0778f9b5	Remove leftovers from initial code rewrite	пре 4 година
JustAnotherArchivist	ab22966fef	Add to log which item a message is coming from	пре 4 година
JustAnotherArchivist	6fafd32685	Error when the retries are exceeded	пре 4 година
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	пре 4 година
JustAnotherArchivist	5008e6e8cd	Deduplicate items	пре 4 година
JustAnotherArchivist	46c95e2157	Disable decoding the response content chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.	пре 4 година
JustAnotherArchivist	ad22a2327a	Support adding headers to individual requests	пре 5 година
JustAnotherArchivist	67076f964c	Add support for POST requests	пре 5 година
JustAnotherArchivist	c1574a06c9	Fix sleep task type	пре 5 година
JustAnotherArchivist	e0ca88c807	Fix reference to get_rss	пре 5 година
JustAnotherArchivist	8a8935810d	Fix references to memory and disk space check methods	пре 5 година
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	пре 5 година
JustAnotherArchivist	e892a6b6a7	Initial commit	пре 5 година

39 Комити (b30ccf8bf80df691c3113658fd1685b12e9217cc)