qwarc

Gráfico de cometimentos

Autor	SHA1	Mensagem	Data
JustAnotherArchivist	168fa78736	Avoid locking the DB when there are no subitems to insert	há 4 anos
JustAnotherArchivist	4484d6c588	Add Item representation	há 4 anos
JustAnotherArchivist	5675118877	Rename id to id_ to avoid clash with builtin	há 4 anos
JustAnotherArchivist	a1e693739e	Replace DB locking with an async context manager	há 4 anos
JustAnotherArchivist	15203bd991	Handle redirect traps/loops	há 4 anos
JustAnotherArchivist	f8f5258197	Track redirect depth	há 4 anos
JustAnotherArchivist	a3d6fb35f8	Turn response handlers into kwarg-only functions for easier extendability without breaking existing code	há 4 anos
JustAnotherArchivist	6cc4adb901	Remove stray TODO The DB creation operates with a DB lock, so that code can't run while another process is filling the DB; it would block on obtaining the lock a few lines prior instead.	há 4 anos
JustAnotherArchivist	c5604ef965	Simplify header merging	há 4 anos
JustAnotherArchivist	59ae1183d2	Add fromResponse parameter for URL completion and automatic Referer header	há 4 anos
JustAnotherArchivist	2324216016	Add baseUrl and evaluate incomplete URLs relative to it	há 4 anos
JustAnotherArchivist	b30ccf8bf8	Move response/exception history to ClientResponse.qhistory It is rarely necessary to access the history, and the tuple return value clutters the spec file code. As a consequence, it's no longer possible to return None if an error occurred without losing the history. To replace that, this also introduces a DummyClientResponse, which is kind of ClientResponse-like, has the same qhistory attribute, and evaluates to False when cast to bool (such that the intuitive `if response` works as expected).	há 4 anos
JustAnotherArchivist	e69527c715	Add defaultResponseHandler on the Item level	há 4 anos
JustAnotherArchivist	03336e4988	Add item to response handler arguments (e.g. for logging)	há 4 anos
JustAnotherArchivist	6bdcfe71f0	Refactor database creation and item generation: call `Item.generate()` on every qwarc run and dedupe its output, allowing the addition of further items by modifying the spec file	há 4 anos
JustAnotherArchivist	c878241f24	Switch from concurrent.futures.CancelledError to asyncio.CancelledError Since Python 3.8, the latter does not inherit from the former anymore.	há 4 anos
JustAnotherArchivist	749158b97a	Use the Future's result directly rather than awaiting again The asyncio documentation does not specify whether awaiting a Future multiple times is supported or not: https://bugs.python.org/issue41275	há 4 anos
JustAnotherArchivist	a85e80ffa2	Configurable request timeout	há 4 anos
JustAnotherArchivist	429ac94689	Make it possible to override and remove headers	há 4 anos
JustAnotherArchivist	e40be54578	Document verify_ssl parameter	há 4 anos
JustAnotherArchivist	d3437bde19	Move default headers to qwarc.const	há 4 anos
JustAnotherArchivist	1678075a89	Log traceback on exceptions raised from an item	há 4 anos
JustAnotherArchivist	b1a1c03f7e	Handle STOP file and high memory usage before full disk to allow stopping while the disk is above the limit	há 4 anos
JustAnotherArchivist	dd44d9b174	Adjust logging levels: log individual request failures only at WARNING and cancelled tasks at ERROR level	há 4 anos
JustAnotherArchivist	91035d769c	Catch exceptions in Item.process and mark the items as errors instead of crashing	há 4 anos
JustAnotherArchivist	69984765b3	Fix taskType typo silencing cancellation warnings	há 4 anos
JustAnotherArchivist	c263ad0b03	Return ClientResponse object from fetch only if the retrieval was successful If an exception was raised and caught, the object is still present in the history.	há 4 anos
JustAnotherArchivist	cb0d11284e	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	há 4 anos
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	há 4 anos
JustAnotherArchivist	08d96b37c5	Support deep/multiple inheritance from Item	há 4 anos
JustAnotherArchivist	9d8de13775	Add Item.flush_subitems to flush the new subitems to the database while the item is still being processed This also renames add_item to add_subitem for clarity.	há 4 anos
JustAnotherArchivist	50b936b18c	Refactor QWARC class to keep relevant variables in instance attributes instead of local variables	há 4 anos
JustAnotherArchivist	c5d8d93166	Remove stray whitespace	há 4 anos
JustAnotherArchivist	7d53577522	Add parameter for disabling SSL/TLS certificate validation	há 4 anos
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	há 4 anos
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	há 4 anos
JustAnotherArchivist	d751844626	Fix starting another item before stopping on STOP file or memory limit exceedance	há 4 anos
JustAnotherArchivist	2b0778f9b5	Remove leftovers from initial code rewrite	há 4 anos
JustAnotherArchivist	ab22966fef	Add to log which item a message is coming from	há 4 anos
JustAnotherArchivist	6fafd32685	Error when the retries are exceeded	há 4 anos
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	há 4 anos
JustAnotherArchivist	5008e6e8cd	Deduplicate items	há 4 anos
JustAnotherArchivist	46c95e2157	Disable decoding the response content chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.	há 4 anos
JustAnotherArchivist	ad22a2327a	Support adding headers to individual requests	há 5 anos
JustAnotherArchivist	67076f964c	Add support for POST requests	há 5 anos
JustAnotherArchivist	c1574a06c9	Fix sleep task type	há 5 anos
JustAnotherArchivist	e0ca88c807	Fix reference to get_rss	há 5 anos
JustAnotherArchivist	8a8935810d	Fix references to memory and disk space check methods	há 5 anos
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	há 5 anos
JustAnotherArchivist	e892a6b6a7	Initial commit	há 5 anos

50 Cometimentos (168fa787366cf31c6ea5955bc6f2bc9b0e6f2bb0)