qwarc

Commit graph

Autor	SHA1	Nachricht	Datum
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	vor 4 Jahren
JustAnotherArchivist	08d96b37c5	Support deep/multiple inheritance from Item	vor 4 Jahren
JustAnotherArchivist	9d8de13775	Add Item.flush_subitems to flush the new subitems to the database while the item is still being processed This also renames add_item to add_subitem for clarity.	vor 4 Jahren
JustAnotherArchivist	50b936b18c	Refactor QWARC class to keep relevant variables in instance attributes instead of local variables	vor 4 Jahren
JustAnotherArchivist	c5d8d93166	Remove stray whitespace	vor 4 Jahren
JustAnotherArchivist	7d53577522	Add parameter for disabling SSL/TLS certificate validation	vor 4 Jahren
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	vor 4 Jahren
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	vor 4 Jahren
JustAnotherArchivist	d751844626	Fix starting another item before stopping on STOP file or memory limit exceedance	vor 4 Jahren
JustAnotherArchivist	2b0778f9b5	Remove leftovers from initial code rewrite	vor 4 Jahren
JustAnotherArchivist	ab22966fef	Add to log which item a message is coming from	vor 4 Jahren
JustAnotherArchivist	6fafd32685	Error when the retries are exceeded	vor 4 Jahren
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	vor 4 Jahren
JustAnotherArchivist	5008e6e8cd	Deduplicate items	vor 4 Jahren
JustAnotherArchivist	46c95e2157	Disable decoding the response content chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.	vor 4 Jahren
JustAnotherArchivist	ad22a2327a	Support adding headers to individual requests	vor 5 Jahren
JustAnotherArchivist	67076f964c	Add support for POST requests	vor 5 Jahren
JustAnotherArchivist	c1574a06c9	Fix sleep task type	vor 5 Jahren
JustAnotherArchivist	e0ca88c807	Fix reference to get_rss	vor 5 Jahren
JustAnotherArchivist	8a8935810d	Fix references to memory and disk space check methods	vor 5 Jahren
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	vor 5 Jahren
JustAnotherArchivist	e892a6b6a7	Initial commit	vor 5 Jahren

22 Commits (1214409a0bad2dced4ae253bf0e28a70e78cb161)