qwarc

Commit Graph

Автор	SHA1	Съобщение	Дата
JustAnotherArchivist	f025c4e9f3	Add extensive debug logging	преди 3 години
JustAnotherArchivist	dbe1ed71ab	"Freeze" log file object before writing to WARC to ensure that further log messages aren't picked up This is a workaround for https://github.com/webrecorder/warcio/issues/90	преди 3 години
JustAnotherArchivist	a4cf1a4225	Fix str_get_all_between yielding half-overlapping matches	преди 3 години
JustAnotherArchivist	15203bd991	Handle redirect traps/loops	преди 3 години
JustAnotherArchivist	f8f5258197	Track redirect depth	преди 3 години
JustAnotherArchivist	a3d6fb35f8	Turn response handlers into kwarg-only functions for easier extendability without breaking existing code	преди 3 години
JustAnotherArchivist	a91cc23d47	Simplify get_software_info's signature to just the extra dependency packages As a consequence, SpecDependencies.extra can now be any data type that can be put into JSON; unhashable types previously caused a crash due to the lru_cache.	преди 3 години
JustAnotherArchivist	b30ccf8bf8	Move response/exception history to ClientResponse.qhistory It is rarely necessary to access the history, and the tuple return value clutters the spec file code. As a consequence, it's no longer possible to return None if an error occurred without losing the history. To replace that, this also introduces a DummyClientResponse, which is kind of ClientResponse-like, has the same qhistory attribute, and evaluates to False when cast to bool (such that the intuitive `if response` works as expected).	преди 3 години
JustAnotherArchivist	03336e4988	Add item to response handler arguments (e.g. for logging)	преди 3 години
JustAnotherArchivist	cb0d11284e	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	преди 4 години
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	преди 4 години
JustAnotherArchivist	37dbcfad21	Don't write responses to WARC that triggered an exception For example, if the connection breaks while retrieving a response but after the headers have been parsed, the response body would be incomplete.	преди 4 години
JustAnotherArchivist	f038cf91db	Fix unfound distribution handling	преди 4 години
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	преди 4 години
JustAnotherArchivist	e99e2304c9	Write meta WARC with log file	преди 4 години
JustAnotherArchivist	85d78cee13	Add warcinfo record with version information on Python, system, and dependencies	преди 4 години
JustAnotherArchivist	6fafd32685	Error when the retries are exceeded	преди 4 години
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	преди 4 години
JustAnotherArchivist	85f6f7bd82	Make qwarc.utils.handle_response_limit_error_retries more useful by passing the deferring handler as an argument	преди 5 години
JustAnotherArchivist	2d52e78d85	Fix reference to aiohttp.CientError	преди 5 години
JustAnotherArchivist	e892a6b6a7	Initial commit	преди 5 години

21 Ревизии (f025c4e9f36a661c44773fc60bba32f8ec1300e2)