qwarc

Gráfico de cometimentos

Autor	SHA1	Mensagem	Data
JustAnotherArchivist	f025c4e9f3	Add extensive debug logging	há 3 anos
JustAnotherArchivist	dbe1ed71ab	"Freeze" log file object before writing to WARC to ensure that further log messages aren't picked up This is a workaround for https://github.com/webrecorder/warcio/issues/90	há 3 anos
JustAnotherArchivist	8ca2a6bde5	Fix exceptions on journal errors	há 3 anos
JustAnotherArchivist	733506aed7	Remove obsolete TODO The fnctl.flock call does not use LOCK_NB, so if it fails with an OSError, that is Really Bad™ and can't be handled cleanly.	há 3 anos
JustAnotherArchivist	c7fac0ec3f	Add WARC journalling with rollback on errors Inspired by the implementation in wpull but structured differently to avoid reopening the journal file constantly.	há 3 anos
JustAnotherArchivist	a91cc23d47	Simplify get_software_info's signature to just the extra dependency packages As a consequence, SpecDependencies.extra can now be any data type that can be put into JSON; unhashable types previously caused a crash due to the lru_cache.	há 3 anos
JustAnotherArchivist	820384fe1e	Stop deduping small responses For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.	há 4 anos
JustAnotherArchivist	461cedbbde	Avoid temporary files created by warcio due to not knowing the record payload length	há 4 anos
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	há 4 anos
JustAnotherArchivist	93df9cd18d	Get rid of the temporary extra log file and read the plain file instead	há 4 anos
JustAnotherArchivist	08c3d55376	Add comment on block digest workaround (cf. `f14a664b`)	há 4 anos
JustAnotherArchivist	413435b7fb	Work around warcio not writing the correct WARC-Profile header for revisit records on WARC/1.1 https://github.com/webrecorder/warcio/issues/94	há 4 anos
JustAnotherArchivist	8ee9b20718	Remove WARC-Target-URI header from warcinfo record WARC 1.1 specification, section 5.14: "A ‘warcinfo’ record shall not have a WARC-Target-URI field."	há 4 anos
JustAnotherArchivist	f14a664b1c	Work around warcio not writing a block digest for warcinfo records (https://github.com/webrecorder/warcio/issues/87 ) The length has to be set manually because otherwise warcio will automatically remove the header again.	há 4 anos
JustAnotherArchivist	bd14ab3901	Fix crash due to closing the log handler on reaching the max WARC size	há 4 anos
JustAnotherArchivist	08117630b0	Remove warcinfo record in each data WARC and refer to the process's warcinfo record in the meta WARC instead	há 4 anos
JustAnotherArchivist	26aab15605	urn:X-qwarc instead of urn:qwarc	há 4 anos
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	há 4 anos
JustAnotherArchivist	e093211496	Set content type for resource records	há 4 anos
JustAnotherArchivist	ae46b53401	Always write a WARC-Warcinfo-ID header	há 4 anos
JustAnotherArchivist	23fcdd4026	Write microsecond dates for request and response records	há 4 anos
JustAnotherArchivist	3030ad10ab	Mark private API accordingly	há 4 anos
JustAnotherArchivist	e0b4104d21	Remove log handler before writing log record since that requires closing the stream	há 4 anos
JustAnotherArchivist	6cfd352f68	Write WARC/1.1 files	há 4 anos
JustAnotherArchivist	e1ad5c232e	Write warcinfo and resource records in meta WARC on firing up qwarc rather than at the end	há 4 anos
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	há 4 anos
JustAnotherArchivist	e99e2304c9	Write meta WARC with log file	há 4 anos
JustAnotherArchivist	85d78cee13	Add warcinfo record with version information on Python, system, and dependencies	há 4 anos
JustAnotherArchivist	9cff6bd5c1	Only open a WARC file when necessary to avoid producing empty WARCs at the end	há 4 anos
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	há 4 anos
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	há 5 anos
JustAnotherArchivist	e892a6b6a7	Initial commit	há 5 anos

32 Cometimentos (f025c4e9f36a661c44773fc60bba32f8ec1300e2)