The WARC writing is only tested and known safe on Linux and relies on its behaviour of flock(2) and files opened in append mode (writes always append regardless of the stream position).

The fnctl.flock call does not use LOCK_NB, so if it fails with an OSError, that is Really Bad™ and can't be handled cleanly.

Inspired by the implementation in wpull but structured differently to avoid reopening the journal file constantly.

As a consequence, SpecDependencies.extra can now be any data type that can be put into JSON; unhashable types previously caused a crash due to the lru_cache.

The DB creation operates with a DB lock, so that code can't run while another process is filling the DB; it would block on obtaining the lock a few lines prior instead.

It is rarely necessary to access the history, and the tuple return value clutters the spec file code.

As a consequence, it's no longer possible to return None if an error occurred without losing the history.
To replace that, this also introduces a DummyClientResponse, which is kind of ClientResponse-like, has the same qhistory attribute, and evaluates to False when cast to bool (such that the intuitive `if response` works as expected).

Since Python 3.8, the latter does not inherit from the former anymore.

The asyncio documentation does not specify whether awaiting a Future multiple times is supported or not: https://bugs.python.org/issue41275

Commit 429ac946 relies on order-preserving dicts, i.e. Python 3.7+

If a response ends with '0\r\n' or '0\r\n\r', ClientResponse._read loops forever trying to read 4 more bytes.

In addition, bump that read to 1 KiB for better worst-case performance.

https://github.com/aio-libs/aiohttp/issues/4630

Closing the raw data tempfiles immediately on connection reuse caused any response reading to fail with an I/O error if another request started on the same connection in the meantime. Delaying the closing until the response object falls out of scope and gets GC'd ensures that as long as there is a reference to that object, it can be read from, at the expense of a possibly larger memory overhead.

Note that this should never matter anyway because the response is already run through the payload parser before.

For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.

If an exception was raised and caught, the object is still present in the history.

For example, if the connection breaks while retrieving a response but after the headers have been parsed, the response body would be incomplete.

https://github.com/webrecorder/warcio/issues/94

This also renames add_item to add_subitem for clarity.

WARC 1.1 specification, section 5.14: "A ‘warcinfo’ record shall not have a WARC-Target-URI field."