Closing the raw data tempfiles immediately on connection reuse caused any response reading to fail with an I/O error if another request started on the same connection in the meantime. Delaying the closing until the response object falls out of scope and gets GC'd ensures that as long as there is a reference to that object, it can be read from, at the expense of a possibly larger memory overhead.

Note that this should never matter anyway because the response is already run through the payload parser before.

For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.

If an exception was raised and caught, the object is still present in the history.

For example, if the connection breaks while retrieving a response but after the headers have been parsed, the response body would be incomplete.

https://github.com/webrecorder/warcio/issues/94

This also renames add_item to add_subitem for clarity.

WARC 1.1 specification, section 5.14: "A ‘warcinfo’ record shall not have a WARC-Target-URI field."

The length has to be set manually because otherwise warcio will automatically remove the header again.