JustAnotherArchivist
|
a5dfd5c805
|
Write spec file + its dependencies and command line to meta WARC
|
4 vuotta sitten |
JustAnotherArchivist
|
e99e2304c9
|
Write meta WARC with log file
|
4 vuotta sitten |
JustAnotherArchivist
|
d751844626
|
Fix starting another item before stopping on STOP file or memory limit exceedance
|
4 vuotta sitten |
JustAnotherArchivist
|
2b0778f9b5
|
Remove leftovers from initial code rewrite
|
4 vuotta sitten |
JustAnotherArchivist
|
85d78cee13
|
Add warcinfo record with version information on Python, system, and dependencies
|
4 vuotta sitten |
JustAnotherArchivist
|
9eaa7be4c8
|
Python 3.7 compatibility
|
4 vuotta sitten |
JustAnotherArchivist
|
9cff6bd5c1
|
Only open a WARC file when necessary to avoid producing empty WARCs at the end
|
4 vuotta sitten |
JustAnotherArchivist
|
21cf784102
|
Use setuptools_scm for versioning
|
4 vuotta sitten |
JustAnotherArchivist
|
ab22966fef
|
Add to log which item a message is coming from
|
5 vuotta sitten |
JustAnotherArchivist
|
6fafd32685
|
Error when the retries are exceeded
|
5 vuotta sitten |
JustAnotherArchivist
|
8647d6b396
|
Use f-strings instead of str.format
|
5 vuotta sitten |
JustAnotherArchivist
|
5008e6e8cd
|
Deduplicate items
|
5 vuotta sitten |
JustAnotherArchivist
|
46c95e2157
|
Disable decoding the response content
chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.
|
5 vuotta sitten |
JustAnotherArchivist
|
91cd20f567
|
Version 0.1.3
|
5 vuotta sitten |
JustAnotherArchivist
|
85f6f7bd82
|
Make qwarc.utils.handle_response_limit_error_retries more useful by passing the deferring handler as an argument
|
5 vuotta sitten |
JustAnotherArchivist
|
ad22a2327a
|
Support adding headers to individual requests
|
5 vuotta sitten |
JustAnotherArchivist
|
67076f964c
|
Add support for POST requests
|
5 vuotta sitten |
JustAnotherArchivist
|
57764eb2b0
|
Version 0.1.2
|
5 vuotta sitten |
JustAnotherArchivist
|
2d52e78d85
|
Fix reference to aiohttp.CientError
|
5 vuotta sitten |
JustAnotherArchivist
|
0f107e988d
|
Version 0.1.1
|
5 vuotta sitten |
JustAnotherArchivist
|
c1574a06c9
|
Fix sleep task type
|
5 vuotta sitten |
JustAnotherArchivist
|
e0ca88c807
|
Fix reference to get_rss
|
5 vuotta sitten |
JustAnotherArchivist
|
984d28ede0
|
Fix type of --memorylimit, --disklimit, and --warcsplit values
|
5 vuotta sitten |
JustAnotherArchivist
|
8a8935810d
|
Fix references to memory and disk space check methods
|
5 vuotta sitten |
JustAnotherArchivist
|
1c8983fc1e
|
Version 0.1.0
|
5 vuotta sitten |
JustAnotherArchivist
|
be5673cfbf
|
Add record deduplication within a process
|
5 vuotta sitten |
JustAnotherArchivist
|
43f1b5e06e
|
Add LICENSE and README
|
5 vuotta sitten |
JustAnotherArchivist
|
e892a6b6a7
|
Initial commit
|
5 vuotta sitten |