JustAnotherArchivist
|
50d46ad51c
|
Use log filename in the target URI of the log resource record
|
4 years ago |
JustAnotherArchivist
|
a5dfd5c805
|
Write spec file + its dependencies and command line to meta WARC
|
4 years ago |
JustAnotherArchivist
|
d751844626
|
Fix starting another item before stopping on STOP file or memory limit exceedance
|
5 years ago |
JustAnotherArchivist
|
2b0778f9b5
|
Remove leftovers from initial code rewrite
|
5 years ago |
JustAnotherArchivist
|
ab22966fef
|
Add to log which item a message is coming from
|
5 years ago |
JustAnotherArchivist
|
6fafd32685
|
Error when the retries are exceeded
|
5 years ago |
JustAnotherArchivist
|
8647d6b396
|
Use f-strings instead of str.format
|
5 years ago |
JustAnotherArchivist
|
5008e6e8cd
|
Deduplicate items
|
5 years ago |
JustAnotherArchivist
|
46c95e2157
|
Disable decoding the response content
chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.
|
5 years ago |
JustAnotherArchivist
|
ad22a2327a
|
Support adding headers to individual requests
|
5 years ago |
JustAnotherArchivist
|
67076f964c
|
Add support for POST requests
|
5 years ago |
JustAnotherArchivist
|
c1574a06c9
|
Fix sleep task type
|
5 years ago |
JustAnotherArchivist
|
e0ca88c807
|
Fix reference to get_rss
|
5 years ago |
JustAnotherArchivist
|
8a8935810d
|
Fix references to memory and disk space check methods
|
5 years ago |
JustAnotherArchivist
|
be5673cfbf
|
Add record deduplication within a process
|
5 years ago |
JustAnotherArchivist
|
e892a6b6a7
|
Initial commit
|
5 years ago |