54 Комити (f025c4e9f36a661c44773fc60bba32f8ec1300e2)

Аутор SHA1 Порука Датум
  JustAnotherArchivist f025c4e9f3 Add extensive debug logging пре 3 година
  JustAnotherArchivist ce7f8fdc92 Make optional arguments to fetch kwarg-only пре 3 година
  JustAnotherArchivist 3c8b45b3a6 Refactor cleanup code пре 3 година
  JustAnotherArchivist dcd5455388 Fix crash on starting a run while the DB is locked пре 3 година
  JustAnotherArchivist 168fa78736 Avoid locking the DB when there are no subitems to insert пре 3 година
  JustAnotherArchivist 4484d6c588 Add Item representation пре 3 година
  JustAnotherArchivist 5675118877 Rename id to id_ to avoid clash with builtin пре 3 година
  JustAnotherArchivist a1e693739e Replace DB locking with an async context manager пре 3 година
  JustAnotherArchivist 15203bd991 Handle redirect traps/loops пре 3 година
  JustAnotherArchivist f8f5258197 Track redirect depth пре 3 година
  JustAnotherArchivist a3d6fb35f8 Turn response handlers into kwarg-only functions for easier extendability without breaking existing code пре 3 година
  JustAnotherArchivist 6cc4adb901 Remove stray TODO пре 3 година
  JustAnotherArchivist c5604ef965 Simplify header merging пре 3 година
  JustAnotherArchivist 59ae1183d2 Add fromResponse parameter for URL completion and automatic Referer header пре 3 година
  JustAnotherArchivist 2324216016 Add baseUrl and evaluate incomplete URLs relative to it пре 3 година
  JustAnotherArchivist b30ccf8bf8 Move response/exception history to ClientResponse.qhistory пре 3 година
  JustAnotherArchivist e69527c715 Add defaultResponseHandler on the Item level пре 3 година
  JustAnotherArchivist 03336e4988 Add item to response handler arguments (e.g. for logging) пре 3 година
  JustAnotherArchivist 6bdcfe71f0 Refactor database creation and item generation: call `Item.generate()` on every qwarc run and dedupe its output, allowing the addition of further items by modifying the spec file пре 3 година
  JustAnotherArchivist c878241f24 Switch from concurrent.futures.CancelledError to asyncio.CancelledError пре 3 година
  JustAnotherArchivist 749158b97a Use the Future's result directly rather than awaiting again пре 3 година
  JustAnotherArchivist a85e80ffa2 Configurable request timeout пре 3 година
  JustAnotherArchivist 429ac94689 Make it possible to override and remove headers пре 3 година
  JustAnotherArchivist e40be54578 Document verify_ssl parameter пре 3 година
  JustAnotherArchivist d3437bde19 Move default headers to qwarc.const пре 3 година
  JustAnotherArchivist 1678075a89 Log traceback on exceptions raised from an item пре 4 година
  JustAnotherArchivist b1a1c03f7e Handle STOP file and high memory usage before full disk to allow stopping while the disk is above the limit пре 4 година
  JustAnotherArchivist dd44d9b174 Adjust logging levels: log individual request failures only at WARNING and cancelled tasks at ERROR level пре 4 година
  JustAnotherArchivist 91035d769c Catch exceptions in Item.process and mark the items as errors instead of crashing пре 4 година
  JustAnotherArchivist 69984765b3 Fix taskType typo silencing cancellation warnings пре 4 година
  JustAnotherArchivist c263ad0b03 Return ClientResponse object from fetch only if the retrieval was successful пре 4 година
  JustAnotherArchivist cb0d11284e Write only successful retrievals (i.e. ones that don't cause an exception) to WARC пре 4 година
  JustAnotherArchivist 1214409a0b Flush big responses to a temporary file instead of trying to keep everything in-memory пре 4 година
  JustAnotherArchivist 08d96b37c5 Support deep/multiple inheritance from Item пре 4 година
  JustAnotherArchivist 9d8de13775 Add Item.flush_subitems to flush the new subitems to the database while the item is still being processed пре 4 година
  JustAnotherArchivist 50b936b18c Refactor QWARC class to keep relevant variables in instance attributes instead of local variables пре 4 година
  JustAnotherArchivist c5d8d93166 Remove stray whitespace пре 4 година
  JustAnotherArchivist 7d53577522 Add parameter for disabling SSL/TLS certificate validation пре 4 година
  JustAnotherArchivist 50d46ad51c Use log filename in the target URI of the log resource record пре 4 година
  JustAnotherArchivist a5dfd5c805 Write spec file + its dependencies and command line to meta WARC пре 4 година
  JustAnotherArchivist d751844626 Fix starting another item before stopping on STOP file or memory limit exceedance пре 4 година
  JustAnotherArchivist 2b0778f9b5 Remove leftovers from initial code rewrite пре 4 година
  JustAnotherArchivist ab22966fef Add to log which item a message is coming from пре 4 година
  JustAnotherArchivist 6fafd32685 Error when the retries are exceeded пре 4 година
  JustAnotherArchivist 8647d6b396 Use f-strings instead of str.format пре 4 година
  JustAnotherArchivist 5008e6e8cd Deduplicate items пре 4 година
  JustAnotherArchivist 46c95e2157 Disable decoding the response content пре 4 година
  JustAnotherArchivist ad22a2327a Support adding headers to individual requests пре 5 година
  JustAnotherArchivist 67076f964c Add support for POST requests пре 5 година
  JustAnotherArchivist c1574a06c9 Fix sleep task type пре 5 година