.gitignore
Add infrastructure for simple C-based tools
pirms 2 gadiem
.make-and-exec
Warnings are bad, mmkay?
pirms 1 gada
.urldecode-test
Get rid of Makefile for more control; add proper debug build support
pirms 1 gada
.warc-dump-responses-test
Add test for warc-dump-responses
pirms 7 mēnešiem
.youtube-extract-rapid-test
Get rid of Makefile for more control; add proper debug build support
pirms 1 gada
LICENSE
Initial commit
pirms 5 gadiem
README.md
Initial commit
pirms 5 gadiem
alphabetseq
Swap syntaxes
pirms 2 gadiem
archivebot-blogspot
Fix HTTPS handling
pirms 5 gadiem
archivebot-compress-db
Add archivebot-compress-db
pirms 6 mēnešiem
archivebot-fix-queue-counters
Fix TypeError
pirms 1 gada
archivebot-high-resources
Replace archivebot-high-memory with more capable archivebot-high-resources
pirms 5 mēnešiem
archivebot-irccloud-paste
Add archivebot-irccloud-paste
pirms 3 gadiem
archivebot-jobid-calculation
More snscrape helper tools
pirms 5 gadiem
archivebot-jobs
Not-so-new new ArchiveBot domain
pirms 1 gada
archivebot-list-stuck-requests
Fix line endings
pirms 5 gadiem
archivebot-log-extract-ignores
Add archivebot-log-extract-ignores
pirms 3 gadiem
archivebot-monitor-job-queue
First set of little things
pirms 5 gadiem
archivebot-pipelines-count-jobs
Add archivebot-pipelines-count-jobs
pirms 7 mēnešiem
archivebot-youtube
Add helper for AB/chromebot-ing YouTube channels and users
pirms 5 gadiem
at-tracker-sample-user-item-size
Add at-tracker-sample-user-item-size
pirms 2 gadiem
azure-storage-list
Add --jsonl option
pirms 2 gadiem
b64grep
Add b64grep
pirms 2 gadiem
base64url
Add base64url
pirms 2 gadiem
bencode2json
Add bencode2json
pirms 1 gada
bing-scrape
Fix extraction of search results
pirms 8 mēnešiem
bugzilla-url-list
Add Bugzilla URL list generator
pirms 2 gadiem
cdx-chunk
Add cdx-chunk
pirms 2 gadiem
cloudflare-email-decode
Add cloudflare-email-decode
pirms 2 gadiem
combine-by-prefix
Add combine-by-prefix
pirms 2 gadiem
curl-ia
Add header mode (e.g. for tasks API)
pirms 1 gada
curl-ua
Add IE6 UA
pirms 3 gadiem
deb-repo-urls
Fix deb file URLs
pirms 4 gadiem
dedupe
Another alternative and performance/memory comparison
pirms 3 gadiem
dir-to-ia
Add dir-to-ia
pirms 1 gada
europarl-meps-collect
Add script for scraping MEP links from europarl.europa.eu
pirms 5 gadiem
extract-urls-for-archiveteam-projects
Add wpull2-extract-ignored-offsite and extract-urls-for-archiveteam-projects
pirms 6 mēnešiem
foolfuuka-search
Better workaround for the 5000 results limit; works for FoolFuuka 2.0.1 and up
pirms 5 gadiem
format-size
Split out size formatting
pirms 5 gadiem
fos-ftp-upload
First set of little things
pirms 5 gadiem
get-crx4chrome-urls
First set of little things
pirms 5 gadiem
github-list-repos
Fix org listings not including archived repos
pirms 2 mēnešiem
gitlab-list-repos
Add support for other instances and full-instance listing
pirms 2 gadiem
gofile.io-dl
Add support for password-protected folders
pirms 2 gadiem
html-extract-stupid
Handle and
pirms 1 gada
http-response-bodies
Add http-response-bodies
pirms 1 gada
http-response-bodies.c
Fix extra LF between chunks
pirms 11 mēnešiem
ia-cdx-search
Fix error when no arguments are provided
pirms 1 gada
ia-cdx-search-subdomains
Fix URLs without a path
pirms 2 gadiem
ia-derive
Queue derives with `ia tasks` instead of this manual curl rubbish
pirms 2 gadiem
ia-files-xml-to-jsonl
Guarantee stable output order
pirms 3 gadiem
ia-upload-progress
Proper script for tracking size of uploaded data
pirms 5 gadiem
ia-upload-stream
Reuse existing BytesIO object
pirms 2 mēnešiem
ia-verify-file
Add a timeout to prevent potentially indefinite blocking
pirms 2 gadiem
ia-wait-item-tasks
Handle error tasks by exiting non-zero
pirms 1 gada
iasha1check
Fix output sometimes appearing after prompt
pirms 1 gada
ix.io-upload
Allow overriding the "remote filename"
pirms 5 gadiem
kill-connections
Handle processes with too many open connections
pirms 2 gadiem
kill-wpull-connections
Merge kill-wpull-connections repository into little-things
pirms 3 gadiem
killcx-all-https
First set of little things
pirms 5 gadiem
mastodon-enumerate-users
Enumerate users on a Mastodon instance
pirms 5 gadiem
mastodon-outdated
Finding outdated Mastodon instances
pirms 5 gadiem
moinmoin-url-list
Add moinmoin-url-list
pirms 5 mēnešiem
parent-urls
Refactor, strip query/fragment
pirms 3 gadiem
pipelines-launch-in-tmux-windows
First set of little things
pirms 5 gadiem
pipelines-monitor-tmux-wget-outcomes
Monitor how a pipeline's wget processes are faring
pirms 5 gadiem
pipelines-stop-gracefully
First set of little things
pirms 5 gadiem
reddit-pushshift-search
Add Bing, Reddit/Pushshift, and FoolFuuka scrapers
pirms 5 gadiem
run-every-five-minutes
First set of little things
pirms 5 gadiem
s3-bucket-find-direct-url
Add support for PermanentRedirect error responses
pirms 8 mēnešiem
s3-bucket-list
Enable line buffering on list URLs FD
pirms 9 mēnešiem
s3-bucket-list-qwarc
Add JSONL output option for S3 listing
pirms 2 gadiem
snscrape-extract
Add support for Twitter hashtag extraction
pirms 5 gadiem
snscrape-facebook-user
Silence by default
pirms 5 gadiem
snscrape-instagram-user
Silence by default
pirms 5 gadiem
snscrape-prepare-commands
Add support for Twitter hashtag extraction
pirms 5 gadiem
snscrape-tmux
Update tmux session commands
pirms 5 gadiem
snscrape-twitter-filter
Filter Twitter hashtag scrapes based on account scrapes
pirms 5 gadiem
snscrape-twitter-hashtag
Extract external links from Twitter
pirms 5 gadiem
snscrape-twitter-user
Extract external links from Twitter
pirms 5 gadiem
snscrape-upload
Print Instagram ignore immediately after upload instead of at the end
pirms 5 gadiem
snscrape-vk-user
Silence by default
pirms 5 gadiem
snscrape-wiki-transfer-merge
Helper tools for snscrape and the wiki pages
pirms 5 gadiem
social-media-extract-profile-link
Fix decoding of links on Facebook profiles
pirms 4 gadiem
sum-sizes
Avoid float roundtrip for integer values
pirms 7 mēnešiem
tar-many-files-progress
First set of little things
pirms 5 gadiem
tcp-closer
Add tcp-closer command
pirms 5 gadiem
torrent-tiny
Fix negative ints
pirms 1 gada
transfer.archivete.am-upload
Handle HTTP/2 lowercase headers
pirms 3 gadiem
transfer.notkiska.pw-check-ia
Switch to HTTPS
pirms 3 gadiem
uniqify
Add uniqify
pirms 5 gadiem
uniqify-recent
Add uniqify-recent
pirms 4 mēnešiem
url-normalise
Normalise domain name to lower-case before further processing
pirms 4 gadiem
urldecode
Add URL/percent decoding tool
pirms 2 gadiem
urldecode.c
Fix unused argc and argv error
pirms 1 gada
urlsort
Add urlsort
pirms 2 gadiem
warc-dump-responses
Add warc-dump-responses
pirms 1 gada
warc-dump-responses.c
Fix error when the terminating CRLFCRLF of a record is truncated
pirms 7 mēnešiem
warc-peek
Allow negative offsets to peek near the end of the file
pirms 2 gadiem
warc-size
Split out size formatting
pirms 5 gadiem
warc-tiny
Fix empty files being considered valid WARCs
pirms 11 mēnešiem
website-extract-social-media
Add support for Facebook /pages/category/Category/Name-ID URLs
pirms 4 gadiem
wget-spider-estimate-size
First set of little things
pirms 5 gadiem
wiki-list-to-main
Add ArchiveBot wiki list helper
pirms 5 gadiem
wiki-recursive-extract-normalise
Fix deduplication within each section processing
pirms 4 gadiem
wiki-sections-sort
Add wiki-sections-sort
pirms 4 gadiem
wiki-website-extract-social-media
Add script for automatic social media discovery
pirms 4 gadiem
wpull1-parallel-progress-monitor
First set of little things
pirms 5 gadiem
wpull1-progress-monitor
First set of little things
pirms 5 gadiem
wpull2-extract-ignored
Remove filtering of onsite URLs because it's unreliable
pirms 6 mēnešiem
wpull2-extract-remaining
Clean up wpull DB commands
pirms 3 gadiem
wpull2-log-colourise
Add wpull2-log-colourise
pirms 1 gada
wpull2-log-extract-errors
Treat NXDOMAIN and no A/AAAA record errors as ok
pirms 3 gadiem
wpull2-requeue
Error on unknown options
pirms 2 gadiem
wpull2-url-origin
Clean up wpull DB commands
pirms 3 gadiem
youtube-channel-list.py
Use _type instead of key check hack
pirms 1 gada
youtube-extract
Exclude backslashes in channel patterns
pirms 1 gada
youtube-extract-rapid
Add youtube-extract-rapid
pirms 2 gadiem
youtube-extract-rapid.c
Add youtube-extract-rapid
pirms 2 gadiem
youtube-filter-autogen-channels
Add youtube-filter-autogen-channels
pirms 4 gadiem
zstdwarccat
Fix 'Dictionary mismatch' error when very small dicts are used because the temporary file isn't written to disk before zstdcat gets executed
pirms 2 gadiem