7d229ccd1a
Add support for scraping classic Skyrim mods
...
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
2022-09-02 00:43:53 -04:00
337b673e02
Tolerate null picture_url in response
2022-01-18 21:03:26 -05:00
d48e2fdb5f
Add option to temporarily backfill mod data
2022-01-18 20:54:50 -05:00
Tyler Hallada
ba6ca0c928
Fix clippy lints, add 7z fallback for failed unrar
2021-08-14 11:43:09 -06:00
3ec7467571
Add more status columns to fix continuation of unprocessed files
2021-07-29 22:55:34 -04:00
8a356ac7f5
Retry connect failures, write plugins to disk instead of zip archive
...
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
2021-07-26 19:31:25 -04:00
523f3eeb3f
Add retry logic to metadata fetch, don't crash on 500
2021-07-25 18:03:54 -04:00
19126f4981
Add retry logic to API calls
...
Since apparently these can 500 occasionally.
2021-07-23 22:29:28 -04:00
62c0f5295f
Attempt to download file 3 times before crashing
2021-07-18 01:45:43 -04:00
c7df9dec13
Fix file metadata parsing
2021-07-15 10:54:53 -04:00
93d4bfeca2
Skip downloading files with no plugins
...
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
2021-07-12 19:33:43 -04:00
af5c11acc7
Tweak rate limit delay logic, log skipped files
2021-07-12 12:15:07 -04:00
ced31f0eb6
Handle null value for size_in_bytes in API files
2021-07-12 00:43:54 -04:00
b609e7059a
Log spans, track size & undownloadable files
2021-07-11 22:49:29 -04:00
792e78391c
Replace dbg macros with tracing events
2021-07-11 19:45:26 -04:00
22757bc475
Modularize scraping and api requests
2021-07-09 00:37:08 -04:00
