Commit Graph

66 Commits

Author SHA1 Message Date
ccac5b71de Upgrade skyrim-cell-dump to 0.1.4 2021-07-15 10:39:06 -04:00
93d4bfeca2 Skip downloading files with no plugins
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
2021-07-12 19:33:43 -04:00
fd8015024f Always uncompress entire rar archive
To avoid libarchive bugs.
2021-07-12 13:49:02 -04:00
af5c11acc7 Tweak rate limit delay logic, log skipped files 2021-07-12 12:15:07 -04:00
81fcdfa381 Skip processing of invalid empty plugins 2021-07-12 10:43:03 -04:00
ced31f0eb6 Handle null value for size_in_bytes in API files 2021-07-12 00:43:54 -04:00
4612ffc3c4 Add some .env instructions to readme 2021-07-11 22:59:53 -04:00
b609e7059a Log spans, track size & undownloadable files 2021-07-11 22:49:29 -04:00
792e78391c Replace dbg macros with tracing events 2021-07-11 19:45:26 -04:00
22757bc475 Modularize scraping and api requests 2021-07-09 00:37:08 -04:00
19350081c3 Move SQL stuff into separate models module 2021-07-08 21:19:16 -04:00
d8080091f2 Skip already processed mods, fix pagination 2021-07-06 23:29:09 -04:00
184d240aac Fix handling rar files
Extracts rar files completely with unrar to avoid issue with compress-tools not supporting extracting certain rar files.
2021-07-04 00:01:59 -04:00
Tyler Hallada
421f2b7071 WIP fix rar file extraction
Also, starting to break up into smaller functions
2021-07-03 16:00:18 -04:00
d6b8f4e74a Mostly working download loop done
Still need to fix a panic that happens on some .rar archive files.
2021-06-13 22:30:40 -04:00
b132a94c64 Initital commit
Not quite working yet. WIP.
2021-06-03 12:30:04 -04:00