62c0f5295f
Attempt to download file 3 times before crashing
2021-07-18 01:45:43 -04:00
4e500f0b0b
Various fixes to errors that occurred during runs
...
Don't crash if metadata link 404s.
If extracting file from zip fails, try using linux unzip command instead. This fixes errors due to deflate64 compression.
2021-07-17 14:19:43 -04:00
5514fb5d29
Upgrade compress-tools to branch w/ fix for crash
...
See https://github.com/OSSystems/compress-tools-rs/issues/59 for details.
2021-07-16 00:13:38 -04:00
621b98cd8c
Only process files (not dirs) extracted from rar
2021-07-15 13:46:33 -04:00
c7df9dec13
Fix file metadata parsing
2021-07-15 10:54:53 -04:00
ccac5b71de
Upgrade skyrim-cell-dump to 0.1.4
2021-07-15 10:39:06 -04:00
93d4bfeca2
Skip downloading files with no plugins
...
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
2021-07-12 19:33:43 -04:00
fd8015024f
Always uncompress entire rar archive
...
To avoid libarchive bugs.
2021-07-12 13:49:02 -04:00
af5c11acc7
Tweak rate limit delay logic, log skipped files
2021-07-12 12:15:07 -04:00
81fcdfa381
Skip processing of invalid empty plugins
2021-07-12 10:43:03 -04:00
ced31f0eb6
Handle null value for size_in_bytes in API files
2021-07-12 00:43:54 -04:00
4612ffc3c4
Add some .env instructions to readme
2021-07-11 22:59:53 -04:00
b609e7059a
Log spans, track size & undownloadable files
2021-07-11 22:49:29 -04:00
792e78391c
Replace dbg macros with tracing events
2021-07-11 19:45:26 -04:00
22757bc475
Modularize scraping and api requests
2021-07-09 00:37:08 -04:00
19350081c3
Move SQL stuff into separate models module
2021-07-08 21:19:16 -04:00
d8080091f2
Skip already processed mods, fix pagination
2021-07-06 23:29:09 -04:00
184d240aac
Fix handling rar files
...
Extracts rar files completely with unrar to avoid issue with compress-tools not supporting extracting certain rar files.
2021-07-04 00:01:59 -04:00
Tyler Hallada
421f2b7071
WIP fix rar file extraction
...
Also, starting to break up into smaller functions
2021-07-03 16:00:18 -04:00
d6b8f4e74a
Mostly working download loop done
...
Still need to fix a panic that happens on some .rar archive files.
2021-06-13 22:30:40 -04:00
b132a94c64
Initital commit
...
Not quite working yet. WIP.
2021-06-03 12:30:04 -04:00
