Commit Graph

122 Commits

Author SHA1 Message Date
a0aa24f360 Log warning and skip invalid plugins
There's not much I can do for plugins with invalid headers (e.g. TES4 Oblivion plugins), so just log the error and skip over it. A plugin row will not be created in the DB for invalid plugins.

It was useful to crash the process on parse errors before when I was working out the kinks in the parser, but I think I'm confident enough in it now to skip over the parse failures and defer analyzing the failures until after the fact to make sure there still isn't a bug in the parser and I skipped over a valid TES5 plugin.

Also fixes the plugin span logging in the delate64 error case when extracting zip files.
2021-07-18 15:09:23 -04:00
62c0f5295f Attempt to download file 3 times before crashing 2021-07-18 01:45:43 -04:00
4e500f0b0b Various fixes to errors that occurred during runs
Don't crash if metadata link 404s.

If extracting file from zip fails, try using linux unzip command instead. This fixes errors due to deflate64 compression.
2021-07-17 14:19:43 -04:00
5514fb5d29 Upgrade compress-tools to branch w/ fix for crash
See https://github.com/OSSystems/compress-tools-rs/issues/59 for details.
2021-07-16 00:13:38 -04:00
621b98cd8c Only process files (not dirs) extracted from rar 2021-07-15 13:46:33 -04:00
c7df9dec13 Fix file metadata parsing 2021-07-15 10:54:53 -04:00
ccac5b71de Upgrade skyrim-cell-dump to 0.1.4 2021-07-15 10:39:06 -04:00
93d4bfeca2 Skip downloading files with no plugins
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
2021-07-12 19:33:43 -04:00
fd8015024f Always uncompress entire rar archive
To avoid libarchive bugs.
2021-07-12 13:49:02 -04:00
af5c11acc7 Tweak rate limit delay logic, log skipped files 2021-07-12 12:15:07 -04:00
81fcdfa381 Skip processing of invalid empty plugins 2021-07-12 10:43:03 -04:00
ced31f0eb6 Handle null value for size_in_bytes in API files 2021-07-12 00:43:54 -04:00
4612ffc3c4 Add some .env instructions to readme 2021-07-11 22:59:53 -04:00
b609e7059a Log spans, track size & undownloadable files 2021-07-11 22:49:29 -04:00
792e78391c Replace dbg macros with tracing events 2021-07-11 19:45:26 -04:00
22757bc475 Modularize scraping and api requests 2021-07-09 00:37:08 -04:00
19350081c3 Move SQL stuff into separate models module 2021-07-08 21:19:16 -04:00
d8080091f2 Skip already processed mods, fix pagination 2021-07-06 23:29:09 -04:00
184d240aac Fix handling rar files
Extracts rar files completely with unrar to avoid issue with compress-tools not supporting extracting certain rar files.
2021-07-04 00:01:59 -04:00
Tyler Hallada
421f2b7071 WIP fix rar file extraction
Also, starting to break up into smaller functions
2021-07-03 16:00:18 -04:00
d6b8f4e74a Mostly working download loop done
Still need to fix a panic that happens on some .rar archive files.
2021-06-13 22:30:40 -04:00
b132a94c64 Initital commit
Not quite working yet. WIP.
2021-06-03 12:30:04 -04:00