f22587d4a0
Handle missing category_id in scraper
2022-01-18 17:03:22 -05:00
e7b5c750cd
Temporarily update all mods to backfill fields
2022-01-17 23:55:22 -05:00
b97689b7fd
Scrape additional fields to mod table
2022-01-17 23:37:58 -05:00
f23cf526e5
Fix duplicate mods in cell data
2022-01-17 20:37:06 -05:00
a78fa4772b
Add cli option to save cell data json to directory
2022-01-17 19:25:07 -05:00
Tyler Hallada
e779e94eff
Add -e option to output cell edits from database
2022-01-15 00:19:05 -05:00
Tyler Hallada
dad58f6154
Add UESP map tile scraper as an example
2022-01-15 00:18:30 -05:00
804fac4eea
Start processing mod updates
2021-09-05 16:02:16 -04:00
0989477b40
Upgrade skyrim-cell-dump to 0.3.1
2021-08-27 00:48:03 -04:00
Tyler Hallada
ba6ca0c928
Fix clippy lints, add 7z fallback for failed unrar
2021-08-14 11:43:09 -06:00
Tyler Hallada
be9c78217d
Skip files with unknown file type
2021-08-12 10:25:28 -06:00
Tyler Hallada
85f1d3d05a
Recover failed 7z uncompress too
2021-08-11 10:10:02 -06:00
Tyler Hallada
1169ef4536
Print human readable file size to log
2021-08-09 22:38:17 -06:00
Tyler Hallada
db2e73aa8e
Skip unrar extract errors, update compress-tools
2021-08-09 11:10:57 -06:00
Tyler Hallada
6136ac18fa
Make mod category nullable
2021-08-08 19:14:23 -06:00
b2d17f6217
Refactor main.rs a bit, some fixes to 7zip fallback
...
Also adds reqwest timeouts
2021-08-07 00:53:56 -04:00
3f48f97080
Fix typo
2021-07-29 23:45:18 -04:00
3ec7467571
Add more status columns to fix continuation of unprocessed files
2021-07-29 22:55:34 -04:00
e1e7dd2e5d
Rename drop_all.sql file
2021-07-29 01:01:27 -04:00
fc5a9d39c2
Add timestamp to mods for some resumability
2021-07-29 00:50:42 -04:00
a8424e830e
Adds download_at to files, skips files that are impossible to download
2021-07-28 23:54:39 -04:00
8a356ac7f5
Retry connect failures, write plugins to disk instead of zip archive
...
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
2021-07-26 19:31:25 -04:00
f62324d36c
Reset main.rs back w/ fix to listing RAR files
2021-07-26 17:02:14 -04:00
5d62fb7d61
Fix plugin listing for rar files, temporary backfil shim
...
`main` is just backfilling the eixsting bad rar files in the db. I will return script to normal once it's run.
2021-07-26 01:16:43 -04:00
87ca90be06
Fix plugin insert SQL binding
2021-07-25 18:33:45 -04:00
523f3eeb3f
Add retry logic to metadata fetch, don't crash on 500
2021-07-25 18:03:54 -04:00
39ae7703b0
Upgrade skyrim-cell-dump and handle Cow<str>s
2021-07-23 23:40:05 -04:00
19126f4981
Add retry logic to API calls
...
Since apparently these can 500 occasionally.
2021-07-23 22:29:28 -04:00
81f12da99c
Upgrade unrar to master branch to fix Utf8Error
2021-07-23 19:34:03 -04:00
e69165fe5a
Set page size to 20 since nexus mods doesn't always obey 80
2021-07-22 21:55:31 -04:00
77562a7a05
Add CLI args with page, skip unreadable archives
2021-07-22 21:50:28 -04:00
e534447310
Simplify bulk_get_present_nexus_mod_ids query
2021-07-22 00:13:20 -04:00
f337917c99
Batch insert mods, add lifetimes to batch inserts
...
Using `&str`s prevents unneeded cloning everywhere, but unfortunately it doesn't work with the sqlx macros.
2021-07-21 23:20:51 -04:00
3fe09a8f8f
Upgrade skyrim-cell-dump to 0.2.1
...
Fixes bug parsing Unofficial Skyrim Special Edition Patch
2021-07-21 22:26:16 -04:00
04773954f9
Remove backfill, add bulk get game_mod method
2021-07-21 22:25:45 -04:00
4b333b3b99
Add world tables and columns, temporary backfill
...
Too lazy to make a new script, main() has a backfill using plugins.zip. Once I run it I will remove it.
2021-07-21 21:35:11 -04:00
0e0fdfd59d
Migration and models for worlds/plugin_worlds
2021-07-19 23:47:00 -04:00
7eba0c6adc
Upgrade skyrim-cell-dump to 0.2.0
2021-07-19 23:45:51 -04:00
1fd979b8e0
Skip archived files which cannot be downloaded
2021-07-18 22:11:03 -04:00
f6a7490214
Skip empty or invalid file archives
2021-07-18 16:06:49 -04:00
a0aa24f360
Log warning and skip invalid plugins
...
There's not much I can do for plugins with invalid headers (e.g. TES4 Oblivion plugins), so just log the error and skip over it. A plugin row will not be created in the DB for invalid plugins.
It was useful to crash the process on parse errors before when I was working out the kinks in the parser, but I think I'm confident enough in it now to skip over the parse failures and defer analyzing the failures until after the fact to make sure there still isn't a bug in the parser and I skipped over a valid TES5 plugin.
Also fixes the plugin span logging in the delate64 error case when extracting zip files.
2021-07-18 15:09:23 -04:00
62c0f5295f
Attempt to download file 3 times before crashing
2021-07-18 01:45:43 -04:00
4e500f0b0b
Various fixes to errors that occurred during runs
...
Don't crash if metadata link 404s.
If extracting file from zip fails, try using linux unzip command instead. This fixes errors due to deflate64 compression.
2021-07-17 14:19:43 -04:00
5514fb5d29
Upgrade compress-tools to branch w/ fix for crash
...
See https://github.com/OSSystems/compress-tools-rs/issues/59 for details.
2021-07-16 00:13:38 -04:00
621b98cd8c
Only process files (not dirs) extracted from rar
2021-07-15 13:46:33 -04:00
c7df9dec13
Fix file metadata parsing
2021-07-15 10:54:53 -04:00
ccac5b71de
Upgrade skyrim-cell-dump to 0.1.4
2021-07-15 10:39:06 -04:00
93d4bfeca2
Skip downloading files with no plugins
...
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
2021-07-12 19:33:43 -04:00
fd8015024f
Always uncompress entire rar archive
...
To avoid libarchive bugs.
2021-07-12 13:49:02 -04:00
af5c11acc7
Tweak rate limit delay logic, log skipped files
2021-07-12 12:15:07 -04:00
