thallada/modmapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
104 Commits 1 Branch 0 Tags
89428da6e079a2643b4f613a9b5db50c75ce2bd7
Commit Graph
94 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada 89428da6e0 Fix plugins data dump with updated_after parameter 
I was accidentally omitting plugins from the plugins data list when the `updated_after` parameter was used. Splitting it up into two queries makes it much simpler.
 2022-08-21 00:16:51 -04:00
thallada b80edb49fa Serialize plugin hashes to strings 
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
 2022-08-08 23:37:27 -04:00
thallada 33a29df2cb Dump file plugins with full_path, not file_name 
Needed to distinguish two plugins of the same name in a file.
 2022-07-23 01:00:48 -04:00
thallada 330fd8ad28 Remove destination folder from sync commands 2022-06-30 01:35:03 -04:00
thallada 1c1303e4f6 Separate buckets for each file type 2022-06-30 01:29:54 -04:00
thallada 4b7a52588c Use rclone for sync, add more instructions 2022-06-29 16:36:32 -04:00
thallada 09c87fa93b Split up batched mod data query to speed up dump 2022-06-05 02:58:11 -04:00
thallada d72ed943ce Append all command output to the log 2022-06-04 18:53:06 -04:00
thallada bafa405223 Add tracing logging to all commands 2022-06-04 18:51:20 -04:00
thallada 8f420c5558 Pass -u option to plugins_data command, not cells 2022-06-04 16:48:10 -04:00
thallada 026d175da4 Add updated_after option to limit files generated on every run 
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
 2022-06-04 00:50:11 -04:00
thallada bed49d0953 Add a new command to dump files to json files 2022-06-01 00:54:29 -04:00
thallada f59a0155d8 Add files array & file/plugin count to mod data json 2022-06-01 00:53:17 -04:00
thallada 04c0501514 Revert change to get_cell_data query 
I didn't meant to commit this, it was an experiment that ended up being way too slow.
 2022-03-23 17:24:41 -04:00
thallada 4e5284a57f Add mod_cell_counts dump to update.sh 2022-03-18 00:30:27 -04:00
thallada 65f841eef3 Go back to INNER JOIN on mod_cell_counts instead 
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
 2022-03-18 00:28:23 -04:00
thallada 02c03bd1d9 Use LEFT OUTER JOIN to include mods with 0 cell edits 2022-03-18 00:16:33 -04:00
thallada 7a87f0fd22 Add command to generate mod_cell_counts.json 
So that I can display cell edit counts in mod lists efficiently.
 2022-03-17 23:51:32 -04:00
thallada a42c22cf4b Add is_translation to mod with backfill 
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
 2022-03-15 23:59:56 -04:00
thallada c9dcd3b7c5 Include cells with plugin data 2022-03-10 23:27:26 -05:00
thallada 7552573d00 Dump plugins data with file and mod data 2022-03-10 22:32:46 -05:00
thallada 17f766da29 Fix rclone command in backup.sh 2022-02-13 14:34:09 -05:00
thallada bb2059e697 Fix find command options in backup.sh 2022-02-12 23:26:45 -05:00
thallada 21fffd3c23 Remove dbg logging 2022-02-12 01:25:30 -05:00
thallada eb92f7191d Add scripts for updating and backing up data 2022-02-12 00:44:32 -05:00
thallada 4d9855552b Add option for full update and default to false 
When false, stop scraping pages after 50 pages of no new updates
 2022-02-11 22:38:44 -05:00
thallada 283cad220b Move tile_scraper from examples to commands 2022-02-08 00:13:14 -05:00
thallada b8c5f63aeb Refactor main.rs into separate files 
Create new `commands` mod with file for each command line option.

Create a couple more extractor files for moving 7zip and unrar methods into.
 2022-02-08 00:02:26 -05:00
thallada 51175933d6 Fix skyrim worldspace filter in mod cells query 2022-02-06 23:31:31 -05:00
thallada 82996f237f Add missing migration for cell lookup index 2022-02-06 23:30:20 -05:00
thallada 891dbd791a Limit cells in mod files to Skyrim worldspace cells 2022-02-06 23:11:03 -05:00
thallada 240349cf1a Add option for generating a mod search index for fuse.js 2022-01-24 23:20:51 -05:00
thallada 4875d2d764 Use LEFT OUTER JOIN to not skip mods in the dump 2022-01-24 22:46:23 -05:00
thallada 0c3d85d697 More efficient batched_get_with_cells 2022-01-24 22:33:50 -05:00
thallada 17facae842 Make ancestor cols not nullable, add fk constraints 
Also removed the temporary backfill, now that it is complete.

Updates models so cols are not optional.
 2022-01-24 21:41:45 -05:00
thallada 10c299dc45 Add indices for new ancestor id columns 2022-01-24 21:21:12 -05:00
thallada e5936683fa Add ancestor id columns to plugins and plugin_cells 
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.

I'm hoping that this will improve query performance.
 2022-01-24 21:14:59 -05:00
thallada 4a8e3cad6c Add option for generating mod files 2022-01-24 01:24:15 -05:00
thallada e7fe7abfb7 Set columns to not null, remove backfill code 2022-01-18 21:50:09 -05:00
thallada 337b673e02 Tolerate null picture_url in response 2022-01-18 21:03:26 -05:00
thallada d48e2fdb5f Add option to temporarily backfill mod data 2022-01-18 20:54:50 -05:00
thallada f22587d4a0 Handle missing category_id in scraper 2022-01-18 17:03:22 -05:00
thallada e7b5c750cd Temporarily update all mods to backfill fields 2022-01-17 23:55:22 -05:00
thallada b97689b7fd Scrape additional fields to mod table 2022-01-17 23:37:58 -05:00
thallada f23cf526e5 Fix duplicate mods in cell data 2022-01-17 20:37:06 -05:00
thallada a78fa4772b Add cli option to save cell data json to directory 2022-01-17 19:25:07 -05:00
thallada 804fac4eea Start processing mod updates 2021-09-05 16:02:16 -04:00
thallada 0989477b40 Upgrade skyrim-cell-dump to 0.3.1 2021-08-27 00:48:03 -04:00
thallada b2d17f6217 Refactor main.rs a bit, some fixes to 7zip fallback 
Also adds reqwest timeouts
 2021-08-07 00:53:56 -04:00
thallada 3f48f97080 Fix typo 2021-07-29 23:45:18 -04:00
thallada 3ec7467571 Add more status columns to fix continuation of unprocessed files 2021-07-29 22:55:34 -04:00
thallada e1e7dd2e5d Rename drop_all.sql file 2021-07-29 01:01:27 -04:00
thallada fc5a9d39c2 Add timestamp to mods for some resumability 2021-07-29 00:50:42 -04:00
thallada a8424e830e Adds download_at to files, skips files that are impossible to download 2021-07-28 23:54:39 -04:00
thallada 8a356ac7f5 Retry connect failures, write plugins to disk instead of zip archive 
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
 2021-07-26 19:31:25 -04:00
thallada f62324d36c Reset main.rs back w/ fix to listing RAR files 2021-07-26 17:02:14 -04:00
thallada 5d62fb7d61 Fix plugin listing for rar files, temporary backfil shim 
`main` is just backfilling the eixsting bad rar files in the db. I will return script to normal once it's run.
 2021-07-26 01:16:43 -04:00
thallada 87ca90be06 Fix plugin insert SQL binding 2021-07-25 18:33:45 -04:00
thallada 523f3eeb3f Add retry logic to metadata fetch, don't crash on 500 2021-07-25 18:03:54 -04:00
thallada 39ae7703b0 Upgrade skyrim-cell-dump and handle Cow<str>s 2021-07-23 23:40:05 -04:00
thallada 19126f4981 Add retry logic to API calls 
Since apparently these can 500 occasionally.
 2021-07-23 22:29:28 -04:00
thallada 81f12da99c Upgrade unrar to master branch to fix Utf8Error 2021-07-23 19:34:03 -04:00
thallada e69165fe5a Set page size to 20 since nexus mods doesn't always obey 80 2021-07-22 21:55:31 -04:00
thallada 77562a7a05 Add CLI args with page, skip unreadable archives 2021-07-22 21:50:28 -04:00
thallada e534447310 Simplify bulk_get_present_nexus_mod_ids query 2021-07-22 00:13:20 -04:00
thallada f337917c99 Batch insert mods, add lifetimes to batch inserts 
Using `&str`s prevents unneeded cloning everywhere, but unfortunately it doesn't work with the sqlx macros.
 2021-07-21 23:20:51 -04:00
thallada 3fe09a8f8f Upgrade skyrim-cell-dump to 0.2.1 
Fixes bug parsing Unofficial Skyrim Special Edition Patch
 2021-07-21 22:26:16 -04:00
thallada 04773954f9 Remove backfill, add bulk get game_mod method 2021-07-21 22:25:45 -04:00
thallada 4b333b3b99 Add world tables and columns, temporary backfill 
Too lazy to make a new script, main() has a backfill using plugins.zip. Once I run it I will remove it.
 2021-07-21 21:35:11 -04:00
thallada 0e0fdfd59d Migration and models for worlds/plugin_worlds 2021-07-19 23:47:00 -04:00
thallada 7eba0c6adc Upgrade skyrim-cell-dump to 0.2.0 2021-07-19 23:45:51 -04:00
thallada 1fd979b8e0 Skip archived files which cannot be downloaded 2021-07-18 22:11:03 -04:00
thallada f6a7490214 Skip empty or invalid file archives 2021-07-18 16:06:49 -04:00
thallada a0aa24f360 Log warning and skip invalid plugins 
There's not much I can do for plugins with invalid headers (e.g. TES4 Oblivion plugins), so just log the error and skip over it. A plugin row will not be created in the DB for invalid plugins.

It was useful to crash the process on parse errors before when I was working out the kinks in the parser, but I think I'm confident enough in it now to skip over the parse failures and defer analyzing the failures until after the fact to make sure there still isn't a bug in the parser and I skipped over a valid TES5 plugin.

Also fixes the plugin span logging in the delate64 error case when extracting zip files.
 2021-07-18 15:09:23 -04:00
thallada 62c0f5295f Attempt to download file 3 times before crashing 2021-07-18 01:45:43 -04:00
thallada 4e500f0b0b Various fixes to errors that occurred during runs 
Don't crash if metadata link 404s.

If extracting file from zip fails, try using linux unzip command instead. This fixes errors due to deflate64 compression.
 2021-07-17 14:19:43 -04:00
thallada 5514fb5d29 Upgrade compress-tools to branch w/ fix for crash 
See https://github.com/OSSystems/compress-tools-rs/issues/59 for details.
 2021-07-16 00:13:38 -04:00
thallada 621b98cd8c Only process files (not dirs) extracted from rar 2021-07-15 13:46:33 -04:00
thallada c7df9dec13 Fix file metadata parsing 2021-07-15 10:54:53 -04:00
thallada ccac5b71de Upgrade skyrim-cell-dump to 0.1.4 2021-07-15 10:39:06 -04:00
thallada 93d4bfeca2 Skip downloading files with no plugins 
Can determine if file contains plugin by recursively walking the file metadata listing until a .esp/.esm/.esl is found.
 2021-07-12 19:33:43 -04:00
thallada fd8015024f Always uncompress entire rar archive 
To avoid libarchive bugs.
 2021-07-12 13:49:02 -04:00
thallada af5c11acc7 Tweak rate limit delay logic, log skipped files 2021-07-12 12:15:07 -04:00
thallada 81fcdfa381 Skip processing of invalid empty plugins 2021-07-12 10:43:03 -04:00
thallada ced31f0eb6 Handle null value for size_in_bytes in API files 2021-07-12 00:43:54 -04:00
thallada 4612ffc3c4 Add some .env instructions to readme 2021-07-11 22:59:53 -04:00
thallada b609e7059a Log spans, track size & undownloadable files 2021-07-11 22:49:29 -04:00
thallada 792e78391c Replace dbg macros with tracing events 2021-07-11 19:45:26 -04:00
thallada 22757bc475 Modularize scraping and api requests 2021-07-09 00:37:08 -04:00
thallada 19350081c3 Move SQL stuff into separate models module 2021-07-08 21:19:16 -04:00
thallada d8080091f2 Skip already processed mods, fix pagination 2021-07-06 23:29:09 -04:00
thallada 184d240aac Fix handling rar files 
Extracts rar files completely with unrar to avoid issue with compress-tools not supporting extracting certain rar files.
 2021-07-04 00:01:59 -04:00
thallada d6b8f4e74a Mostly working download loop done 
Still need to fix a panic that happens on some .rar archive files.
 2021-06-13 22:30:40 -04:00
thallada b132a94c64 Initital commit 
Not quite working yet. WIP.
 2021-06-03 12:30:04 -04:00