thallada/modmapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
126 Commits 1 Branch 0 Tags
675c1dda8a91f873631864658b24a52966663db5
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada 675c1dda8a Update skyrim-cell-dump to 0.4.1 2022-10-01 17:57:02 -04:00
thallada 2bb7b78cd5 Update skyrim-cell-dump to 0.4.0 2022-09-30 16:47:43 -04:00
thallada 183c60f52d Update dependencies 2022-09-30 13:26:30 -04:00
thallada 3c16787756 Update skyrim-cell-dump to 0.3.4 2022-09-30 13:03:59 -04:00
thallada fa2de316cb Try to extract .exe before skipping it 2022-09-29 20:16:36 -04:00
thallada 7f0f5ca2ff Merge branch 'main' of github.com:thallada/modmapper into main 2022-09-29 20:10:58 -04:00
thallada 49fcde836a Skip .exe files 2022-09-29 20:10:55 -04:00
thallada 662bd7e355 Merge branch 'main' of github.com:thallada/modmapper into main 2022-09-27 12:19:10 -04:00
thallada 589b00923e Replace trailing period in folder names when saving plugin file 
NTFS apparently does not like that and it breaks.
 2022-09-27 12:18:25 -04:00
thallada cf4e83c054 Fix has_next_page scraping logic 
The next button doesn't appear on the last four pages. Instead, check if the selected page is the last page.
 2022-09-26 00:15:25 -04:00
thallada 203af0adc9 Scope bulk_get_last_updated_by_nexus_mod_ids by game_id 
Was accidentally skipping a ton of skyrim mods because of this.
 2022-09-26 00:03:39 -04:00
thallada c3264b3b11 Enable updating skyrim mods 2022-09-24 19:58:26 -04:00
thallada f2a4205b89 Handle missing author names 
For some reason, some older Skyrim mods have an author link but no author text. In that case, default the name to "Unknown".
 2022-09-17 12:25:59 -04:00
thallada afefc8fef3 Don't try to parse dirs after extracting with 7z 2022-09-12 21:03:29 -04:00
thallada fe908799d7 Try to fix saving plugins with "\\" in path 2022-09-05 18:47:43 -04:00
thallada e531386095 Detect failed extract with 7zip 
And skip the file if the metadata on the server is also missing (high probability that the archive is just corrupt if this is the case).
 2022-09-04 17:42:21 -04:00
thallada d2510c72aa Remove unused import 2022-09-03 15:43:36 -04:00
thallada e96d760a1c Create mod subdirectories for each game 2022-09-03 14:29:50 -04:00
thallada 14be03cd3d Make separate mod search indices per game instead 
Saves significant amount of space in the json if I don't need to list a game name for every mod.
 2022-09-03 00:49:57 -04:00
thallada 84a02971a0 Add game names to mod search index 2022-09-03 00:35:26 -04:00
thallada 782f7a473f Add dump_games command 
So the frontend can map game db ids to nexus names.
 2022-09-03 00:34:59 -04:00
thallada 7d229ccd1a Add support for scraping classic Skyrim mods 
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
 2022-09-02 00:43:53 -04:00
thallada 89428da6e0 Fix plugins data dump with updated_after parameter 
I was accidentally omitting plugins from the plugins data list when the `updated_after` parameter was used. Splitting it up into two queries makes it much simpler.
 2022-08-21 00:16:51 -04:00
thallada b80edb49fa Serialize plugin hashes to strings 
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
 2022-08-08 23:37:27 -04:00
thallada 33a29df2cb Dump file plugins with full_path, not file_name 
Needed to distinguish two plugins of the same name in a file.
 2022-07-23 01:00:48 -04:00
thallada 330fd8ad28 Remove destination folder from sync commands 2022-06-30 01:35:03 -04:00
thallada 1c1303e4f6 Separate buckets for each file type 2022-06-30 01:29:54 -04:00
thallada 4b7a52588c Use rclone for sync, add more instructions 2022-06-29 16:36:32 -04:00
thallada 09c87fa93b Split up batched mod data query to speed up dump 2022-06-05 02:58:11 -04:00
thallada d72ed943ce Append all command output to the log 2022-06-04 18:53:06 -04:00
thallada bafa405223 Add tracing logging to all commands 2022-06-04 18:51:20 -04:00
thallada 8f420c5558 Pass -u option to plugins_data command, not cells 2022-06-04 16:48:10 -04:00
thallada 026d175da4 Add updated_after option to limit files generated on every run 
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
 2022-06-04 00:50:11 -04:00
thallada bed49d0953 Add a new command to dump files to json files 2022-06-01 00:54:29 -04:00
thallada f59a0155d8 Add files array & file/plugin count to mod data json 2022-06-01 00:53:17 -04:00
thallada 04c0501514 Revert change to get_cell_data query 
I didn't meant to commit this, it was an experiment that ended up being way too slow.
 2022-03-23 17:24:41 -04:00
thallada 4e5284a57f Add mod_cell_counts dump to update.sh 2022-03-18 00:30:27 -04:00
thallada 65f841eef3 Go back to INNER JOIN on mod_cell_counts instead 
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
 2022-03-18 00:28:23 -04:00
thallada 02c03bd1d9 Use LEFT OUTER JOIN to include mods with 0 cell edits 2022-03-18 00:16:33 -04:00
thallada 7a87f0fd22 Add command to generate mod_cell_counts.json 
So that I can display cell edit counts in mod lists efficiently.
 2022-03-17 23:51:32 -04:00
thallada a42c22cf4b Add is_translation to mod with backfill 
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
 2022-03-15 23:59:56 -04:00
thallada c9dcd3b7c5 Include cells with plugin data 2022-03-10 23:27:26 -05:00
thallada 7552573d00 Dump plugins data with file and mod data 2022-03-10 22:32:46 -05:00
thallada 17f766da29 Fix rclone command in backup.sh 2022-02-13 14:34:09 -05:00
thallada bb2059e697 Fix find command options in backup.sh 2022-02-12 23:26:45 -05:00
thallada 21fffd3c23 Remove dbg logging 2022-02-12 01:25:30 -05:00
thallada eb92f7191d Add scripts for updating and backing up data 2022-02-12 00:44:32 -05:00
thallada 4d9855552b Add option for full update and default to false 
When false, stop scraping pages after 50 pages of no new updates
 2022-02-11 22:38:44 -05:00
thallada 283cad220b Move tile_scraper from examples to commands 2022-02-08 00:13:14 -05:00
thallada b8c5f63aeb Refactor main.rs into separate files 
Create new `commands` mod with file for each command line option.

Create a couple more extractor files for moving 7zip and unrar methods into.
 2022-02-08 00:02:26 -05:00
thallada 51175933d6 Fix skyrim worldspace filter in mod cells query 2022-02-06 23:31:31 -05:00
thallada 82996f237f Add missing migration for cell lookup index 2022-02-06 23:30:20 -05:00
thallada 891dbd791a Limit cells in mod files to Skyrim worldspace cells 2022-02-06 23:11:03 -05:00
thallada 240349cf1a Add option for generating a mod search index for fuse.js 2022-01-24 23:20:51 -05:00
thallada 4875d2d764 Use LEFT OUTER JOIN to not skip mods in the dump 2022-01-24 22:46:23 -05:00
thallada 0c3d85d697 More efficient batched_get_with_cells 2022-01-24 22:33:50 -05:00
thallada 17facae842 Make ancestor cols not nullable, add fk constraints 
Also removed the temporary backfill, now that it is complete.

Updates models so cols are not optional.
 2022-01-24 21:41:45 -05:00
thallada 10c299dc45 Add indices for new ancestor id columns 2022-01-24 21:21:12 -05:00
thallada e5936683fa Add ancestor id columns to plugins and plugin_cells 
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.

I'm hoping that this will improve query performance.
 2022-01-24 21:14:59 -05:00
thallada 4a8e3cad6c Add option for generating mod files 2022-01-24 01:24:15 -05:00
thallada e7fe7abfb7 Set columns to not null, remove backfill code 2022-01-18 21:50:09 -05:00
thallada 337b673e02 Tolerate null picture_url in response 2022-01-18 21:03:26 -05:00
thallada d48e2fdb5f Add option to temporarily backfill mod data 2022-01-18 20:54:50 -05:00
thallada f22587d4a0 Handle missing category_id in scraper 2022-01-18 17:03:22 -05:00
thallada e7b5c750cd Temporarily update all mods to backfill fields 2022-01-17 23:55:22 -05:00
thallada b97689b7fd Scrape additional fields to mod table 2022-01-17 23:37:58 -05:00
thallada f23cf526e5 Fix duplicate mods in cell data 2022-01-17 20:37:06 -05:00
thallada a78fa4772b Add cli option to save cell data json to directory 2022-01-17 19:25:07 -05:00
thallada 804fac4eea Start processing mod updates 2021-09-05 16:02:16 -04:00
thallada 0989477b40 Upgrade skyrim-cell-dump to 0.3.1 2021-08-27 00:48:03 -04:00
thallada b2d17f6217 Refactor main.rs a bit, some fixes to 7zip fallback 
Also adds reqwest timeouts
 2021-08-07 00:53:56 -04:00
thallada 3f48f97080 Fix typo 2021-07-29 23:45:18 -04:00
thallada 3ec7467571 Add more status columns to fix continuation of unprocessed files 2021-07-29 22:55:34 -04:00
thallada e1e7dd2e5d Rename drop_all.sql file 2021-07-29 01:01:27 -04:00
thallada fc5a9d39c2 Add timestamp to mods for some resumability 2021-07-29 00:50:42 -04:00
thallada a8424e830e Adds download_at to files, skips files that are impossible to download 2021-07-28 23:54:39 -04:00
thallada 8a356ac7f5 Retry connect failures, write plugins to disk instead of zip archive 
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
 2021-07-26 19:31:25 -04:00
thallada f62324d36c Reset main.rs back w/ fix to listing RAR files 2021-07-26 17:02:14 -04:00
thallada 5d62fb7d61 Fix plugin listing for rar files, temporary backfil shim 
`main` is just backfilling the eixsting bad rar files in the db. I will return script to normal once it's run.
 2021-07-26 01:16:43 -04:00
thallada 87ca90be06 Fix plugin insert SQL binding 2021-07-25 18:33:45 -04:00
thallada 523f3eeb3f Add retry logic to metadata fetch, don't crash on 500 2021-07-25 18:03:54 -04:00
thallada 39ae7703b0 Upgrade skyrim-cell-dump and handle Cow<str>s 2021-07-23 23:40:05 -04:00
thallada 19126f4981 Add retry logic to API calls 
Since apparently these can 500 occasionally.
 2021-07-23 22:29:28 -04:00
thallada 81f12da99c Upgrade unrar to master branch to fix Utf8Error 2021-07-23 19:34:03 -04:00
thallada e69165fe5a Set page size to 20 since nexus mods doesn't always obey 80 2021-07-22 21:55:31 -04:00
thallada 77562a7a05 Add CLI args with page, skip unreadable archives 2021-07-22 21:50:28 -04:00
thallada e534447310 Simplify bulk_get_present_nexus_mod_ids query 2021-07-22 00:13:20 -04:00
thallada f337917c99 Batch insert mods, add lifetimes to batch inserts 
Using `&str`s prevents unneeded cloning everywhere, but unfortunately it doesn't work with the sqlx macros.
 2021-07-21 23:20:51 -04:00
thallada 3fe09a8f8f Upgrade skyrim-cell-dump to 0.2.1 
Fixes bug parsing Unofficial Skyrim Special Edition Patch
 2021-07-21 22:26:16 -04:00
thallada 04773954f9 Remove backfill, add bulk get game_mod method 2021-07-21 22:25:45 -04:00
thallada 4b333b3b99 Add world tables and columns, temporary backfill 
Too lazy to make a new script, main() has a backfill using plugins.zip. Once I run it I will remove it.
 2021-07-21 21:35:11 -04:00
thallada 0e0fdfd59d Migration and models for worlds/plugin_worlds 2021-07-19 23:47:00 -04:00
thallada 7eba0c6adc Upgrade skyrim-cell-dump to 0.2.0 2021-07-19 23:45:51 -04:00
thallada 1fd979b8e0 Skip archived files which cannot be downloaded 2021-07-18 22:11:03 -04:00
thallada f6a7490214 Skip empty or invalid file archives 2021-07-18 16:06:49 -04:00
thallada a0aa24f360 Log warning and skip invalid plugins 
There's not much I can do for plugins with invalid headers (e.g. TES4 Oblivion plugins), so just log the error and skip over it. A plugin row will not be created in the DB for invalid plugins.

It was useful to crash the process on parse errors before when I was working out the kinks in the parser, but I think I'm confident enough in it now to skip over the parse failures and defer analyzing the failures until after the fact to make sure there still isn't a bug in the parser and I skipped over a valid TES5 plugin.

Also fixes the plugin span logging in the delate64 error case when extracting zip files.
 2021-07-18 15:09:23 -04:00
thallada 62c0f5295f Attempt to download file 3 times before crashing 2021-07-18 01:45:43 -04:00
thallada 4e500f0b0b Various fixes to errors that occurred during runs 
Don't crash if metadata link 404s.

If extracting file from zip fails, try using linux unzip command instead. This fixes errors due to deflate64 compression.
 2021-07-17 14:19:43 -04:00
thallada 5514fb5d29 Upgrade compress-tools to branch w/ fix for crash 
See https://github.com/OSSystems/compress-tools-rs/issues/59 for details.
 2021-07-16 00:13:38 -04:00
thallada 621b98cd8c Only process files (not dirs) extracted from rar 2021-07-15 13:46:33 -04:00