thallada/modmapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
144 Commits 1 Branch 0 Tags
main
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada d31f44a349 Add healthchecks.io badge img to README 2025-09-03 22:43:10 -04:00
thallada ec3fa5e980 Marked scripts as executable 2025-09-03 22:37:26 -04:00
thallada 68c5cfe8a6 Add monitoring of scripts through healthchecks.io 
Updates the README with more cron script setup instructions.
 2025-09-03 22:28:11 -04:00
thallada 7b8fbea7d4 Update nexus scraper to use new graphql api 
Modmapper hasn't been pulling in new mods for a while because nexus mods changed their mod listing page to use a new graphql API and the old API I was using no longer works.
 2025-05-30 15:07:53 -04:00
thallada 1a105a3ea2 Update packages, fix unrar after major version upgrade 2025-03-02 16:01:21 -05:00
thallada a677325c4d Prevent overwriting of base game cells is_base_game value 
This fixes a bug that was causing cell data to not get written since
November 9th, 2023. When batch inserting cells while processing plugins,
I allowed overwriting cells that had `is_base_game = true`. Since I
always set `is_base_game = false` for cell upserts from plugins, this
was causing the base game cells to revert to `is_base_game = false`. All
it took was one mod to bundle `Skyrim.esm` for this to happen.

This broke writing cell data since `get_cell_data` depends on the
`is_base_game` value to find edits to the Skyrim base game cells.

To prevent this in the future, batch inserts when processing plugins is
no longer allowed to update cells which have `is_base_game = true`. The
only time we allow upserting these rows is when running the
`is_base_game` backfill which initially seeds the database with the base
game cells.
 2025-03-02 14:15:05 -05:00
thallada a5135b30f4 Add timestep option to dump_cell_edit_counts_over_time 2023-11-18 02:02:07 -05:00
thallada 7579db364b Recreate pg pool connection in all dump commands 
Since the slow query issue seems to be affecting all dump commands that
run queries in loops.
 2023-11-15 18:35:48 -05:00
thallada 8cb9bfa4ca Recreate pg connection pool in command to prevent slowdown 2023-11-15 18:07:26 -05:00
thallada c931eeca86 Add command for writing cell edits over time 2023-11-15 16:59:28 -05:00
thallada 9fcac0ce39 Fix reversed is_base_game_only logic in get_cell_data 2023-11-09 00:44:32 -05:00
thallada c7ead28cfa Fix cells unique index 2023-11-08 16:07:00 -05:00
thallada e85edff30a Fix bad interior cell data in db 
I had a bad unique index on the `cells` table. This commit does not fix
it but has two backfill scripts that will allow me to fixup the data
before creating the new unique index.

`is_base_game` identifies cells that are from the `Skyrim.esm` shipped
with the game (and not from some mod that happens to also have a
`Skyrim.esm` with a 0,0 cell with a different form id, see:
https://github.com/thallada/modmapper-web/issues/8).

`dump_cell_data` now only dumps cells from `is_base_game` cells.

`deduplicate_interior_cells` removes duplicate cell and plugin_cell rows
caused by the buggy index.

I will need to upgrade the database and create a new index in a later
commit.
 2023-11-08 01:10:49 -05:00
thallada 56864c7f29 Fix mode page scraping 
Was returning 403s for the past month.
 2023-06-30 12:30:12 -04:00
thallada 694ef6e89b Move spammy command log messages to debug level 
And add some info level logs that summarize each command.
 2023-04-24 00:04:29 -04:00
thallada 01444cd928 Update deps, rustfmt, clippy fixes 2023-04-20 00:47:14 -04:00
thallada 03cf20029e Revert "Login before scraping mods to include adult mods" 
This reverts commit bb02771b87.
 2022-10-02 02:37:15 -04:00
thallada bb02771b87 Login before scraping mods to include adult mods 2022-10-02 00:34:38 -04:00
thallada 675c1dda8a Update skyrim-cell-dump to 0.4.1 2022-10-01 17:57:02 -04:00
thallada 2bb7b78cd5 Update skyrim-cell-dump to 0.4.0 2022-09-30 16:47:43 -04:00
thallada 183c60f52d Update dependencies 2022-09-30 13:26:30 -04:00
thallada 3c16787756 Update skyrim-cell-dump to 0.3.4 2022-09-30 13:03:59 -04:00
thallada fa2de316cb Try to extract .exe before skipping it 2022-09-29 20:16:36 -04:00
thallada 7f0f5ca2ff Merge branch 'main' of github.com:thallada/modmapper into main 2022-09-29 20:10:58 -04:00
thallada 49fcde836a Skip .exe files 2022-09-29 20:10:55 -04:00
thallada 662bd7e355 Merge branch 'main' of github.com:thallada/modmapper into main 2022-09-27 12:19:10 -04:00
thallada 589b00923e Replace trailing period in folder names when saving plugin file 
NTFS apparently does not like that and it breaks.
 2022-09-27 12:18:25 -04:00
thallada cf4e83c054 Fix has_next_page scraping logic 
The next button doesn't appear on the last four pages. Instead, check if the selected page is the last page.
 2022-09-26 00:15:25 -04:00
thallada 203af0adc9 Scope bulk_get_last_updated_by_nexus_mod_ids by game_id 
Was accidentally skipping a ton of skyrim mods because of this.
 2022-09-26 00:03:39 -04:00
thallada c3264b3b11 Enable updating skyrim mods 2022-09-24 19:58:26 -04:00
thallada f2a4205b89 Handle missing author names 
For some reason, some older Skyrim mods have an author link but no author text. In that case, default the name to "Unknown".
 2022-09-17 12:25:59 -04:00
thallada afefc8fef3 Don't try to parse dirs after extracting with 7z 2022-09-12 21:03:29 -04:00
thallada fe908799d7 Try to fix saving plugins with "\\" in path 2022-09-05 18:47:43 -04:00
thallada e531386095 Detect failed extract with 7zip 
And skip the file if the metadata on the server is also missing (high probability that the archive is just corrupt if this is the case).
 2022-09-04 17:42:21 -04:00
thallada d2510c72aa Remove unused import 2022-09-03 15:43:36 -04:00
thallada e96d760a1c Create mod subdirectories for each game 2022-09-03 14:29:50 -04:00
thallada 14be03cd3d Make separate mod search indices per game instead 
Saves significant amount of space in the json if I don't need to list a game name for every mod.
 2022-09-03 00:49:57 -04:00
thallada 84a02971a0 Add game names to mod search index 2022-09-03 00:35:26 -04:00
thallada 782f7a473f Add dump_games command 
So the frontend can map game db ids to nexus names.
 2022-09-03 00:34:59 -04:00
thallada 7d229ccd1a Add support for scraping classic Skyrim mods 
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
 2022-09-02 00:43:53 -04:00
thallada 89428da6e0 Fix plugins data dump with updated_after parameter 
I was accidentally omitting plugins from the plugins data list when the `updated_after` parameter was used. Splitting it up into two queries makes it much simpler.
 2022-08-21 00:16:51 -04:00
thallada b80edb49fa Serialize plugin hashes to strings 
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
 2022-08-08 23:37:27 -04:00
thallada 33a29df2cb Dump file plugins with full_path, not file_name 
Needed to distinguish two plugins of the same name in a file.
 2022-07-23 01:00:48 -04:00
thallada 330fd8ad28 Remove destination folder from sync commands 2022-06-30 01:35:03 -04:00
thallada 1c1303e4f6 Separate buckets for each file type 2022-06-30 01:29:54 -04:00
thallada 4b7a52588c Use rclone for sync, add more instructions 2022-06-29 16:36:32 -04:00
thallada 09c87fa93b Split up batched mod data query to speed up dump 2022-06-05 02:58:11 -04:00
thallada d72ed943ce Append all command output to the log 2022-06-04 18:53:06 -04:00
thallada bafa405223 Add tracing logging to all commands 2022-06-04 18:51:20 -04:00
thallada 8f420c5558 Pass -u option to plugins_data command, not cells 2022-06-04 16:48:10 -04:00
thallada 026d175da4 Add updated_after option to limit files generated on every run 
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
 2022-06-04 00:50:11 -04:00
thallada bed49d0953 Add a new command to dump files to json files 2022-06-01 00:54:29 -04:00
thallada f59a0155d8 Add files array & file/plugin count to mod data json 2022-06-01 00:53:17 -04:00
thallada 04c0501514 Revert change to get_cell_data query 
I didn't meant to commit this, it was an experiment that ended up being way too slow.
 2022-03-23 17:24:41 -04:00
thallada 4e5284a57f Add mod_cell_counts dump to update.sh 2022-03-18 00:30:27 -04:00
thallada 65f841eef3 Go back to INNER JOIN on mod_cell_counts instead 
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
 2022-03-18 00:28:23 -04:00
thallada 02c03bd1d9 Use LEFT OUTER JOIN to include mods with 0 cell edits 2022-03-18 00:16:33 -04:00
thallada 7a87f0fd22 Add command to generate mod_cell_counts.json 
So that I can display cell edit counts in mod lists efficiently.
 2022-03-17 23:51:32 -04:00
thallada a42c22cf4b Add is_translation to mod with backfill 
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
 2022-03-15 23:59:56 -04:00
thallada c9dcd3b7c5 Include cells with plugin data 2022-03-10 23:27:26 -05:00
thallada 7552573d00 Dump plugins data with file and mod data 2022-03-10 22:32:46 -05:00
thallada 17f766da29 Fix rclone command in backup.sh 2022-02-13 14:34:09 -05:00
thallada bb2059e697 Fix find command options in backup.sh 2022-02-12 23:26:45 -05:00
thallada 21fffd3c23 Remove dbg logging 2022-02-12 01:25:30 -05:00
thallada eb92f7191d Add scripts for updating and backing up data 2022-02-12 00:44:32 -05:00
thallada 4d9855552b Add option for full update and default to false 
When false, stop scraping pages after 50 pages of no new updates
 2022-02-11 22:38:44 -05:00
thallada 283cad220b Move tile_scraper from examples to commands 2022-02-08 00:13:14 -05:00
thallada b8c5f63aeb Refactor main.rs into separate files 
Create new `commands` mod with file for each command line option.

Create a couple more extractor files for moving 7zip and unrar methods into.
 2022-02-08 00:02:26 -05:00
thallada 51175933d6 Fix skyrim worldspace filter in mod cells query 2022-02-06 23:31:31 -05:00
thallada 82996f237f Add missing migration for cell lookup index 2022-02-06 23:30:20 -05:00
thallada 891dbd791a Limit cells in mod files to Skyrim worldspace cells 2022-02-06 23:11:03 -05:00
thallada 240349cf1a Add option for generating a mod search index for fuse.js 2022-01-24 23:20:51 -05:00
thallada 4875d2d764 Use LEFT OUTER JOIN to not skip mods in the dump 2022-01-24 22:46:23 -05:00
thallada 0c3d85d697 More efficient batched_get_with_cells 2022-01-24 22:33:50 -05:00
thallada 17facae842 Make ancestor cols not nullable, add fk constraints 
Also removed the temporary backfill, now that it is complete.

Updates models so cols are not optional.
 2022-01-24 21:41:45 -05:00
thallada 10c299dc45 Add indices for new ancestor id columns 2022-01-24 21:21:12 -05:00
thallada e5936683fa Add ancestor id columns to plugins and plugin_cells 
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.

I'm hoping that this will improve query performance.
 2022-01-24 21:14:59 -05:00
thallada 4a8e3cad6c Add option for generating mod files 2022-01-24 01:24:15 -05:00
thallada e7fe7abfb7 Set columns to not null, remove backfill code 2022-01-18 21:50:09 -05:00
thallada 337b673e02 Tolerate null picture_url in response 2022-01-18 21:03:26 -05:00
thallada d48e2fdb5f Add option to temporarily backfill mod data 2022-01-18 20:54:50 -05:00
thallada f22587d4a0 Handle missing category_id in scraper 2022-01-18 17:03:22 -05:00
thallada e7b5c750cd Temporarily update all mods to backfill fields 2022-01-17 23:55:22 -05:00
thallada b97689b7fd Scrape additional fields to mod table 2022-01-17 23:37:58 -05:00
thallada f23cf526e5 Fix duplicate mods in cell data 2022-01-17 20:37:06 -05:00
thallada a78fa4772b Add cli option to save cell data json to directory 2022-01-17 19:25:07 -05:00
thallada 804fac4eea Start processing mod updates 2021-09-05 16:02:16 -04:00
thallada 0989477b40 Upgrade skyrim-cell-dump to 0.3.1 2021-08-27 00:48:03 -04:00
thallada b2d17f6217 Refactor main.rs a bit, some fixes to 7zip fallback 
Also adds reqwest timeouts
 2021-08-07 00:53:56 -04:00
thallada 3f48f97080 Fix typo 2021-07-29 23:45:18 -04:00
thallada 3ec7467571 Add more status columns to fix continuation of unprocessed files 2021-07-29 22:55:34 -04:00
thallada e1e7dd2e5d Rename drop_all.sql file 2021-07-29 01:01:27 -04:00
thallada fc5a9d39c2 Add timestamp to mods for some resumability 2021-07-29 00:50:42 -04:00
thallada a8424e830e Adds download_at to files, skips files that are impossible to download 2021-07-28 23:54:39 -04:00
thallada 8a356ac7f5 Retry connect failures, write plugins to disk instead of zip archive 
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
 2021-07-26 19:31:25 -04:00
thallada f62324d36c Reset main.rs back w/ fix to listing RAR files 2021-07-26 17:02:14 -04:00
thallada 5d62fb7d61 Fix plugin listing for rar files, temporary backfil shim 
`main` is just backfilling the eixsting bad rar files in the db. I will return script to normal once it's run.
 2021-07-26 01:16:43 -04:00
thallada 87ca90be06 Fix plugin insert SQL binding 2021-07-25 18:33:45 -04:00
thallada 523f3eeb3f Add retry logic to metadata fetch, don't crash on 500 2021-07-25 18:03:54 -04:00
thallada 39ae7703b0 Upgrade skyrim-cell-dump and handle Cow<str>s 2021-07-23 23:40:05 -04:00