e85edff30a
Fix bad interior cell data in db
...
I had a bad unique index on the `cells` table. This commit does not fix
it but has two backfill scripts that will allow me to fixup the data
before creating the new unique index.
`is_base_game` identifies cells that are from the `Skyrim.esm` shipped
with the game (and not from some mod that happens to also have a
`Skyrim.esm` with a 0,0 cell with a different form id, see:
https://github.com/thallada/modmapper-web/issues/8 ).
`dump_cell_data` now only dumps cells from `is_base_game` cells.
`deduplicate_interior_cells` removes duplicate cell and plugin_cell rows
caused by the buggy index.
I will need to upgrade the database and create a new index in a later
commit.
2023-11-08 01:10:49 -05:00
56864c7f29
Fix mode page scraping
...
Was returning 403s for the past month.
2023-06-30 12:30:12 -04:00
694ef6e89b
Move spammy command log messages to debug level
...
And add some info level logs that summarize each command.
2023-04-24 00:04:29 -04:00
01444cd928
Update deps, rustfmt, clippy fixes
2023-04-20 00:47:14 -04:00
03cf20029e
Revert "Login before scraping mods to include adult mods"
...
This reverts commit bb02771b87
2022-10-02 02:37:15 -04:00
bb02771b87
Login before scraping mods to include adult mods
2022-10-02 00:34:38 -04:00
675c1dda8a
Update skyrim-cell-dump to 0.4.1
2022-10-01 17:57:02 -04:00
2bb7b78cd5
Update skyrim-cell-dump to 0.4.0
2022-09-30 16:47:43 -04:00
183c60f52d
Update dependencies
2022-09-30 13:26:30 -04:00
3c16787756
Update skyrim-cell-dump to 0.3.4
2022-09-30 13:03:59 -04:00
fa2de316cb
Try to extract .exe before skipping it
2022-09-29 20:16:36 -04:00
7f0f5ca2ff
Merge branch 'main' of github.com:thallada/modmapper into main
2022-09-29 20:10:58 -04:00
49fcde836a
Skip .exe files
2022-09-29 20:10:55 -04:00
662bd7e355
Merge branch 'main' of github.com:thallada/modmapper into main
2022-09-27 12:19:10 -04:00
589b00923e
Replace trailing period in folder names when saving plugin file
...
NTFS apparently does not like that and it breaks.
2022-09-27 12:18:25 -04:00
cf4e83c054
Fix has_next_page scraping logic
...
The next button doesn't appear on the last four pages. Instead, check if the selected page is the last page.
2022-09-26 00:15:25 -04:00
203af0adc9
Scope bulk_get_last_updated_by_nexus_mod_ids by game_id
...
Was accidentally skipping a ton of skyrim mods because of this.
2022-09-26 00:03:39 -04:00
c3264b3b11
Enable updating skyrim mods
2022-09-24 19:58:26 -04:00
f2a4205b89
Handle missing author names
...
For some reason, some older Skyrim mods have an author link but no author text. In that case, default the name to "Unknown".
2022-09-17 12:25:59 -04:00
afefc8fef3
Don't try to parse dirs after extracting with 7z
2022-09-12 21:03:29 -04:00
fe908799d7
Try to fix saving plugins with "\\" in path
2022-09-05 18:47:43 -04:00
e531386095
Detect failed extract with 7zip
...
And skip the file if the metadata on the server is also missing (high probability that the archive is just corrupt if this is the case).
2022-09-04 17:42:21 -04:00
d2510c72aa
Remove unused import
2022-09-03 15:43:36 -04:00
e96d760a1c
Create mod subdirectories for each game
2022-09-03 14:29:50 -04:00
14be03cd3d
Make separate mod search indices per game instead
...
Saves significant amount of space in the json if I don't need to list a game name for every mod.
2022-09-03 00:49:57 -04:00
84a02971a0
Add game names to mod search index
2022-09-03 00:35:26 -04:00
782f7a473f
Add dump_games command
...
So the frontend can map game db ids to nexus names.
2022-09-03 00:34:59 -04:00
7d229ccd1a
Add support for scraping classic Skyrim mods
...
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
2022-09-02 00:43:53 -04:00
89428da6e0
Fix plugins data dump with updated_after parameter
...
I was accidentally omitting plugins from the plugins data list when the `updated_after` parameter was used. Splitting it up into two queries makes it much simpler.
2022-08-21 00:16:51 -04:00
b80edb49fa
Serialize plugin hashes to strings
...
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
2022-08-08 23:37:27 -04:00
33a29df2cb
Dump file plugins with full_path, not file_name
...
Needed to distinguish two plugins of the same name in a file.
2022-07-23 01:00:48 -04:00
330fd8ad28
Remove destination folder from sync commands
2022-06-30 01:35:03 -04:00
1c1303e4f6
Separate buckets for each file type
2022-06-30 01:29:54 -04:00
4b7a52588c
Use rclone for sync, add more instructions
2022-06-29 16:36:32 -04:00
09c87fa93b
Split up batched mod data query to speed up dump
2022-06-05 02:58:11 -04:00
d72ed943ce
Append all command output to the log
2022-06-04 18:53:06 -04:00
bafa405223
Add tracing logging to all commands
2022-06-04 18:51:20 -04:00
8f420c5558
Pass -u option to plugins_data command, not cells
2022-06-04 16:48:10 -04:00
026d175da4
Add updated_after option to limit files generated on every run
...
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
2022-06-04 00:50:11 -04:00
bed49d0953
Add a new command to dump files to json files
2022-06-01 00:54:29 -04:00
f59a0155d8
Add files array & file/plugin count to mod data json
2022-06-01 00:53:17 -04:00
04c0501514
Revert change to get_cell_data query
...
I didn't meant to commit this, it was an experiment that ended up being way too slow.
2022-03-23 17:24:41 -04:00
4e5284a57f
Add mod_cell_counts dump to update.sh
2022-03-18 00:30:27 -04:00
65f841eef3
Go back to INNER JOIN on mod_cell_counts instead
...
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
2022-03-18 00:28:23 -04:00
02c03bd1d9
Use LEFT OUTER JOIN to include mods with 0 cell edits
2022-03-18 00:16:33 -04:00
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
Tyler Hallada
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
