fa2de316cb
Try to extract .exe before skipping it
2022-09-29 20:16:36 -04:00
49fcde836a
Skip .exe files
2022-09-29 20:10:55 -04:00
203af0adc9
Scope bulk_get_last_updated_by_nexus_mod_ids by game_id
...
Was accidentally skipping a ton of skyrim mods because of this.
2022-09-26 00:03:39 -04:00
e531386095
Detect failed extract with 7zip
...
And skip the file if the metadata on the server is also missing (high probability that the archive is just corrupt if this is the case).
2022-09-04 17:42:21 -04:00
d2510c72aa
Remove unused import
2022-09-03 15:43:36 -04:00
e96d760a1c
Create mod subdirectories for each game
2022-09-03 14:29:50 -04:00
14be03cd3d
Make separate mod search indices per game instead
...
Saves significant amount of space in the json if I don't need to list a game name for every mod.
2022-09-03 00:49:57 -04:00
84a02971a0
Add game names to mod search index
2022-09-03 00:35:26 -04:00
782f7a473f
Add dump_games command
...
So the frontend can map game db ids to nexus names.
2022-09-03 00:34:59 -04:00
7d229ccd1a
Add support for scraping classic Skyrim mods
...
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
2022-09-02 00:43:53 -04:00
b80edb49fa
Serialize plugin hashes to strings
...
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
2022-08-08 23:37:27 -04:00
09c87fa93b
Split up batched mod data query to speed up dump
2022-06-05 02:58:11 -04:00
bafa405223
Add tracing logging to all commands
2022-06-04 18:51:20 -04:00
026d175da4
Add updated_after option to limit files generated on every run
...
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
2022-06-04 00:50:11 -04:00
bed49d0953
Add a new command to dump files to json files
2022-06-01 00:54:29 -04:00
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
Tyler Hallada
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
21fffd3c23
Remove dbg logging
2022-02-12 01:25:30 -05:00
4d9855552b
Add option for full update and default to false
...
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
283cad220b
Move tile_scraper from examples to commands
2022-02-08 00:13:14 -05:00
b8c5f63aeb
Refactor main.rs into separate files
...
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00
