thallada
fa2de316cb
Try to extract .exe before skipping it
2022-09-29 20:16:36 -04:00
thallada
49fcde836a
Skip .exe files
2022-09-29 20:10:55 -04:00
thallada
203af0adc9
Scope bulk_get_last_updated_by_nexus_mod_ids by game_id
...
Was accidentally skipping a ton of skyrim mods because of this.
2022-09-26 00:03:39 -04:00
thallada
e531386095
Detect failed extract with 7zip
...
And skip the file if the metadata on the server is also missing (high probability that the archive is just corrupt if this is the case).
2022-09-04 17:42:21 -04:00
thallada
d2510c72aa
Remove unused import
2022-09-03 15:43:36 -04:00
thallada
e96d760a1c
Create mod subdirectories for each game
2022-09-03 14:29:50 -04:00
thallada
14be03cd3d
Make separate mod search indices per game instead
...
Saves significant amount of space in the json if I don't need to list a game name for every mod.
2022-09-03 00:49:57 -04:00
thallada
84a02971a0
Add game names to mod search index
2022-09-03 00:35:26 -04:00
thallada
782f7a473f
Add dump_games command
...
So the frontend can map game db ids to nexus names.
2022-09-03 00:34:59 -04:00
thallada
7d229ccd1a
Add support for scraping classic Skyrim mods
...
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
2022-09-02 00:43:53 -04:00
thallada
b80edb49fa
Serialize plugin hashes to strings
...
Avoids having dealing with JSON parsers on the frontend that loose precision and avoids having to convert integers to strings.
2022-08-08 23:37:27 -04:00
thallada
09c87fa93b
Split up batched mod data query to speed up dump
2022-06-05 02:58:11 -04:00
thallada
bafa405223
Add tracing logging to all commands
2022-06-04 18:51:20 -04:00
thallada
026d175da4
Add updated_after option to limit files generated on every run
...
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
2022-06-04 00:50:11 -04:00
thallada
bed49d0953
Add a new command to dump files to json files
2022-06-01 00:54:29 -04:00
thallada
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
thallada
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
thallada
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
thallada
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
thallada
21fffd3c23
Remove dbg logging
2022-02-12 01:25:30 -05:00
thallada
4d9855552b
Add option for full update and default to false
...
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
thallada
283cad220b
Move tile_scraper from examples to commands
2022-02-08 00:13:14 -05:00
thallada
b8c5f63aeb
Refactor main.rs into separate files
...
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00
thallada
Tyler Hallada