Commit Graph

12 Commits

Author SHA1 Message Date
bafa405223 Add tracing logging to all commands 2022-06-04 18:51:20 -04:00
026d175da4 Add updated_after option to limit files generated on every run
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
2022-06-04 00:50:11 -04:00
bed49d0953 Add a new command to dump files to json files 2022-06-01 00:54:29 -04:00
7a87f0fd22 Add command to generate mod_cell_counts.json
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
a42c22cf4b Add is_translation to mod with backfill
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
Tyler Hallada
5d55e78283 Dump plugins by hash instead of id
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.

Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
c9dcd3b7c5 Include cells with plugin data 2022-03-10 23:27:26 -05:00
7552573d00 Dump plugins data with file and mod data 2022-03-10 22:32:46 -05:00
21fffd3c23 Remove dbg logging 2022-02-12 01:25:30 -05:00
4d9855552b Add option for full update and default to false
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
283cad220b Move tile_scraper from examples to commands 2022-02-08 00:13:14 -05:00
b8c5f63aeb Refactor main.rs into separate files
Create new `commands` mod with file for each command line option.

Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00